patrick/Vosklet
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Change file structure

Browse Source
This commit is contained in:
msqr1
2024-02-05 17:27:30 -08:00
parent 6d097a9853
commit 56398e54a9
8 changed files with 77 additions and 6181 deletions
Download Patch File Download Diff File Expand all files Collapse all files

100
README.md
View File

@@ -1,18 +1,45 @@
# Browser-recognizer
# Overview
- A speech recognizer built on Vosk that can be run on the browser, inspired by [vosk-browser](https://github.com/ccoreilly/vosk-browser), but built from scratch and no code taken!
- The API is also designed with strong exception safety
## Global and all objects' common interface
| Function signature (global) | Description |
- Designed with strong exception safety
# Additions to vosk-browser:
- Download multiple models
- Model storage path management (for multiple models)
- Model ID management (for model updates)
# User agent notes
## SharedArrayBuffer
Browser-recognizer require SharedArrayBuffer to share thread's data, so these response headers must be set:
- ***Cross-Origin-Embedder-Policy*** ---> ***require-corp***
- ***Cross-Origin-Opener-Policy*** ---> ***same-origin***
If you can't set them, you may use a HACKY workaround at *src/addCOI.js*.
## Origin Private Filesystem (OPFS)
Browser-recognizer needs the Emscripten WASMFS' OPFS to store its model, IDBFS was considered, but discarded because there is no direct way to read from IDBFS to C++ without copying to MEMFS. To be safe, always:
- Try catch ```window.loadBR()``` to to check for OPFS availability.
- Check if there is enough space for **model + compressed model** before calling makeModel via ```navigator.storage.estimate()```
# API interface
## JS ```window``` object
| Function signature | Description |
|---|---|
|```Promise<Module> loadBR()``` | Load the Emscripten Module
## Shared interface
| Function signature | Description |
|---|---|
| ```delete()``` | Delete this object, see [why](https://emscripten.org/docs/getting_started/FAQ.html#what-does-exiting-the-runtime-mean-why-don-t-atexit-s-run) this is neccessary.
## ```Module``` object
| Function signature | Description |
|---|---|
| ```Promise<Model> makeModel(path: string, url: string, id: string)```<br><br>```Promise<SpkModel> makeSpkModel(path: string, url: string, id: string)``` | Make a ```Model``` or ```SpkModel```<br>- If **path** contains valid model files and **id** is the same, there will not be a fetch from **url**.<br>- If **path** doesn't contain valid model files, or if it contains valid model files but **id** is different, there will be a fetch from **url**, and the model is stored with **id**. |
| ```Promise<Recognizer> makeRecognizer(model: Model, sampleRate: float)``` | Make a ```Recognizer```, it will use **model**'s thread if it's the first user of **model**, else it will use a new thread.
| ```setLogLevel(lvl: int)``` | Set Vosk's log level (default: ```0```: Info) <br>```-2```: Error<br>```-1```: Warning<br>```1```: Verbose<br>```2```: More verbose<br>```3```: Debug |
| ```cleanUp()``` | Call ```delete()``` on all objects and revoke all Blob URLs. |
| ```cleanUp()``` | A convenient function that call ```delete()``` on all objects and revoke all Blob URLs. |
| Function signature (all objects) | Description
|---|---|
| ```delete()``` | Delete this object, see [why](https://emscripten.org/docs/getting_started/FAQ.html#what-does-exiting-the-runtime-mean-why-don-t-atexit-s-run) this is neccessary.
## ```Recognizer``` object
## ```Recognizer``` object
| Function signature | Description |
|---|---|
| ```Promise<AudioWorkletNode> getNode(ctx: AudioContext, channelIndex = 0: int)``` | Get a pass-through node that recognize audio and is connectable to a processing graph. It has 1 input and 1 output, **channelIndex** must point to a 16-bit mono channel of the input |
@@ -28,12 +55,13 @@
|---|---|
| ```partialResult``` | There is a partial recognition result, check the event's "details" property |
| ```result``` | There is a full recognition result, check the event's "details" property |
## Compilation
Changing any setting to non-default values requires recompilation
# Compilation
Changing any option to non-default values requires recompilation
```
git clone --depth=1 https://github.com/msqr1/Browser-recognizer &&
cd Browser-recognizer &&
[Options] ./compile.sh
[Options] make
```
| Option | Description | Default value |
|---|---|---|
@@ -41,45 +69,9 @@ cd Browser-recognizer &&
| MAX_THREADS | Set the max number of thread (2 min) | ```2``` (1 OPFS thread + 1 model/recognizer thread) |
| COMPILE_JOBS | Set the number of jobs (threads) when compiling | ```$(nproc)``` |
| EMSDK | Set EMSDK's path (will install EMSDK in root folder if unset) | ```../emsdk``` |
## Response headers
Browser-recognizer require SharedArrayBuffer, so these response headers must be set:
- ***Cross-Origin-Embedder-Policy*** ---> ***require-corp***
- ***Cross-Origin-Opener-Policy*** ---> ***same-origin***
If you can't set them, you may use a VERY HACKY workaround at *src/addCOI.js*.
## Additions to vosk-browser:
- Download multiple models
- Model storage path management (when many models are required)
- Model ID management (when model updates are required)
## Usage
```
<!--Load this from a script tag-->
<script src="BrowserRecognizer.js"></script>
<script>
// Select name
const BrRec = await loadBR()
// Prepare
const model = await BrRec.makeModel(")
const recognizer = await BrRec.makeRecognizer(model)
recognizer.addEventListener("result", e => {
console.log("Result: ",e.details)
})
recognizer.addEventListener("partialResult", e => {
console.log("Partial result: ",e.details)
})
// Microphone setup
media = await navigator.mediaDevices.getUserMedia({
video: false,
audio: {
echoCancellation: true,
noiseSuppression: true,
channelCount: 1,
sampleRate: 16000
},
})
</script>
```
# TODO:
- Fix libarchive extract closing issue
- setSpkModel avoid spawning extra thread
- Top level await in usage
- Write examples
- Balance code size and speed

0
test/en-model.tzst → examples/en-model.tzst
View File

0
test/test.wav → examples/example.wav
View File

3
test/index.html → examples/fromMic.html
View File

@@ -28,7 +28,4 @@
})()*/
</script>
</head>
<body>
<button>I am just another button lol</button>
</body>
</html>

31
examples/fromWav.html Normal file
View File

@@ -0,0 +1,31 @@
<!DOCTYPE html>
<html>
<head>
<script src="BrowserRecognizer.js"></script>
<script>
/*(async () => {
const BrRec = await loadBR()
const model = await BrRec.makeModel("en-model.tzst","model","en0.0.1")
let ctx = new AudioContext()
const recognizer = await BrRec.makeRecognizer(model, ctx.sampleRate)
recognizer.addEventListener("result", e => {
console.log("Result: ",e.details)
})
recognizer.addEventListener("partialResult", e => {
console.log("Partial result: ",e.details)
})
let micNode = ctx.createMediaStreamSource(await navigator.mediaDevices.getUserMedia({
video: false,
audio: {
echoCancellation: true,
noiseSuppression: true,
channelCount: 1,
sampleRate: 16000
},
}));
let recNode = recognizer.getNode(ctx)
})()*/
</script>
</head>
</html>

0
examples/withSpkModel.html Normal file
View File

6082
test/BrowserRecognizer.js
View File

File diff suppressed because one or more lines are too long

42
test/test.sh
View File

@@ -1,42 +0,0 @@
cd .. &&
SRC=$(realpath src) &&
KALDI=$(realpath kaldi) &&
VOSK=$(realpath vosk) &&
OPENFST=$(realpath openfst) &&
LIBARCHIVE=$(realpath libarchive) &&
ZSTD=$(realpath zstd) &&
CLAPACK_WASM=$(realpath clapack-wasm) &&
MAX_MEMORY=${MAX_MEMORY:-300mb} &&
MAX_THREADS=${MAX_THREADS:-2} &&
EMSDK=${EMSDK:-$(realpath ../emsdk)} &&
if [ ! -d $EMSDK_PATH ]; then
echo "Invalid EMSDK path"
exit 1
fi
if [ $MAX_THREADS -lt 2 ]; then
echo "MAX_THREAD must be greater or equal to 2" &&
exit 1
fi
if ! [[ $MAX_MEMORY =~ ^[0-9]+([kmgt]b)?$ ]]; then
echo "MAX_MEMORY valid suffixes are kb, mb, gb, tb, none (bytes)" &&
exit 1
fi
if [ $EMSDK = ../emsdk ]; then
echo "EMSDK is current directory, installing emsdk and Emscripten..." &&
git clone --depth=1 https://github.com/emscripten-core/emsdk.git &&
cd emsdk &&
./emsdk install 3.1.53 &&
./emsdk activate 3.1.53
fi
. $EMSDK/emsdk_env.sh &&
export PATH=:$PATH:$EMSDK/upstream/bin &&
cd $SRC &&
em++ -O0 global.cc genericModel.cc model.cc spkModel.cc recognizer.cc bindings.cc -sWASMFS -sWASM_BIGINT -sSINGLE_FILE -sMODULARIZE -sTRUSTED_TYPES -sEMBIND_STD_STRING_IS_UTF8 -sPTHREAD_POOL_DELAY_LOAD -sTEXTDECODER=2 -sPTHREAD_POOL_SIZE_STRICT=2 -sINITIAL_MEMORY=$MAX_MEMORY -sPTHREAD_POOL_SIZE=$MAX_THREADS -sPOLYFILL=0 -sSUPPORT_LONGJMP=0 -sEXPORTED_FUNCTIONS=_malloc,_main -sEXPORT_NAME=loadBR -sEXPORTED_RUNTIME_METHODS=UTF8ToString,stringToUTF8OnStack -sMALLOC=emmalloc -sENVIRONMENT=web,worker -I. -I$LIBARCHIVE/include -I$VOSK/src -L$LIBARCHIVE/lib -larchive -L$ZSTD/lib -lzstd -L$KALDI/src -l:online2/kaldi-online2.a -l:decoder/kaldi-decoder.a -l:ivector/kaldi-ivector.a -l:gmm/kaldi-gmm.a -l:tree/kaldi-tree.a -l:feat/kaldi-feat.a -l:cudamatrix/kaldi-cudamatrix.a -l:lat/kaldi-lat.a -l:lm/kaldi-lm.a -l:rnnlm/kaldi-rnnlm.a -l:hmm/kaldi-hmm.a -l:nnet3/kaldi-nnet3.a -l:transform/kaldi-transform.a -l:matrix/kaldi-matrix.a -l:fstext/kaldi-fstext.a -l:util/kaldi-util.a -l:base/kaldi-base.a -L$OPENFST/lib -l:libfst.a -l:libfstngram.a -L$CLAPACK_WASM -l:CBLAS/lib/cblas.a -l:CLAPACK-3.2.1/lapack.a -l:CLAPACK-3.2.1/libcblaswr.a -l:f2c_BLAS-3.8.0/blas.a -l:libf2c/libf2c.a -L$VOSK/src -l:vosk.a -lopfs.js -lembind -pthread -flto --pre-js pre.js -o ../test/BrowserRecognizer.js &&
cd ../test &&
rm -f BrowserRecognizer.worker.js &&
sed -i "s/locateFile('BrowserRecognizer.worker.js')/pthreadUrl/g" BrowserRecognizer.js &&
sed -i 's/locateFile("BrowserRecognizer.worker.js")/pthreadUrl/g' BrowserRecognizer.js