patrick/Vosklet
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Finally

Browse Source
This commit is contained in:
msqr1
2024-03-26 22:04:02 -07:00
parent df6b240bde
commit 7b595ec0ee
21 changed files with 328 additions and 6300 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
usage/README.md → API.md
View File

@@ -1,4 +1,4 @@
# API interface # API reference
## JS' ```window``` object ## JS' ```window``` object
| Function/Object | Description | | Function/Object | Description |
|---|---| |---|---|
@@ -12,16 +12,17 @@
## ```Module``` object ## ```Module``` object
| Function/Object | Description | | Function/Object | Description |
|---|---| |---|---|
| ```Promise<Model> makeModel(path: string, url: string, id: string)```<br><br>```Promise<SpkModel> makeSpkModel(path: string, url: string, id: string)``` | Make a ```Model``` or ```SpkModel```, model files must be directly under the model root, and compressed model must be in .tgz format. If:<br>- **path** contains valid model files and **id** is the same, there will not be a fetch from **url**.<br>- **path** doesn't contain valid model files, or if it contains valid model files but **id** is different, there will be a fetch from **url**, and the model is stored with **id**. Models are thread-safe, reuse them as much as possible! | | ```Promise<Model> createModel(path: string, url: string, id: string)```<br><br>```Promise<SpkModel> createSpkModel(path: string, url: string, id: string)``` | Create a ```Model``` or ```SpkModel```, model files must be directly under the model root, and compressed model must be in .tgz format. If:<br>- **path** contains valid model files and **id** is the same, there will not be a fetch from **url**.<br>- **path** doesn't contain valid model files, or if it contains valid model files but **id** is different, there will be a fetch from **url**, and the model is stored with **id**. Models are thread-safe, reuse them as much as possible! |
| ```Promise<Recognizer> makeRecognizer(model: Model, sampleRate: float)```<br><br>```Promise<Recognizer> makeRecognizerWithSpkModel(model: Model, spkModel: spkModel, sampleRate: float)```<br><br>```Promise<Recognizer> makeRecognizerWithGrm(model: Model, grammar: string, sampleRate: float)``` | Make a ```Recognizer```, it will use **model**'s thread if it's the first user of **model**, else it will use a new thread. | | ```Promise<Recognizer> createRecognizer(model: Model, sampleRate: float)```<br><br>```Promise<Recognizer> createRecognizerWithSpkModel(model: Model, spkModel: spkModel, sampleRate: float)```<br><br>```Promise<Recognizer> createRecognizerWithGrm(model: Model, grammar: string, sampleRate: float)``` | Create a ```Recognizer```, it will use **model**'s thread if it's the first user of **model**, else it will use a new thread. |
| ```setLogLevel(lvl: int)``` | Set Vosk's log level (default: ```0```: Info) <br>```-2```: Error<br>```-1```: Warning<br>```1```: Verbose<br>```2```: More verbose<br>```3```: Debug | | ```setLogLevel(lvl: int)``` | Set Vosk's log level (default: ```0```: Info) <br>```-2```: Error<br>```-1```: Warning<br>```1```: Verbose<br>```2```: More verbose<br>```3```: Debug |
| ```Promise<AudioWorkletNode> createTransferer(ctx: AudioContext)``` | Create a node that transfer its inputs back to the main thread. Its port's ```onmessage``` handler can be set to get audio data. Has 1 input with 1 channel and 0 output.
| ```cleanUp()``` | A convenience function that call ```delete()``` on all objects and revoke all URLs. **Put this at the end of your code!** | | ```cleanUp()``` | A convenience function that call ```delete()``` on all objects and revoke all URLs. **Put this at the end of your code!** |
| ```epMode``` | Endpointer modes (enum) | See Vosk's description | | ```epMode``` | Endpointer modes (enum) | See Vosk's description |
## ```Recognizer``` object ## ```Recognizer``` object
| Function/Object | Description | | Function/Object | Description |
|---|---| |---|---|
| ```acceptWaveform(buf: AudioBuffer || Float32Array )``` | Recognize an AudioBuffer, usually from something like ```BaseAudioContext.decodeAudioData()```, **channelIndex** must point to a 16-bit mono channel of **buf** | ```acceptWaveform(audioData: Float32Array)``` | Recognize audio data in a ```Float32Array``` with elements from -1.0 to 1.0. |
| ```setPartialWords(partialWords: bool)``` | See Vosk's description (default: false) | | ```setPartialWords(partialWords: bool)``` | See Vosk's description (default: false) |
| ```setWords(words: bool)``` | See Vosk's description (default: false) | | ```setWords(words: bool)``` | See Vosk's description (default: false) |
| ```setNLSML(nlsml: bool)``` | See Vosk's description (default: false) | | ```setNLSML(nlsml: bool)``` | See Vosk's description (default: false) |
@@ -38,7 +39,7 @@
# User agent notes # User agent notes
## SharedArrayBuffer ## SharedArrayBuffer
Vosklet require SharedArrayBuffer to share thread's data, so these response headers must be set: Vosklet require SharedArrayBuffer to share threads' data, so these response headers must be set:
- ***Cross-Origin-Embedder-Policy*** ---> ***require-corp*** - ***Cross-Origin-Embedder-Policy*** ---> ***require-corp***
- ***Cross-Origin-Opener-Policy*** ---> ***same-origin*** - ***Cross-Origin-Opener-Policy*** ---> ***same-origin***

6
README.md
View File

@@ -1,17 +1,19 @@
# Overview # Overview
- A speech recognizer built on Vosk that can be run on the browser, inspired by [vosk-browser](https://github.com/ccoreilly/vosk-browser), but built from scratch and no code taken! - A speech recognizer built on Vosk that can be run on the browser, inspired by [vosk-browser](https://github.com/ccoreilly/vosk-browser), but built from scratch and no code taken!
- Designed with basic/nothrow exception safety - Designed with basic/nothrow exception safety
- See the *usage* folder's README.md for API documentation, notes. - See the *examples* folder for examples on using the API
- See the *devel* folder for the newest build (not guaranteed to work) and the JS build script - See the *devel* folder for the newest build (not guaranteed to work) and the JS build script
# Additions to vosk-browser: # Compared to vosk-browser:
- Support multiple models - Support multiple models
- Model storage path management - Model storage path management
- Model ID management (for model updates) - Model ID management (for model updates)
- Smaller JS size (>3.1MB vs 1.4MB) - Smaller JS size (>3.1MB vs 1.4MB)
- All related files (pthread worker, worklet processor,...) are merged - All related files (pthread worker, worklet processor,...) are merged
- Shorter from-scratch build time - Shorter from-scratch build time
- Faster loading and processing time
# Basic usage # Basic usage
- Use the prebuilt binary through JsDelivr - Use the prebuilt binary through JsDelivr
- See

5915
devel/Vosklet.js
View File

File diff suppressed because one or more lines are too long

163
devel/Vosklet.worker.js
View File

@@ -1,163 +0,0 @@
/**
* @license
* Copyright 2015 The Emscripten Authors
* SPDX-License-Identifier: MIT
*/
// Pthread Web Worker startup routine:
// This is the entry point file that is loaded first by each Web Worker
// that executes pthreads on the Emscripten application.
'use strict';
var Module = {};
// Thread-local guard variable for one-time init of the JS state
var initializedJS = false;
function assert(condition, text) {
if (!condition) abort('Assertion failed: ' + text);
}
function threadPrintErr(...args) {
var text = args.join(' ');
console.error(text);
}
function threadAlert(...args) {
var text = args.join(' ');
postMessage({cmd: 'alert', text, threadId: Module['_pthread_self']()});
}
// We don't need out() for now, but may need to add it if we want to use it
// here. Or, if this code all moves into the main JS, that problem will go
// away. (For now, adding it here increases code size for no benefit.)
var out = () => { throw 'out() is not defined in worker.js.'; }
var err = threadPrintErr;
self.alert = threadAlert;
var dbg = threadPrintErr;
Module['instantiateWasm'] = (info, receiveInstance) => {
// Instantiate from the module posted from the main thread.
// We can just use sync instantiation in the worker.
var module = Module['wasmModule'];
// We don't need the module anymore; new threads will be spawned from the main thread.
Module['wasmModule'] = null;
var instance = new WebAssembly.Instance(module, info);
// TODO: Due to Closure regression https://github.com/google/closure-compiler/issues/3193,
// the above line no longer optimizes out down to the following line.
// When the regression is fixed, we can remove this if/else.
return receiveInstance(instance);
}
// Turn unhandled rejected promises into errors so that the main thread will be
// notified about them.
self.onunhandledrejection = (e) => {
throw e.reason || e;
};
function handleMessage(e) {
try {
if (e.data.cmd === 'load') { // Preload command that is called once per worker to parse and load the Emscripten code.
// Until we initialize the runtime, queue up any further incoming messages.
let messageQueue = [];
self.onmessage = (e) => messageQueue.push(e);
// And add a callback for when the runtime is initialized.
self.startWorker = (instance) => {
Module = instance;
// Notify the main thread that this thread has loaded.
postMessage({ 'cmd': 'loaded' });
// Process any messages that were queued before the thread was ready.
for (let msg of messageQueue) {
handleMessage(msg);
}
// Restore the real message handler.
self.onmessage = handleMessage;
};
// Module and memory were sent from main thread
Module['wasmModule'] = e.data.wasmModule;
// Use `const` here to ensure that the variable is scoped only to
// that iteration, allowing safe reference from a closure.
for (const handler of e.data.handlers) {
Module[handler] = (...args) => {
dbg(`calling handler on main thread: ${handler}`);
postMessage({ cmd: 'callHandler', handler, args: args });
}
}
Module['wasmMemory'] = e.data.wasmMemory;
Module['buffer'] = Module['wasmMemory'].buffer;
Module['workerID'] = e.data.workerID;
Module['ENVIRONMENT_IS_PTHREAD'] = true;
if (typeof e.data.urlOrBlob == 'string') {
importScripts(e.data.urlOrBlob);
} else {
var objectUrl = URL.createObjectURL(e.data.urlOrBlob);
importScripts(objectUrl);
URL.revokeObjectURL(objectUrl);
}
loadVosklet(Module);
} else if (e.data.cmd === 'run') {
// Pass the thread address to wasm to store it for fast access.
Module['__emscripten_thread_init'](e.data.pthread_ptr, /*is_main=*/0, /*is_runtime=*/0, /*can_block=*/1);
// Await mailbox notifications with `Atomics.waitAsync` so we can start
// using the fast `Atomics.notify` notification path.
Module['__emscripten_thread_mailbox_await'](e.data.pthread_ptr);
assert(e.data.pthread_ptr);
// Also call inside JS module to set up the stack frame for this pthread in JS module scope
Module['establishStackSpace']();
Module['PThread'].receiveObjectTransfer(e.data);
Module['PThread'].threadInitTLS();
if (!initializedJS) {
// Embind must initialize itself on all threads, as it generates support JS.
// We only do this once per worker since they get reused
Module['__embind_initialize_bindings']();
initializedJS = true;
}
try {
Module['invokeEntryPoint'](e.data.start_routine, e.data.arg);
} catch(ex) {
if (ex != 'unwind') {
// The pthread "crashed". Do not call `_emscripten_thread_exit` (which
// would make this thread joinable). Instead, re-throw the exception
// and let the top level handler propagate it back to the main thread.
throw ex;
}
dbg(`Pthread 0x${Module['_pthread_self']().toString(16)} completed its main entry point with an 'unwind', keeping the worker alive for asynchronous operation.`);
}
} else if (e.data.cmd === 'cancel') { // Main thread is asking for a pthread_cancel() on this thread.
if (Module['_pthread_self']()) {
Module['__emscripten_thread_exit'](-1);
}
} else if (e.data.target === 'setimmediate') {
// no-op
} else if (e.data.cmd === 'checkMailbox') {
if (initializedJS) {
Module['checkMailbox']();
}
} else if (e.data.cmd) {
// The received message looks like something that should be handled by this message
// handler, (since there is a e.data.cmd field present), but is not one of the
// recognized commands:
err(`worker.js received unknown command ${e.data.cmd}`);
err(e.data);
}
} catch(ex) {
err(`worker.js onmessage() captured an uncaught exception: ${ex}`);
if (ex?.stack) err(ex.stack);
Module['__emscripten_thread_crashed']?.();
throw ex;
}
};
self.onmessage = handleMessage;

49
devel/index.html
View File

@@ -3,20 +3,8 @@
<head> <head>
<script src="Vosklet.js"></script> <script src="Vosklet.js"></script>
<script> <script>
async function func() { async function start() {
let processorURL = URL.createObjectURL(new Blob(['(', let ctx = new AudioContext({sampleRate : 16000})
(() => {
registerProcessor('processor', class extends AudioWorkletProcessor {
process(inputs, outputs) {
for (let channel = 0; channel < outputs[0].length; ++channel) {
outputs[0][channel].set(inputs[0][channel])
}
return true
}
})
}).toString(),
')()'], { type : "text/javascript" }))
window.ctx = new AudioContext({sampleRate : 16000})
let micNode = ctx.createMediaStreamSource(await navigator.mediaDevices.getUserMedia({ let micNode = ctx.createMediaStreamSource(await navigator.mediaDevices.getUserMedia({
video: false, video: false,
audio: { audio: {
@@ -26,26 +14,23 @@
sampleRate: 16000 sampleRate: 16000
}, },
})) }))
await ctx.audioWorklet.addModule(processorURL) let module = await loadVosklet()
window.node = new AudioWorkletNode(ctx, "processor", { channelCountMode: "explicit", numberOfInputs : 1, numberOfOutputs : 1}) let model = await module.createModel("../usage/en-model.tgz","model","ID")
micNode.connect(node).connect(ctx.destination) let recognizer = await module.createRecognizer(model, 16000)
ctx.resume() recognizer.addEventListener("result", ev => {
} console.log("Result: ", ev.detail)
window.onload = async () => {
/*window.module = await loadVosklet()
window.model = await module.makeModel("../usage/en-model.tgz","model","ID")
window.recognizer = await module.makeRecognizer(model, 16000)
recognizer.addEventListener("result", e => {
console.log("Result: ", e.detail)
}) })
recognizer.addEventListener("partialResult", e => { recognizer.addEventListener("partialResult", ev => {
console.log("Partial result: ", e.detail) console.log("Partial result: ", ev.detail)
})*/ })
//let recNode = await recognizer.getNode(ctx) let transferer = await module.createTransferer(ctx)
//recNode.connect(ctx.destination) console.log(transferer)
//micNode.connect(recNode) transferer.port.onmessage = ev => {
recognizer.acceptWaveform(ev.data)
}
micNode.connect(transferer)
} }
</script> </script>
<button onclick="func()">Just another button</button> <button onclick="start()">Start</button>
</head> </head>
</html> </html>

7
devel/test.sh
View File

@@ -28,12 +28,13 @@ fi
. $EMSDK/emsdk_env.sh && . $EMSDK/emsdk_env.sh &&
cd $SRC && cd $SRC &&
MODE=2 && # 0: Ultra debug info, 1: Extremely optimized release, else custom MODE=1 && # 0: Ultra debug info, 1: Extremely optimized release, else custom
echo "Mode = $MODE" && echo "Mode = $MODE" &&
if [ $MODE = 0 ]; then if [ $MODE = 0 ]; then
em++ -O0 link.cc genericModel.cc recognizer.cc bindings.cc -sWASMFS -sWASM_BIGINT -sSINGLE_FILE -sMODULARIZE -sEMBIND_STD_STRING_IS_UTF8 -sPTHREAD_POOL_DELAY_LOAD -sRUNTIME_DEBUG -sSTACK_OVERFLOW_CHECK=2 -sTEXTDECODER=2 -sPTHREAD_POOL_SIZE_STRICT=2 -sASSERTIONS=2 -sINITIAL_MEMORY=$MAX_MEMORY -sPTHREAD_POOL_SIZE=$MAX_THREADS -sDISABLE_EXCEPTION_CATCHING=0 -sEXIT_RUNTIME=0 -sINVOKE_RUN=0 -sPOLYFILL=0 -sEXPORTED_FUNCTIONS=_malloc -sEXPORT_NAME=loadVosklet -sMALLOC=emmalloc -sEXPORTED_RUNTIME_METHODS=UTF8ToString,stringToUTF8OnStack -sENVIRONMENT=web,worker -I. -I$LIBARCHIVE/include -I$VOSK/src -L$LIBARCHIVE/lib -larchive -L$KALDI/src -l:online2/kaldi-online2.a -l:decoder/kaldi-decoder.a -l:ivector/kaldi-ivector.a -l:gmm/kaldi-gmm.a -l:tree/kaldi-tree.a -l:feat/kaldi-feat.a -l:cudamatrix/kaldi-cudamatrix.a -l:lat/kaldi-lat.a -l:lm/kaldi-lm.a -l:rnnlm/kaldi-rnnlm.a -l:hmm/kaldi-hmm.a -l:nnet3/kaldi-nnet3.a -l:transform/kaldi-transform.a -l:matrix/kaldi-matrix.a -l:fstext/kaldi-fstext.a -l:util/kaldi-util.a -l:base/kaldi-base.a -L$OPENFST/lib -l:libfst.a -l:libfstngram.a -L$CLAPACK_WASM -l:CBLAS/lib/cblas.a -l:CLAPACK-3.2.1/lapack.a -l:CLAPACK-3.2.1/libcblaswr.a -l:f2c_BLAS-3.8.0/blas.a -l:libf2c/libf2c.a -L$VOSK/src -l:vosk.a -lembind -pthread -flto -msimd128 --embind-emit-tsd Vosklet.d.ts -fsanitize=undefined -fsanitize=address -fsanitize=leak -msimd128 -mreference-types -mnontrapping-fptoint -mextended-const -msign-ext -g3 --pre-js pre.js -o ../devel/Vosklet.js em++ -O0 link.cc genericModel.cc recognizer.cc bindings.cc -sWASMFS -sWASM_BIGINT -sSINGLE_FILE -sMODULARIZE -sEMBIND_STD_STRING_IS_UTF8 -sPTHREAD_POOL_DELAY_LOAD -sRUNTIME_DEBUG -sSTACK_OVERFLOW_CHECK=2 -sTEXTDECODER=2 -sPTHREAD_POOL_SIZE_STRICT=2 -sASSERTIONS=2 -sINITIAL_MEMORY=$MAX_MEMORY -sPTHREAD_POOL_SIZE=$MAX_THREADS -sDISABLE_EXCEPTION_CATCHING=0 -sEXIT_RUNTIME=0 -sINVOKE_RUN=0 -sPOLYFILL=0 -sEXPORTED_FUNCTIONS=_malloc -sEXPORT_NAME=loadVosklet -sMALLOC=emmalloc -sEXPORTED_RUNTIME_METHODS=UTF8ToString,stringToUTF8OnStack -sENVIRONMENT=web,worker -I. -I$LIBARCHIVE/include -I$VOSK/src -L$LIBARCHIVE/lib -larchive -L$KALDI/src -l:online2/kaldi-online2.a -l:decoder/kaldi-decoder.a -l:ivector/kaldi-ivector.a -l:gmm/kaldi-gmm.a -l:tree/kaldi-tree.a -l:feat/kaldi-feat.a -l:cudamatrix/kaldi-cudamatrix.a -l:lat/kaldi-lat.a -l:lm/kaldi-lm.a -l:rnnlm/kaldi-rnnlm.a -l:hmm/kaldi-hmm.a -l:nnet3/kaldi-nnet3.a -l:transform/kaldi-transform.a -l:matrix/kaldi-matrix.a -l:fstext/kaldi-fstext.a -l:util/kaldi-util.a -l:base/kaldi-base.a -L$OPENFST/lib -l:libfst.a -l:libfstngram.a -L$CLAPACK_WASM -l:CBLAS/lib/cblas.a -l:CLAPACK-3.2.1/lapack.a -l:CLAPACK-3.2.1/libcblaswr.a -l:f2c_BLAS-3.8.0/blas.a -l:libf2c/libf2c.a -L$VOSK/src -l:vosk.a -lembind -pthread -flto -msimd128 --embind-emit-tsd Vosklet.d.ts -fsanitize=undefined -fsanitize=address -fsanitize=leak -msimd128 -mreference-types -mnontrapping-fptoint -mextended-const -msign-ext -g3 --pre-js pre.js -o ../devel/Vosklet.js
elif [ $MODE = 1 ]; then elif [ $MODE = 1 ]; then
em++ -O3 link.cc genericModel.cc recognizer.cc bindings.cc -sWASMFS -sWASM_BIGINT -sSINGLE_FILE -sMODULARIZE -sEMBIND_STD_STRING_IS_UTF8 -sPTHREAD_POOL_DELAY_LOAD -sTEXTDECODER=2 -sPTHREAD_POOL_SIZE_STRICT=2 -sINITIAL_MEMORY=$MAX_MEMORY -sPTHREAD_POOL_SIZE=$MAX_THREADS -sPOLYFILL=0 -sEXIT_RUNTIME=0 -sINVOKE_RUN=0 -sSUPPORT_LONGJMP=0 -sEXPORTED_FUNCTIONS=_malloc -sEXPORT_NAME=loadVosklet -sMALLOC=emmalloc -sEXPORTED_RUNTIME_METHODS=UTF8ToString,stringToUTF8OnStack -sENVIRONMENT=web,worker -I. -I$LIBARCHIVE/include -I$VOSK/src -L$LIBARCHIVE/lib -larchive -L$KALDI/src -l:online2/kaldi-online2.a -l:decoder/kaldi-decoder.a -l:ivector/kaldi-ivector.a -l:gmm/kaldi-gmm.a -l:tree/kaldi-tree.a -l:feat/kaldi-feat.a -l:cudamatrix/kaldi-cudamatrix.a -l:lat/kaldi-lat.a -l:lm/kaldi-lm.a -l:rnnlm/kaldi-rnnlm.a -l:hmm/kaldi-hmm.a -l:nnet3/kaldi-nnet3.a -l:transform/kaldi-transform.a -l:matrix/kaldi-matrix.a -l:fstext/kaldi-fstext.a -l:util/kaldi-util.a -l:base/kaldi-base.a -L$OPENFST/lib -l:libfst.a -l:libfstngram.a -L$CLAPACK_WASM -l:CBLAS/lib/cblas.a -l:CLAPACK-3.2.1/lapack.a -l:CLAPACK-3.2.1/libcblaswr.a -l:f2c_BLAS-3.8.0/blas.a -l:libf2c/libf2c.a -L$VOSK/src -l:vosk.a -lembind -pthread -flto -msimd128 -mreference-types -mnontrapping-fptoint -mextended-const -msign-ext --pre-js pre.js -o ../devel/Vosklet.js em++ -O3 link.cc genericModel.cc recognizer.cc bindings.cc -sWASMFS -sWASM_BIGINT -sSINGLE_FILE -sMODULARIZE -sEMBIND_STD_STRING_IS_UTF8 -sPTHREAD_POOL_DELAY_LOAD -sTEXTDECODER=2 -sPTHREAD_POOL_SIZE_STRICT=2 -sINITIAL_MEMORY=$MAX_MEMORY -sPTHREAD_POOL_SIZE=$MAX_THREADS -sPOLYFILL=0 -sEXIT_RUNTIME=0 -sINVOKE_RUN=0 -sSUPPORT_LONGJMP=0 -sEXPORTED_FUNCTIONS=_malloc -sEXPORT_NAME=loadVosklet -sMALLOC=emmalloc -sEXPORTED_RUNTIME_METHODS=UTF8ToString,stringToUTF8OnStack -sENVIRONMENT=web,worker -I. -I$LIBARCHIVE/include -I$VOSK/src -L$LIBARCHIVE/lib -larchive -L$KALDI/src -l:online2/kaldi-online2.a -l:decoder/kaldi-decoder.a -l:ivector/kaldi-ivector.a -l:gmm/kaldi-gmm.a -l:tree/kaldi-tree.a -l:feat/kaldi-feat.a -l:cudamatrix/kaldi-cudamatrix.a -l:lat/kaldi-lat.a -l:lm/kaldi-lm.a -l:rnnlm/kaldi-rnnlm.a -l:hmm/kaldi-hmm.a -l:nnet3/kaldi-nnet3.a -l:transform/kaldi-transform.a -l:matrix/kaldi-matrix.a -l:fstext/kaldi-fstext.a -l:util/kaldi-util.a -l:base/kaldi-base.a -L$OPENFST/lib -l:libfst.a -l:libfstngram.a -L$CLAPACK_WASM -l:CBLAS/lib/cblas.a -l:CLAPACK-3.2.1/lapack.a -l:CLAPACK-3.2.1/libcblaswr.a -l:f2c_BLAS-3.8.0/blas.a -l:libf2c/libf2c.a -L$VOSK/src -l:vosk.a -lembind -pthread -flto -msimd128 -mreference-types -mnontrapping-fptoint -mextended-const -msign-ext --pre-js pre.js -o ../devel/Vosklet.js
else else
em++ -O0 link.cc genericModel.cc recognizer.cc bindings.cc -sWASMFS -sWASM_BIGINT -sSINGLE_FILE -sMODULARIZE -sEMBIND_STD_STRING_IS_UTF8 -sPTHREAD_POOL_DELAY_LOAD -sRUNTIME_DEBUG -sSTACK_OVERFLOW_CHECK=2 -sTEXTDECODER=2 -sPTHREAD_POOL_SIZE_STRICT=2 -sASSERTIONS=2 -sINITIAL_MEMORY=$MAX_MEMORY -sPTHREAD_POOL_SIZE=$MAX_THREADS -sDISABLE_EXCEPTION_CATCHING=0 -sEXIT_RUNTIME=0 -sINVOKE_RUN=0 -sPOLYFILL=0 -sEXPORTED_FUNCTIONS=_malloc -sEXPORT_NAME=loadVosklet -sMALLOC=emmalloc -sEXPORTED_RUNTIME_METHODS=UTF8ToString,stringToUTF8OnStack -sENVIRONMENT=web,worker -I. -I$LIBARCHIVE/include -I$VOSK/src -L$LIBARCHIVE/lib -larchive -L$KALDI/src -l:online2/kaldi-online2.a -l:decoder/kaldi-decoder.a -l:ivector/kaldi-ivector.a -l:gmm/kaldi-gmm.a -l:tree/kaldi-tree.a -l:feat/kaldi-feat.a -l:cudamatrix/kaldi-cudamatrix.a -l:lat/kaldi-lat.a -l:lm/kaldi-lm.a -l:rnnlm/kaldi-rnnlm.a -l:hmm/kaldi-hmm.a -l:nnet3/kaldi-nnet3.a -l:transform/kaldi-transform.a -l:matrix/kaldi-matrix.a -l:fstext/kaldi-fstext.a -l:util/kaldi-util.a -l:base/kaldi-base.a -L$OPENFST/lib -l:libfst.a -l:libfstngram.a -L$CLAPACK_WASM -l:CBLAS/lib/cblas.a -l:CLAPACK-3.2.1/lapack.a -l:CLAPACK-3.2.1/libcblaswr.a -l:f2c_BLAS-3.8.0/blas.a -l:libf2c/libf2c.a -L$VOSK/src -l:vosk.a -lembind -pthread -flto -msimd128 -msimd128 -mreference-types -mnontrapping-fptoint -mextended-const -msign-ext -g3 --pre-js pre.js -o ../devel/Vosklet.js em++ -O0 link.cc genericModel.cc recognizer.cc bindings.cc -sWASMFS -sWASM_BIGINT -sSINGLE_FILE -sMODULARIZE -sEMBIND_STD_STRING_IS_UTF8 -sPTHREAD_POOL_DELAY_LOAD -sTEXTDECODER=2 -sPTHREAD_POOL_SIZE_STRICT=2 -sINITIAL_MEMORY=$MAX_MEMORY -sPTHREAD_POOL_SIZE=$MAX_THREADS -sPOLYFILL=0 -sEXIT_RUNTIME=0 -sINVOKE_RUN=0 -sSUPPORT_LONGJMP=0 -sEXPORTED_FUNCTIONS=_malloc -sEXPORT_NAME=loadVosklet -sMALLOC=emmalloc -sEXPORTED_RUNTIME_METHODS=UTF8ToString,stringToUTF8OnStack -sENVIRONMENT=web,worker -I. -I$LIBARCHIVE/include -I$VOSK/src -L$LIBARCHIVE/lib -larchive -L$KALDI/src -l:online2/kaldi-online2.a -l:decoder/kaldi-decoder.a -l:ivector/kaldi-ivector.a -l:gmm/kaldi-gmm.a -l:tree/kaldi-tree.a -l:feat/kaldi-feat.a -l:cudamatrix/kaldi-cudamatrix.a -l:lat/kaldi-lat.a -l:lm/kaldi-lm.a -l:rnnlm/kaldi-rnnlm.a -l:hmm/kaldi-hmm.a -l:nnet3/kaldi-nnet3.a -l:transform/kaldi-transform.a -l:matrix/kaldi-matrix.a -l:fstext/kaldi-fstext.a -l:util/kaldi-util.a -l:base/kaldi-base.a -L$OPENFST/lib -l:libfst.a -l:libfstngram.a -L$CLAPACK_WASM -l:CBLAS/lib/cblas.a -l:CLAPACK-3.2.1/lapack.a -l:CLAPACK-3.2.1/libcblaswr.a -l:f2c_BLAS-3.8.0/blas.a -l:libf2c/libf2c.a -L$VOSK/src -l:vosk.a -lembind -pthread -flto -msimd128 -mreference-types -mnontrapping-fptoint -mextended-const -msign-ext --pre-js pre.js -o ../devel/Vosklet.js
fi fi
rm -f ../devel/Vosklet.worker.js

0
usage/en-model.tgz → examples/en-model.tgz
View File

0
usage/example.wav → examples/example.wav
View File

35
examples/fromMic.html Normal file
View File

@@ -0,0 +1,35 @@
<!DOCTYPE html>
<html>
<head>
<script src="Vosklet.js" async defer></script>
<script>
async function start() {
let ctx = new AudioContext({sampleRate : 16000})
let micNode = ctx.createMediaStreamSource(await navigator.mediaDevices.getUserMedia({
video: false,
audio: {
echoCancellation: true,
noiseSuppression: true,
channelCount: 1,
sampleRate: 16000
},
}))
let module = await loadVosklet()
let model = await module.createModel("../usage/en-model.tgz","model","ID")
let recognizer = await module.createRecognizer(model, 16000)
recognizer.addEventListener("result", ev => {
console.log("Result: ", ev.detail)
})
recognizer.addEventListener("partialResult", ev => {
console.log("Partial result: ", ev.detail)
})
let transferer = await module.createTransferer(ctx)
transferer.port.onmessage = ev => {
recognizer.acceptWaveform(ev.data)
}
micNode.connect(transferer)
}
</script>
<button onclick="start()">Start</button>
</head>
</html>

23
examples/fromWav.html Normal file
View File

@@ -0,0 +1,23 @@
<!DOCTYPE html>
<html>
<head>
<script src="../Vosklet.js" async defer></script>
<script>
window.onload = async () => {
let ctx = new AudioContext({sampleRate : 16000})
let module = await loadVosklet()
let model = await module.createModel("en-model.tgz","model","ID")
let recognizer = await module.createRecognizer(model, 16000)
recognizer.addEventListener("result", ev => {
console.log("Result: ", ev.detail)
})
recognizer.addEventListener("partialResult", ev => {
console.log("Partial result: ", ev.detail)
})
let wav = await fetch("example.wav")
let audioBuf = await ctx.decodeAudioData(await wav.arrayBuffer())
recognizer.acceptWaveform(audioBuf.getChannelData(0))
}
</script>
</head>
</html>

0
usage/withSpkModel.html → examples/withSpkModel.html
View File

12
src/Makefile
View File

@@ -20,7 +20,7 @@ CLAPACK_WASM:=$(realpath clapack-wasm)
Vosklet.js: | vosk libarchive Vosklet.js: | vosk libarchive
cd $(SRC) && \ cd $(SRC) && \
em++ -O3 link.cc genericModel.cc recognizer.cc bindings.cc -sWASMFS \ -sWASM_BIGINT -sSINGLE_FILE -sGZIP_EMBEDDINGS -sMODULARIZE -sEMBIND_STD_STRING_IS_UTF8 -sPTHREAD_POOL_DELAY_LOAD -sTEXTDECODER=2 -sPTHREAD_POOL_SIZE_STRICT=2 -sINITIAL_MEMORY=$(MAX_MEMORY) -sPTHREAD_POOL_SIZE=$(MAX_THREADS) -sPOLYFILL=0 -sEXIT_RUNTIME=0, -sINVOKE_RUN=0 -sSUPPORT_LONGJMP=0 -sEXPORTED_FUNCTIONS=_malloc -sEXPORT_NAME=loadVosklet -sMALLOC=emmalloc -sEXPORTED_RUNTIME_METHODS=UTF8ToString,stringToUTF8OnStack -sENVIRONMENT=web,worker -I. -I$(LIBARCHIVE)/include -I$(VOSK)/src -L$(LIBARCHIVE)/lib -larchive -L$(KALDI)/src -l:online2/kaldi-online2.a -l:decoder/kaldi-decoder.a -l:ivector/kaldi-ivector.a -l:gmm/kaldi-gmm.a -l:tree/kaldi-tree.a -l:feat/kaldi-feat.a -l:cudamatrix/kaldi-cudamatrix.a -l:lat/kaldi-lat.a -l:lm/kaldi-lm.a -l:rnnlm/kaldi-rnnlm.a -l:hmm/kaldi-hmm.a -l:nnet3/kaldi-nnet3.a -l:transform/kaldi-transform.a -l:matrix/kaldi-matrix.a -l:fstext/kaldi-fstext.a -l:util/kaldi-util.a -l:base/kaldi-base.a -L$(OPENFST)/lib -l:libfst.a -l:libfstngram.a -L$(CLAPACK_WASM) -l:CBLAS/lib/cblas.a -l:CLAPACK-3.2.1/lapack.a -l:CLAPACK-3.2.1/libcblaswr.a -l:f2c_BLAS-3.8.0/blas.a -l:libf2c/libf2c.a -L$(VOSK)/src -l:vosk.a -lembind -pthread -flto -msimd128 --pre-js pre.js -o ../Vosklet.js && \ em++ -O3 link.cc genericModel.cc recognizer.cc bindings.cc -sWASMFS -sWASM_BIGINT -sSINGLE_FILE -sMODULARIZE -sEMBIND_STD_STRING_IS_UTF8 -sPTHREAD_POOL_DELAY_LOAD -sTEXTDECODER=2 -sPTHREAD_POOL_SIZE_STRICT=2 -sINITIAL_MEMORY=$(MAX_MEMORY) -sPTHREAD_POOL_SIZE=$(MAX_THREADS) -sPOLYFILL=0 -sEXIT_RUNTIME=0 -sINVOKE_RUN=0 -sSUPPORT_LONGJMP=0 -sEXPORTED_FUNCTIONS=_malloc -sEXPORT_NAME=loadVosklet -sMALLOC=emmalloc -sEXPORTED_RUNTIME_METHODS=UTF8ToString,stringToUTF8OnStack -sENVIRONMENT=web,worker -I. -I$(LIBARCHIVE)/include -I$(VOSK)/src -L$(LIBARCHIVE)/lib -larchive -L$(KALDI)/src -l:online2/kaldi-online2.a -l:decoder/kaldi-decoder.a -l:ivector/kaldi-ivector.a -l:gmm/kaldi-gmm.a -l:tree/kaldi-tree.a -l:feat/kaldi-feat.a -l:cudamatrix/kaldi-cudamatrix.a -l:lat/kaldi-lat.a -l:lm/kaldi-lm.a -l:rnnlm/kaldi-rnnlm.a -l:hmm/kaldi-hmm.a -l:nnet3/kaldi-nnet3.a -l:transform/kaldi-transform.a -l:matrix/kaldi-matrix.a -l:fstext/kaldi-fstext.a -l:util/kaldi-util.a -l:base/kaldi-base.a -L$(OPENFST)/lib -l:libfst.a -l:libfstngram.a -L$(CLAPACK_WASM) -l:CBLAS/lib/cblas.a -l:CLAPACK-3.2.1/lapack.a -l:CLAPACK-3.2.1/libcblaswr.a -l:f2c_BLAS-3.8.0/blas.a -l:libf2c/libf2c.a -L$(VOSK)/src -l:vosk.a -lembind -pthread -flto -msimd128 -mreference-types -mnontrapping-fptoint -mextended-const -msign-ext --pre-js pre.js -o ../Vosklet.js && \
cd .. && \ cd .. && \
rm -f Vosklet.worker.js && \ rm -f Vosklet.worker.js && \
@@ -68,24 +68,24 @@ openfst: prepare
git clone --depth=1 https://github.com/alphacep/openfst /tmp/openfst && \ git clone --depth=1 https://github.com/alphacep/openfst /tmp/openfst && \
cd /tmp/openfst && \ cd /tmp/openfst && \
autoreconf -i && \ autoreconf -i && \
CXXFLAGS="-pthread -r -O3 -flto -msimd128" LDFLAGS="-O3 -pthread -flto" emconfigure ./configure --prefix=$(OPENFST) --enable-static --disable-shared --enable-ngram-fsts --enable-lookahead-fsts --disable-bin --with-pic && \ CXXFLAGS="-r -O3 -flto -msimd128" LDFLAGS="-O3 -flto" emconfigure ./configure --prefix=$(OPENFST) --enable-static --disable-shared --enable-lookahead-fsts --enable-ngram-fsts --disable-bin && \
emmake make -j$(COMPILE_JOBS) install && \ emmake make -j$(COMPILE_JOBS) install && \
echo "PACKAGE_VERSION = 1.8.0" >> $(OPENFST)/Makefile && \ echo "PACKAGE_VERSION = 1.8.0" >> $(OPENFST)/Makefile && \
rm -rf /tmp/openfst rm -rf /tmp/openfst
kaldi: | openfst clapack-wasm kaldi: | openfst clapack-wasm
git clone -b vosk --depth=1 https://github.com/alphacep/kaldi $(KALDI) && \ git clone -b vosk --depth=1 https://github.com/alphacep/kaldi && \
cd $(KALDI)/src && \ cd $(KALDI)/src && \
git apply $(SRC)/kaldi.patch && \ git apply $(SRC)/kaldi.patch && \
CXXFLAGS="-O3 -UHAVE_EXECINFO_H -pthread -flto -msimd128" LDFLAGS="-O3 -sERROR_ON_UNDEFINED_SYMBOLS=0 -lembind -pthread -flto" emconfigure ./configure --use-cuda=no --with-cudadecoder=no --static --static-math=yes --static-fst=yes --debug-level=0 --double-precision=yes --fst-root=$(OPENFST) --clapack-root=$(CLAPACK_WASM) --host=WASM && \ CXXFLAGS="-O3 -UHAVE_EXECINFO_H -flto -msimd128 -Wno-unused-variable -Wno-unused-but-set-variable -g0" LDFLAGS="-O3 -lembind -flto -g0" emconfigure ./configure --use-cuda=no --with-cudadecoder=no --static --static-math=yes --static-fst=yes --debug-level=0 --fst-root=$(OPENFST) --clapack-root=$(CLAPACK_WASM) --host=WASM && \
emmake make -j$(COMPILE_JOBS) online2 lm rnnlm emmake make -j$(COMPILE_JOBS) online2 rnnlm
vosk: | kaldi vosk: | kaldi
git clone -b go/v0.3.46 --depth=1 https://github.com/alphacep/vosk-api $(VOSK) && \ git clone -b go/v0.3.46 --depth=1 https://github.com/alphacep/vosk-api $(VOSK) && \
cd $(VOSK)/src && \ cd $(VOSK)/src && \
git apply $(SRC)/vosk.patch && \ git apply $(SRC)/vosk.patch && \
VOSK_FILES="recognizer.cc language_model.cc model.cc spk_model.cc vosk_api.cc" && \ VOSK_FILES="recognizer.cc language_model.cc model.cc spk_model.cc vosk_api.cc" && \
em++ -pthread -O3 -flto -msimd128 -Wno-deprecated -I. -I$(KALDI)/src -I$(OPENFST)/include $(VOSK_FILES) -c && \ em++ -O3 -flto -msimd128 -Wno-deprecated -I. -I$(KALDI)/src -I$(OPENFST)/include $(VOSK_FILES) -c && \
emar -rcs vosk.a $(VOSK_FILES:.cc=.o) emar -rcs vosk.a $(VOSK_FILES:.cc=.o)
.PHONY: prepare .PHONY: prepare

2
src/bindings.cc
View File

@@ -21,7 +21,7 @@ EMSCRIPTEN_BINDINGS() {
.constructor<int, float, genericModel*>(allow_raw_pointers()) .constructor<int, float, genericModel*>(allow_raw_pointers())
.constructor<int, float, genericModel*, genericModel*>(allow_raw_pointers()) .constructor<int, float, genericModel*, genericModel*>(allow_raw_pointers())
.constructor<int, float, genericModel*, std::string, int>(allow_raw_pointers()) .constructor<int, float, genericModel*, std::string, int>(allow_raw_pointers())
.function("acceptWaveform", &recognizer::acceptWaveform, allow_raw_pointers()) .function("pushData", &recognizer::pushData, allow_raw_pointers())
.function("reset", &recognizer::reset, allow_raw_pointers()) .function("reset", &recognizer::reset, allow_raw_pointers())
.function("setEndpointerMode", &recognizer::setEndpointerMode, allow_raw_pointers()) .function("setEndpointerMode", &recognizer::setEndpointerMode, allow_raw_pointers())
.function("setEndpointerDelays", &recognizer::setEndpointerDelays, allow_raw_pointers()) .function("setEndpointerDelays", &recognizer::setEndpointerDelays, allow_raw_pointers())

11
src/genericModel.cc
View File

@@ -1,10 +1,9 @@
#include "genericModel.h" #include "genericModel.h"
genericModel::genericModel(int index, bool normalMdl, std::string storepath, std::string id) : index{index}, normalMdl{normalMdl}, storepath{std::move(storepath)}, id{std::move(id)}, entry{archive_entry_new()} { genericModel::genericModel(int index, bool normalMdl, std::string storepath, std::string id) : index{index}, normalMdl{normalMdl}, storepath{std::move(storepath)}, id{std::move(id)}, entry{archive_entry_new()} {
blocker.lock(); blocker.acquire();
} }
void genericModel::extractAndLoad(int tarStart, int tarSize) { void genericModel::extractAndLoad(int tarStart, int tarSize) {
emscripten_console_log("Untaring");
static fs::path path{}; static fs::path path{};
static int fd{}; static int fd{};
func = [this, tarStart, tarSize](){ func = [this, tarStart, tarSize](){
@@ -12,7 +11,6 @@ void genericModel::extractAndLoad(int tarStart, int tarSize) {
archive_read_support_format_tar(src); archive_read_support_format_tar(src);
archive_read_open_memory(src, reinterpret_cast<void*>(tarStart), tarSize); archive_read_open_memory(src, reinterpret_cast<void*>(tarStart), tarSize);
if(archive_errno(src) != 0) { if(archive_errno(src) != 0) {
emscripten_console_logf("Unable to open tar in WASM memory: %s", archive_error_string(src));
free(reinterpret_cast<void*>(tarStart)); free(reinterpret_cast<void*>(tarStart));
fireEv(index, "Unable to open tar in WASM memory"); fireEv(index, "Unable to open tar in WASM memory");
return; return;
@@ -32,14 +30,12 @@ void genericModel::extractAndLoad(int tarStart, int tarSize) {
} }
fd = creat(path.c_str(),0777); fd = creat(path.c_str(),0777);
if(fd == -1) { if(fd == -1) {
emscripten_console_log("Unable to create model files");
fireEv(index, "Unable to create model files"); fireEv(index, "Unable to create model files");
return; return;
} }
archive_read_data_into_fd(src, fd); archive_read_data_into_fd(src, fd);
close(fd); close(fd);
if(archive_errno(src) != 0) { if(archive_errno(src) != 0) {
emscripten_console_logf("Cannot write into model files: %s", archive_error_string(src));
fireEv(index, "Cannot write into model files"); fireEv(index, "Cannot write into model files");
return; return;
} }
@@ -49,14 +45,13 @@ void genericModel::extractAndLoad(int tarStart, int tarSize) {
archive_read_free(src); archive_read_free(src);
if(normalMdl) mdl = vosk_model_new(storepath.c_str()); if(normalMdl) mdl = vosk_model_new(storepath.c_str());
else vosk_spk_model_new(storepath.c_str()); else vosk_spk_model_new(storepath.c_str());
emscripten_console_log("Model loaded!");
if(normalMdl ? std::get<0>(mdl) == nullptr : std::get<1>(mdl) == nullptr) fireEv(index, "Unable to load model for recognition"); if(normalMdl ? std::get<0>(mdl) == nullptr : std::get<1>(mdl) == nullptr) fireEv(index, "Unable to load model for recognition");
else fireEv(index, "0"); else fireEv(index, "0");
}; };
std::thread t{[this](){ std::thread t{[this](){
func(); func();
blocker.lock(); blocker.acquire();
blocker.unlock(); blocker.release();
func(); func();
}}; }};
t.detach(); t.detach();

3
src/genericModel.h
View File

@@ -5,6 +5,7 @@
#include <filesystem> #include <filesystem>
#include <variant> #include <variant>
#include <thread> #include <thread>
#include <semaphore>
#include <fcntl.h> #include <fcntl.h>
#include <emscripten/console.h> #include <emscripten/console.h>
@@ -20,8 +21,8 @@ struct genericModel {
std::string storepath; std::string storepath;
std::string id; std::string id;
std::variant<VoskModel*, VoskSpkModel*> mdl; std::variant<VoskModel*, VoskSpkModel*> mdl;
std::binary_semaphore blocker{1};
std::function<void()> func; std::function<void()> func;
std::mutex blocker{};
archive_entry* entry; archive_entry* entry;
void extractAndLoad(int tarStart, int tarSize); void extractAndLoad(int tarStart, int tarSize);
genericModel(int index, bool normalMdl, std::string storepath, std::string id); genericModel(int index, bool normalMdl, std::string storepath, std::string id);

13
src/kaldi.patch
View File

@@ -39,15 +39,4 @@ index c3a122281..71d37256d 100644
TaskSequencer<IvectorExtractorComputeDerivedVarsClass> sequencer( TaskSequencer<IvectorExtractorComputeDerivedVarsClass> sequencer(
sequencer_opts); sequencer_opts);
for (int32 i = 0; i < NumGauss(); i++) for (int32 i = 0; i < NumGauss(); i++)
diff --git a/src/base/kaldi-types.h b/src/base/kaldi-types.h
index 7ebf4f8..2f5979e 100644
--- a/src/base/kaldi-types.h
+++ b/src/base/kaldi-types.h
@@ -20,6 +20,7 @@
#ifndef KALDI_BASE_KALDI_TYPES_H_
#define KALDI_BASE_KALDI_TYPES_H_ 1
+#define KALDI_DOUBLEPRECISION 1
namespace kaldi {
// TYPEDEFS ..................................................................

294
src/pre.js
View File

@@ -1,12 +1,193 @@
let objs = [] let objs = []
let processorURL = URL.createObjectURL(new Blob(['(', (() => {
registerProcessor("VoskletTransferer", class extends AudioWorkletProcessor {
process(inputs) {
this.port.postMessage(inputs[0][0].buffer, [inputs[0][0].buffer])
return true
}
})
}).toString(), ')()'], { type : "text/javascript" }))
let pthreadURL = URL.createObjectURL(new Blob(['(', (() => {
/**
* @license
* Copyright 2015 The Emscripten Authors
* SPDX-License-Identifier: MIT
*/
// Pthread Web Worker startup routine:
// This is the entry point file that is loaded first by each Web Worker
// that executes pthreads on the Emscripten application.
'use strict';
var Module = {};
// Thread-local guard variable for one-time init of the JS state
var initializedJS = false;
function assert(condition, text) {
if (!condition) abort('Assertion failed: ' + text);
}
function threadPrintErr(...args) {
var text = args.join(' ');
console.error(text);
}
function threadAlert(...args) {
var text = args.join(' ');
postMessage({cmd: 'alert', text, threadId: Module['_pthread_self']()});
}
// We don't need out() for now, but may need to add it if we want to use it
// here. Or, if this code all moves into the main JS, that problem will go
// away. (For now, adding it here increases code size for no benefit.)
var out = () => { throw 'out() is not defined in worker.js.'; }
var err = threadPrintErr;
self.alert = threadAlert;
var dbg = threadPrintErr;
Module['instantiateWasm'] = (info, receiveInstance) => {
// Instantiate from the module posted from the main thread.
// We can just use sync instantiation in the worker.
var module = Module['wasmModule'];
// We don't need the module anymore; new threads will be spawned from the main thread.
Module['wasmModule'] = null;
var instance = new WebAssembly.Instance(module, info);
// TODO: Due to Closure regression https://github.com/google/closure-compiler/issues/3193,
// the above line no longer optimizes out down to the following line.
// When the regression is fixed, we can remove this if/else.
return receiveInstance(instance);
}
// Turn unhandled rejected promises into errors so that the main thread will be
// notified about them.
self.onunhandledrejection = (e) => {
throw e.reason || e;
};
function handleMessage(e) {
try {
if (e.data.cmd === 'load') { // Preload command that is called once per worker to parse and load the Emscripten code.
// Until we initialize the runtime, queue up any further incoming messages.
let messageQueue = [];
self.onmessage = (e) => messageQueue.push(e);
// And add a callback for when the runtime is initialized.
self.startWorker = (instance) => {
Module = instance;
// Notify the main thread that this thread has loaded.
postMessage({ 'cmd': 'loaded' });
// Process any messages that were queued before the thread was ready.
for (let msg of messageQueue) {
handleMessage(msg);
}
// Restore the real message handler.
self.onmessage = handleMessage;
};
// Module and memory were sent from main thread
Module['wasmModule'] = e.data.wasmModule;
// Use `const` here to ensure that the variable is scoped only to
// that iteration, allowing safe reference from a closure.
for (const handler of e.data.handlers) {
Module[handler] = (...args) => {
postMessage({ cmd: 'callHandler', handler, args: args });
}
}
Module['wasmMemory'] = e.data.wasmMemory;
Module['buffer'] = Module['wasmMemory'].buffer;
Module['workerID'] = e.data.workerID;
Module['ENVIRONMENT_IS_PTHREAD'] = true;
if (typeof e.data.urlOrBlob == 'string') {
importScripts(e.data.urlOrBlob);
} else {
var objectUrl = URL.createObjectURL(e.data.urlOrBlob);
importScripts(objectUrl);
URL.revokeObjectURL(objectUrl);
}
loadVosklet(Module);
} else if (e.data.cmd === 'run') {
// Pass the thread address to wasm to store it for fast access.
Module['__emscripten_thread_init'](e.data.pthread_ptr, /*is_main=*/0, /*is_runtime=*/0, /*can_block=*/1);
// Await mailbox notifications with `Atomics.waitAsync` so we can start
// using the fast `Atomics.notify` notification path.
Module['__emscripten_thread_mailbox_await'](e.data.pthread_ptr);
assert(e.data.pthread_ptr);
// Also call inside JS module to set up the stack frame for this pthread in JS module scope
Module['establishStackSpace']();
Module['PThread'].receiveObjectTransfer(e.data);
Module['PThread'].threadInitTLS();
if (!initializedJS) {
// Embind must initialize itself on all threads, as it generates support JS.
// We only do this once per worker since they get reused
Module['__embind_initialize_bindings']();
initializedJS = true;
}
try {
Module['invokeEntryPoint'](e.data.start_routine, e.data.arg);
} catch(ex) {
if (ex != 'unwind') {
// The pthread "crashed". Do not call `_emscripten_thread_exit` (which
// would make this thread joinable). Instead, re-throw the exception
// and let the top level handler propagate it back to the main thread.
throw ex;
}
}
} else if (e.data.cmd === 'cancel') { // Main thread is asking for a pthread_cancel() on this thread.
if (Module['_pthread_self']()) {
Module['__emscripten_thread_exit'](-1);
}
} else if (e.data.target === 'setimmediate') {
// no-op
} else if (e.data.cmd === 'checkMailbox') {
if (initializedJS) {
Module['checkMailbox']();
}
} else if (e.data.cmd) {
// The received message looks like something that should be handled by this message
// handler, (since there is a e.data.cmd field present), but is not one of the
// recognized commands:
err(`worker.js received unknown command ${e.data.cmd}`);
err(e.data);
}
} catch(ex) {
err(`worker.js onmessage() captured an uncaught exception: ${ex}`);
if (ex?.stack) err(ex.stack);
Module['__emscripten_thread_crashed']?.();
throw ex;
}
};
self.onmessage = handleMessage;
}).toString(), ')()'], { type : "text/javascript" }))
Module.cleanUp = () => { Module.cleanUp = () => {
objs.forEach(obj => obj.delete()) objs.forEach(obj => obj.obj.delete())
URL.revokeObjectURL(pthreadUrl) URL.revokeObjectURL(pthreadURL)
URL.revokeObjectURL(processorUrl) URL.revokeObjectURL(processorURL)
}
Module.createTransferer = async (ctx) => {
await ctx.audioWorklet.addModule(processorURL)
return new AudioWorkletNode(ctx, "VoskletTransferer", {
channelCountMode : "explicit",
numberOfInputs : 1,
numberOfOutputs : 0,
channelCount : 1
})
} }
Module.locateFile = (path, scriptDir) => { Module.locateFile = (path, scriptDir) => {
if(path === "Vosklet.js") return pthreadUrl if(path === "Vosklet.worker.js") return pthreadURL
return scriptDir+path return scriptDir + path
} }
async function getFileHandle(path, create = false) { async function getFileHandle(path, create = false) {
let components = path.split("/") let components = path.split("/")
@@ -17,16 +198,12 @@ async function getFileHandle(path, create = false) {
return prevDir.getFileHandle(components[components.length - 1], { create : create }) return prevDir.getFileHandle(components[components.length - 1], { create : create })
} }
class genericModel extends EventTarget { class genericModel extends EventTarget {
constructor(url, storepath, id, normalMdl) { constructor() {
super() super()
objs.push(this) objs.push(this)
this.url = url
this.storepath = storepath
this.id = id
this.normalMdl = normalMdl
} }
static async _init(url, storepath, id, normalMdl) { static async create(url, storepath, id, normalMdl) {
let mdl = new genericModel(url, storepath, id, normalMdl) let mdl = new genericModel()
let result = new Promise((resolve, reject) => { let result = new Promise((resolve, reject) => {
mdl.addEventListener("0", ev => { mdl.addEventListener("0", ev => {
if(ev.detail === "0") return resolve(mdl) if(ev.detail === "0") return resolve(mdl)
@@ -37,9 +214,7 @@ class genericModel extends EventTarget {
let tar let tar
mdl.obj = new Module.genericModel(objs.length - 1, normalMdl, "/" + storepath, id) mdl.obj = new Module.genericModel(objs.length - 1, normalMdl, "/" + storepath, id)
try { try {
console.log("Getting Data file")
let dataFile = await (await getFileHandle(storepath + "/model.tgz")).getFile() let dataFile = await (await getFileHandle(storepath + "/model.tgz")).getFile()
console.log("Getting ID file")
let idFile = await (await getFileHandle(storepath + "/id")).getFile() let idFile = await (await getFileHandle(storepath + "/id")).getFile()
if(await idFile.text() !== id) throw "" if(await idFile.text() !== id) throw ""
tar = dataFile.stream() tar = dataFile.stream()
@@ -71,22 +246,27 @@ class genericModel extends EventTarget {
return result return result
} }
delete() { delete() {
if (this.obj) this.obj.delete() this.obj.delete()
} }
} }
Module.makeModel = async (url, storepath, id) => { Module.createModel = async (url, storepath, id) => {
return genericModel._init(url, storepath, id, true) return genericModel.create(url, storepath, id, true)
} }
Module.makeSpkModel = async (url, storepath, id) => { Module.createSpkModel = async (url, storepath, id) => {
return genericModel._init(url, storepath, id, false) return genericModel.create(url, storepath, id, false)
} }
class Recognizer extends EventTarget { class recognizer extends EventTarget {
constructor() { constructor() {
super() super()
objs.push(this) objs.push(this)
return new Proxy(this, {
get(self, prop, receiver) {
return self.obj && Object.keys(Object.getPrototypeOf(self.obj)).includes(prop) ? self.obj[prop].bind(self.obj) : self[prop] ? self[prop].bind ? self[prop].bind(self) : self[prop] : undefined
}
})
} }
static async _init(model, sampleRate, mode, grammar, spkModel) { static async create(model, sampleRate, mode, grammar, spkModel) {
let rec = new Recognizer() let rec = new recognizer()
let result = new Promise((resolve, reject) => { let result = new Promise((resolve, reject) => {
rec.addEventListener("0", ev => { rec.addEventListener("0", ev => {
if(ev.detail === "0") return resolve(rec) if(ev.detail === "0") return resolve(rec)
@@ -106,66 +286,18 @@ class Recognizer extends EventTarget {
} }
return result return result
} }
async getNode(ctx) { acceptWaveform(audioData) {
if(typeof this.node === "undefined") { let start = Module._malloc(audioData.length * 4)
await ctx.audioWorklet.addModule("../src/processor.js", { credentials : "omit"}) Module.HEAPF32.set(audioData, start / 4)
this.node = new AudioWorkletNode(ctx, 'VoskletProcessor', { channelCountMode: "explicit", channelCount: 1, numberOfInputs: 1, numberOfOutputs: 1, processorOptions: { dataBuf: this.dataBuf, state: this.state }}) this.obj.pushData(start, audioData.length)
}
return this.node
}
recognize(buf) {
Module.HEAPF32.set(buf.getChannelData(0).subarray(0, 512), this.ptr)
}
delete() {
if (this.obj) this.obj.delete()
if(this.node) this.node.postMessage(0)
}
setWords(words) {
this.obj.setWords(words)
}
setPartialWords(partialWords) {
this.obj.setPartialWords(partialWords)
}
setGrm(grm) {
this.obj.setGrm(grm)
}
setSpkModel(spkModel) {
this.obj.setSpkModel(spkModel.obj)
}
setNLSML(nlsml) {
this.obj.setNLSML(nlsml)
}
setMaxAlternatives(alts) {
this.obj.setMaxAlternatives(alts)
} }
} }
Module.makeRecognizer = (model, sampleRate) => { Module.createRecognizer = (model, sampleRate) => {
return Recognizer._init(model.obj, sampleRate, 1) return recognizer.create(model.obj, sampleRate, 1)
} }
Module.makeRecognizerWithSpkModel = (model, sampleRate, spkModel) => { Module.createRecognizerWithSpkModel = (model, sampleRate, spkModel) => {
return Recognizer._init(model.obj, sampleRate, 2, null, spkModel) return recognizer.create(model.obj, sampleRate, 2, null, spkModel)
} }
Module.makeRecognizerWithGrm = (model, sampleRate, grammar) => { Module.createRecognizerWithGrm = (model, sampleRate, grammar) => {
return Recognizer._init(model.obj, sampleRate, 3, grammar, null) return recognizer.create(model.obj, sampleRate, 3, grammar, null)
} }
/*let processorURL = URL.createObjectURL(new Blob(['(',
(() => {
registerProcessor("VoskletProcessor", class extends AudioWorkletProcessor {
constructor(options) {
this.channelIndex = options.processorOptions.channelIndex
this.dataBuf = options.processorOptions.dataBuf
this.state = options.processorOptions.state
}
process(inputs, outputs, params) {
while(state[0])
inputs.copyFromChannel(this.dataBuf, 0)
return true
}
})
}).toString(),
')()'], {type : "text/javascript"}))
let pthreadURL = URL.createObjectURL(new Blob(['(',
(() => {
{ PTHREAD_SCRIPT }
}).toString()
, ')()'], {type : "text/javascript"})) */

14
src/recognizer.cc
View File

@@ -19,7 +19,10 @@ void recognizer::finishConstruction(genericModel* model, genericModel* spkModel)
return; return;
} }
auto main {[this](){ auto main {[this](){
fireEv(index, "0");
while(!done) { while(!done) {
blocker.acquire();
blocker.release();
while(!dataQ.empty()) { while(!dataQ.empty()) {
switch(vosk_recognizer_accept_waveform_f(rec, dataQ.front().data, dataQ.front().len)) { switch(vosk_recognizer_accept_waveform_f(rec, dataQ.front().data, dataQ.front().len)) {
case 0: case 0:
@@ -36,23 +39,22 @@ void recognizer::finishConstruction(genericModel* model, genericModel* spkModel)
if(!model->resourceUsed) { if(!model->resourceUsed) {
model->resourceUsed = true; model->resourceUsed = true;
model->func = main; model->func = main;
model->blocker.unlock(); model->blocker.release();
emscripten_console_log("Using model's thread");
return; return;
} }
if(spkModel != nullptr && !spkModel->resourceUsed) { if(spkModel != nullptr && !spkModel->resourceUsed) {
spkModel->resourceUsed = true; spkModel->resourceUsed = true;
spkModel->func = main; spkModel->func = main;
spkModel->blocker.unlock(); spkModel->blocker.release();
emscripten_console_log("Using speaker model's thread");
return; return;
} }
emscripten_console_log("New recognizer thread");
std::thread t{main}; std::thread t{main};
t.detach(); t.detach();
} }
void recognizer::acceptWaveform(int start, int len) { void recognizer::pushData(int start, int len) {
dataQ.emplace(start, len); dataQ.emplace(start, len);
blocker.release();
blocker.acquire();
} }
void recognizer::reset() { void recognizer::reset() {
vosk_recognizer_reset(rec); vosk_recognizer_reset(rec);

8
src/recognizer.h
View File

@@ -1,8 +1,7 @@
#pragma once #pragma once
#include "genericModel.h" #include "genericModel.h"
#include <condition_variable>
#include <queue> #include <queue>
struct audioData { struct audioData {
float* data; float* data;
int len; int len;
@@ -10,15 +9,16 @@ struct audioData {
}; };
struct recognizer { struct recognizer {
std::atomic_bool done; std::atomic_bool done;
std::queue<audioData> dataQ{};
int index; int index;
std::binary_semaphore blocker{1};
std::queue<audioData> dataQ{};
VoskRecognizer* rec; VoskRecognizer* rec;
recognizer(int index, float sampleRate, genericModel* model); recognizer(int index, float sampleRate, genericModel* model);
recognizer(int index, float sampleRate, genericModel* model, genericModel* spkModel); recognizer(int index, float sampleRate, genericModel* model, genericModel* spkModel);
recognizer(int index, float sampleRate, genericModel* model, const std::string& grm, int dummy); recognizer(int index, float sampleRate, genericModel* model, const std::string& grm, int dummy);
~recognizer(); ~recognizer();
void finishConstruction(genericModel* model, genericModel* spkModel = nullptr); void finishConstruction(genericModel* model, genericModel* spkModel = nullptr);
void acceptWaveform(int start, int len); void pushData(int start, int len);
void reset(); void reset();
void setEndpointerMode(VoskEndpointerMode mode); void setEndpointerMode(VoskEndpointerMode mode);
void setEndpointerDelays(float tStartMax, float tEnd, float tMax); void setEndpointerDelays(float tStartMax, float tEnd, float tMax);

31
usage/fromMic.html
View File

@@ -1,31 +0,0 @@
<!DOCTYPE html>
<html>
<head>
<script src="BrowserRecognizer.js"></script>
<script>
/*(async () => {
const BrRec = await loadBR()
const model = await BrRec.makeModel("en-model.tzst","model","en0.0.1")
let ctx = new AudioContext()
const recognizer = await BrRec.makeRecognizer(model, ctx.sampleRate)
recognizer.addEventListener("result", e => {
console.log("Result: ",e.details)
})
recognizer.addEventListener("partialResult", e => {
console.log("Partial result: ",e.details)
})
let micNode = ctx.createMediaStreamSource(await navigator.mediaDevices.getUserMedia({
video: false,
audio: {
echoCancellation: true,
noiseSuppression: true,
channelCount: 1,
sampleRate: 16000
},
}));
let recNode = recognizer.getNode(ctx)
})()*/
</script>
</head>
</html>

31
usage/fromWav.html
View File

@@ -1,31 +0,0 @@
<!DOCTYPE html>
<html>
<head>
<script src="BrowserRecognizer.js"></script>
<script>
/*(async () => {
const BrRec = await loadBR()
const model = await BrRec.makeModel("en-model.tzst","model","en0.0.1")
let ctx = new AudioContext()
const recognizer = await BrRec.makeRecognizer(model, ctx.sampleRate)
recognizer.addEventListener("result", e => {
console.log("Result: ",e.details)
})
recognizer.addEventListener("partialResult", e => {
console.log("Partial result: ",e.details)
})
let micNode = ctx.createMediaStreamSource(await navigator.mediaDevices.getUserMedia({
video: false,
audio: {
echoCancellation: true,
noiseSuppression: true,
channelCount: 1,
sampleRate: 16000
},
}));
let recNode = recognizer.getNode(ctx)
})()*/
</script>
</head>
</html>