bergamot-translator/.gitignore

# vim temporary files
*.swp
*.swo

# CMake
CMakeLists.txt.user
CMakeCache.txt
CMakeFiles
CMakeScripts
Testing
Makefile
cmake_install.cmake
install_manifest.txt
compile_commands.json
CTestTestfile.cmake
_deps


wasm/test_page/node_modules
build-wasm
models
wasm/module/worker/bergamot-translator-worker.*
wasm/module/browsermt-bergamot-translator-*.tgz

# VSCode
.vscode
Adding vim temporary files to .gitignore 2021-01-22 14:31:20 +03:00			`# vim temporary files`
			`*.swp`
			`*.swo`

Updates: marian-dev, ssplit for marian-decoder-new Updates marian-dev and ssplit submodules to point to the upstream commits which implements the following: - marian-dev: encodeWithByteRanges(...) to get source token byte-ranges - ssplit: Has a trivial sentencesplitter functionality implemented, and now is faster to benchmark with marian-decoder. This enables a marian-decoder replacement written through ssplit in this source to be benchmarked constantly with existing marian-decoder. Nits: Removes logging introduced for multiple workers, and respective log statements. 2021-02-05 15:55:57 +03:00			`# CMake`
			`CMakeLists.txt.user`
			`CMakeCache.txt`
			`CMakeFiles`
			`CMakeScripts`
			`Testing`
			`Makefile`
			`cmake_install.cmake`
			`install_manifest.txt`
			`compile_commands.json`
			`CTestTestfile.cmake`
			`_deps`


Including a more elaborated test page, a node webserver containing the proper cors headers and wasm mimetype 2021-02-14 01:54:36 +03:00			`wasm/test_page/node_modules`
Imported CI scripts from mozilla/bergamot-translator-old (#1) * CircleCI config, docs and badge * Increase CircleCI RAM from 4gb to 16gb Co-authored-by: Motin <motin@motin.eu> 2021-03-10 20:30:39 +03:00			`build-wasm`
Ignore contents in models directory 2021-02-15 12:23:08 +03:00			`models`
More portable WASM demo (#437) * Replace most of the wasm demo page with code from the firefox extension This code should be more generic and copy/pastable into other projects. Maybe one day it will be an npm package? * Fix Ukrainian model support * Add quality estimation output Automatically enabled when the model(s) support it * Little "Translating…" indicator * Don't make Safari fail on something tiny * Rewire lots of async state to be able to predictably know when the translator is working or not Previously so much was lazy loaded that it was not easy to catch lack of SIMD support. Now I can just enable the interface only after it has properly loaded. * No need for a two-stage setup for the worker. Just promise to call `initialize()`! * More (correct) types and comments for code * Keyboard shortcuts for input area for bold, italic and underline. Enough to demo mark-up translation * Fix `delete()` * Move javascript glue code into its own npm package * Add nodejs support and test to package * More stand-alone build command …for now, not really used by anything I think * Ignore build packages * Use local filesystem for build so it is automatically cached * fix overflow on demo page But this might break the mobile demo? I'll have to check into that * Bring back integrity check, except for NodeJS for now * Make `build` part of `prepare` so we always make sure we build a complete package * Move worker code into its own folder This way I can mark it as a commonjs module which will help cause nodejs treat the files the same as WebWorkers do right now. Firefox doesn't implement `{type: 'module'}` yet for WebWorkers. * Add README * Fix paths * Add npm publish automation * Make sure webpack ignores node compatibility code * Add missing webpack:ignore around a worker * Default to getting models from S3 * Separate "loading" and "translating" indicators * Bump npm package version * Add credits * Don't block on the worker loading * Not just Mozilla, but Bergamot! * Make individual translation requests cancelable * Swap button turns vertically when in skyscraper mode * Make it easier to debug errors from inside the worker * Don't bork on deleting a failed worker * Don't bork on calling translate() with a failed worker * Handle compilation error with more grace * `contenteditable=true` seems to work better with some browser extensions Looking at you, Vimium! * Clean up abort promise * Bump npm package version * Remove `workerUrl` option in favour of better webpack support With that option it was hard for Webpack to figure out dependencies, and it did not enter my worker script for rewriting. With the hardcoded url it does, and with a bit of `new webpack.DefinePlugin({'typeof self': JSON.stringify('object')}),` we can have webpack remove node-specific code on build! * Bump version Minor API change hehe Co-authored-by: Nikolay Bogoychev <nheart@gmail.com> 2023-01-18 22:41:39 +03:00			`wasm/module/worker/bergamot-translator-worker.*`
			`wasm/module/browsermt-bergamot-translator-*.tgz`
QualityEstimation: Preliminary Implementation (#197) Unifies quality estimation with an interface, refactors previously available quality scores to fit this interface. Adds a new class of model with Logistic Regression powering the predictions as an implementation of said interface. QE now provides annotations on words using subwords to word rule-based algorithms working with space characters. QualityEstimation ----------------- Implementations of QE are bound together by a `QualityEstimator` Interface. 1. The log-probabilities from the machine-translation model re-interpreted as quality scores are crafted as an implementation of QualityEstimator. 2. A Logistic-Regression based model is added. This class of models is trained supervised with scores labeled by a human annotator. Handcrafted features - number of words, log probs from MT model and statistics over the sequence are used to generate the numeric features. LogisticRegressor, Matrix (to hold features) are added. The creation of an instance is switched by the `AlignedMemory` supplied (be it loaded from the file-system or supplied as a parameter). An empty AlignedMemory leads to quality scores from NMT while supplying weights of a trained logistic-regression model in binary format as the contents lead to an additional pass through the said model to provide more refined scores. Both the above now transform subwords into "words" using a heuristic algorithm, scanning for spaces. This allows the client to work with "words" to denote quality instead of subwords, as the former is more sensible to the user. Testing ------- 1. BRT now has two new test apps to check the QE outputs in text (covers subword to words) and numbers domain (covers quality scores). These are tested with en-et models for which QualityEstimation is available now, on a new input to avoid architecture/compiler issues. 2. Unit test for LogisticRegression model is added. Docs ---- Doxygen now supports MathJax properly to render explanations for Logistic Regressions' reductions in place to make computation more efficient correctly. Co-authored-by: Felipe C. Dos Santos <felipe.santos.k@gmail.com> Co-authored-by: Jerin Philip <jerinphilip@live.in> 2021-09-16 18:28:40 +03:00
			`# VSCode`
			`.vscode`