mirror of https://github.com/browsermt/bergamot-translator.git synced 2024-09-11 05:35:33 +03:00

Cross platform C++ library focusing on optimized machine translation on the consumer-grade device.

cpp cross-platform emscripten machine-translation neural-machine-translation neural-networks python starred-browsermt-repo starred-repo wasm webassembly

Go to file

Jerin Philip 34228d37bf Collapse Service into one class instead of three (#62 ) * Merging two Services * Moving stop() logic to destructor * We have WITH_PTHREADS back * string based constructor on Service * Removing now empty service_base.* files * Hiding away pcqueue_ construction Ugliest ifdefs I have done in my life. * Another ifdef to hide pcqueue header file * Missing semicolons in WITH_PTHREADS path * Fixing async_translate residue argument from copy * Adding comments * Initialize batchtranslator only at one place To reduce tax for bytebuffer loads, initialize batchtranslator only at one place. * \#ifdef WITH_PTHREADS -> #ifndef WASM_HIDE_THREADS Sane platform (non WASM) is default. This truly only hide-threads from compilation path and not switch unswitch pthreads (-lpthread). * Review comments: Rearranging destructor, fix wrong comment * Move loadVocabularies to service.cpp and put in anonymous namespace * Prettifying diff: Removing unwanted empty lines * Indicate in comments multithreaded has numWorkers translators * Typo fix: bergamot_translator -> bergamot-translator * Safety guards to avoid pcqueue illegal init * Add WASM_HIDE_THREADS as a global WASM_COMPILE_FLAG * Compile Defs: WASM_HIDE_THREADS -> __EMSCRIPTEN__ * Removing dead CMakeLists.txt code following __EMSCRIPTEN__ * Compile defs: __EMSCRIPTEN__ -> WASM		2021-03-23 16:36:13 +00:00
.github/workflows	apt-update for ubuntu github actions	2021-03-11 11:30:21 +01:00
3rd_party	Load mode as a byte array (#55 )	2021-03-22 14:22:56 +00:00
app	Collapse Service into one class instead of three (#62 )	2021-03-23 16:36:13 +00:00
doc	Cleanup CMakeFiles.txt	2021-02-26 14:17:48 +01:00
src	Collapse Service into one class instead of three (#62 )	2021-03-23 16:36:13 +00:00
wasm	Removed erroneous '?' in example text in wasm test page	2021-02-23 18:43:51 +01:00
.gitignore	Merge remote-tracking branch 'origin/wasm-integration' into jp/absorb-batch-translator	2021-02-17 13:08:58 +00:00
.gitmodules	Update submodule ssplit-cpp	2021-03-03 11:48:56 +01:00
CMakeLists.txt	Fixed native builds while using wasm compatible sources	2021-02-26 14:55:30 +01:00
LICENSE	Initial commit	2020-10-19 13:49:38 +02:00
README.md	Improved README	2021-02-25 15:22:52 +01:00

README.md

Bergamot Translator

Bergamot translator provides a unified API for (Marian NMT framework based) neural machine translation functionality in accordance with the Bergamot project that focuses on improving client-side machine translation in a web browser.

Build Instructions

Build Natively

Clone the repository using these instructions:

git clone https://github.com/browsermt/bergamot-translator
cd bergamot-translator

Compile

Create a folder where you want to build all the artifacts (build-native in this case) and compile in that folder
```
mkdir build-native
cd build-native
cmake ../
make -j
```

Build WASM

Compiling for the first time

Download and Install Emscripten using following instructions
- Get the latest sdk: git clone https://github.com/emscripten-core/emsdk.git
- Enter the cloned directory: cd emsdk
- Install the lastest sdk tools: ./emsdk install latest
- Activate the latest sdk tools: ./emsdk activate latest
- Activate path variables: source ./emsdk_env.sh

Clone the repository using these instructions:

git clone https://github.com/browsermt/bergamot-translator
cd bergamot-translator

Download files (only required if you want to package files in wasm binary)

This step is only required if you want to package files (e.g. models, vocabularies etc.) into wasm binary. If you don't then just skip this step.

The build preloads the files in Emscripten’s virtual file system.

If you want to package bergamot project specific models, please follow these instructions:
```
mkdir models
git clone https://github.com/mozilla-applied-ml/bergamot-models
cp -rf bergamot-models/* models
gunzip models/*/*
```
Compile
1. Create a folder where you want to build all the artefacts (build-wasm in this case)
```
mkdir build-wasm
cd build-wasm
```
2. Compile the artefacts
  - If you want to package files into wasm binary then execute following commands (Replace FILES_TO_PACKAGE with the path of the directory containing the files to be packaged in wasm binary)
```
emcmake cmake -DCOMPILE_WASM=on -DPACKAGE_DIR=FILES_TO_PACKAGE ../
emmake make -j
```
    e.g. If you want to package bergamot project specific models (downloaded using step 3 above) then replace FILES_TO_PACKAGE with ../models
  - If you don't want to package any file into wasm binary then execute following commands:
```
emcmake cmake -DCOMPILE_WASM=on ../
emmake make -j
```
The artefacts (.js and .wasm files) will be available in wasm folder of build directory ("build-wasm" in this case).

Recompiling

As long as you don't update any submodule, just follow steps in 4.ii to recompile.
If you update a submodule, execute following command before executing steps in 4.ii to recompile.

git submodule update --init --recursive

How to use

Using Native version

The builds generate library that can be integrated to any project. All the public header files are specified in src folder.
A short example of how to use the APIs is provided in app/main.cpp file.

Using WASM version

Please follow the README inside the wasm folder of this repository that demonstrates how to use the translator in JavaScript.

README.md Unescape Escape