acbc46d816
Allow the self-closing `/>` end for void tags. For non-void tags these were already "allowed" due to how the HTML parser works, but for elements where they actually occur, like `<br/>`, they caused a parse error. Support for them was not implemented since we only expect valid HTML5, e.g. the output of Firefox' Element.innerHTML. Use case: TranslateLocally uses Qt's HTML representation of rich text. That HTML uses self-closing tags like `<meta .../>` and `<br/>`. Implementing a string replace operation that would only match these elements without parsing HTML is tricky. Fixing it in bergamot-translator is not. Implementation: Currently `<img>` is marked as a void tag (an element which cannot have children or text, and therefore treated differently. Since void tags normally have no close tag, they are treated as immediately closed. The HTML parser we use reads `<img/>` as `<img></img>` which thus causes a problem since now we close an element that was never open, to begin with. This fix ignores the `TT_TAG_END` token from the parser when the tag name is that of a void tag. |
||
|---|---|---|
| .circleci | ||
| .github/workflows | ||
| 3rd_party | ||
| app | ||
| bergamot-translator-tests@e49686b7ca | ||
| cmake | ||
| doc | ||
| examples | ||
| scripts/ci | ||
| src | ||
| vcpkg-override/ports/pcre2 | ||
| wasm | ||
| .clang-format | ||
| .clang-format-ignore | ||
| .clang-tidy | ||
| .gitignore | ||
| .gitmodules | ||
| BERGAMOT_VERSION | ||
| build-wasm.sh | ||
| CMakeLists.txt | ||
| Doxyfile.in | ||
| LICENSE | ||
| README.md | ||
| run-clang-format.py | ||
Bergamot Translator
Bergamot translator provides a unified API for (Marian NMT framework based) neural machine translation functionality in accordance with the Bergamot project that focuses on improving client-side machine translation in a web browser.
Build Instructions
Build Natively
Create a folder where you want to build all the artifacts (build-native in this case) and compile
mkdir build-native
cd build-native
cmake ../
make -j2
Build WASM
Prerequisite
Building on wasm requires Emscripten toolchain. It can be downloaded and installed using following instructions:
- Get the latest sdk:
git clone https://github.com/emscripten-core/emsdk.git - Enter the cloned directory:
cd emsdk - Install the lastest sdk tools:
./emsdk install 2.0.9 - Activate the latest sdk tools:
./emsdk activate 2.0.9 - Activate path variables:
source ./emsdk_env.sh
Compile
To build a version that translates with higher speeds on Firefox Nightly browser, follow these instructions:
-
Create a folder where you want to build all the artifacts (
build-wasmin this case) and compilemkdir build-wasm cd build-wasm emcmake cmake -DCOMPILE_WASM=on ../ emmake make -j2The wasm artifacts (.js and .wasm files) will be available in the build directory ("build-wasm" in this case).
-
Enable SIMD Wormhole via Wasm instantiation API in generated artifacts
bash ../wasm/patch-artifacts-enable-wormhole.sh -
Patch generated artifacts to import GEMM library from a separate wasm module
bash ../wasm/patch-artifacts-import-gemm-module.sh
To build a version that runs on all browsers (including Firefox Nightly) but translates slowly, follow these instructions:
-
Create a folder where you want to build all the artifacts (
build-wasmin this case) and compilemkdir build-wasm cd build-wasm emcmake cmake -DCOMPILE_WASM=on -DWORMHOLE=off ../ emmake make -j2 -
Patch generated artifacts to import GEMM library from a separate wasm module
bash ../wasm/patch-artifacts-import-gemm-module.sh
Recompiling
As long as you don't update any submodule, just follow Compile steps.
If you update a submodule, execute following command in repository root folder before executing
Compile steps.
git submodule update --init --recursive
How to use
Using Native version
The builds generate library that can be integrated to any project. All the public header files are specified in src folder.
A short example of how to use the APIs is provided in app/main.cpp file.
Using WASM version
Please follow the README inside the wasm folder of this repository that demonstrates how to use the translator in JavaScript.