* Create github release via circleci only for mozilla fork
- The extension uses mozilla fork for translator artifacts
-- Hence create github release via circleci only when
running in mozilla fork
* Small refactoring in ci script
Hide `cache-mutex-buckets` from the user. Now configured to be equal to number
of workers. Python bindings which had exposed these are modified to reflect
the API change. `std::optional` enabled on cache, constructed only if enabled.
Pointers used are replaced with an equivalent `std::optional.`
Fixes: #317
Fixes memory leak
ifdef for -fno-exceptions including clang-cl
Move spacing back to intgemm upstream
Co-authored-by: Jerin Philip <jerin.philip@research.iiit.ac.in>
Changes signature of BlockingService::{translate,pivot}Multiple
functions to take per input options, so a mix of HTML and plaintext
can be sent from the extension. Templating over testing is adjusted
to allow for continuous evaluations by modifying the test code.
Updates WebAssembly bindings to reflect the change in signature
and the javascript test-page to work with the new bindings.
This change lacks an accompanying test specific to the mixed HTML
and plaintext inputs.
Fixes: #345
See also: mozilla/firefox-translations#94
Co-authored-by: Jelmer van der Linde <jelmer@ikhoefgeen.nl>
Changes `ABORT` on non `.bin` model to an additional check for a `.npz`
extension. If `.bin`, the fast load path is activated by returning `AlignedMemory`.
Otherwise, the return of empty `AlignedMemory` causes fallback to
filesystem-based loads.
BRT: A test that checks if translation using `.npz` is approximately similar to
that of default CLI translation is checked in to ensure stability going ahead.
Previously, we only supported `.bin` models' loading via a fast mmap
path. While we had the underlying capability to load non `.bin` models, this
was not exposed, encouraging fast loads. Loading `.npz` models are helpful
for quick debugging and broader coverage of models available, which will
enhance user experience at translateLocally and python bindings.
Fixes#341.
See also: XapaJIaMnu/translateLocally#89
This reverts commit 62ff781ed4.
Sorry I should have realized Jerin was only amending python and
therefore this didn't break WASM.
Apologies to Jerin on this.
Brings in the previously wasm.yml into python.yml and the new file is
renamed as build.yml.
python.yml already had a version and pre-release jobs. These jobs
downloaded artefacts from prior ran jobs (python wheel builds). The
newly attached emscripten build now uploads artefacts of a WebAssembly
binary and javascript file which are fed into the release and
pre-release jobs in addition to the existing python builds.
Enables ccache for emscripten. The configuration uses pyiodide for a
reference (https://github.com/pyodide/pyodide/pull/1805).
Two workflows to run on macOS and Ubuntu, reduced to one on Ubuntu. As
emscripten and the target is cross-platform, also macOS runners being
limited - it makes sense to have this removed.
Upload artefact enabled in preparation for a release action to be
scheduled which will upload the bergamot*.wasm and bergamot*.js for
consumption.
Previously BlockingService pivoting missed preproc and postproc for HTML
leading to issues in WebAssembly API. This change adds fixes for the
same, along with test coverage for the functionality over both async and
blocking services.
Imports python bindings and associated sources incubated in
https://github.com/jerinphilip/lemonade to bergamot-translator. Adds
a pybind11 dependency for python bindings.
Following the import, the python build is integrated into the existing
CMake based build system here. There is a command-line application
provided through python which provides the ability to fetch and prepare
models from model-repositories (like browsermt/students or OPUS).
Wheels built for a few common operating systems are provided via GitHub
releases through automated actions configured to run at tagged semantic
versions and pushes to main.
The documentation for python is also integrated into our existing
documentation setup. Previous documentation GitHub action is now
configured to run behind python builds in Ubuntu 18.04 Python3.7,
in order to pick up the packaged as a wheel bergamot module and the
sphinx documentation using the python module.
Formatting checks of black, isort with profile black and a pytype type
checker is configured for the python component residing in this repository.
Add a way to AsyncService to shut down without finishing the full queue
through `AsyncService::clear()`. The default behaviour is that
`AsyncService::~AsyncService()` will wait for any pending translation
requests to finish.
One can call `AsyncService::clear()` before the calls to the destructor
to ensure there is no work for the service to finish before the workers
can stop and join. Marian batches that are already in progress will not
stop. We are not trying to cause interrupts in threads or something that
complex. However, these single batches often do not take that long to
complete.
Changes:
- Add clear() to AsyncService
- Add clear() to BatchingPool
- Documentation
See also: XapaJIaMnu/translateLocally#80
Provides an additional constructor which takes care of the bundle
loading inside the boundary of the source here, when a configuration
file is supplied from a client like translateLocally or python bindings.
Once the config file is read, we have access to the information required
to construct the MemoryBundle.
- The command-line application supplied from here, app/bergamot is
configured to use the fast-load path now.
- Changes to binary-loading additionally revealed a bug in the
example-run script used in docs and tied to CI and the fix is
included.
- Shortlist is made optional in the memory bundle, making changes to
getModelMemoryFromConfig.
Fixes#304.
Fixes#306.
See also: XapaJIaMnu/translateLocally#82.
Allow the self-closing `/>` end for void tags. For non-void tags these
were already "allowed" due to how the HTML parser works, but for
elements where they actually occur, like `<br/>`, they caused a parse
error. Support for them was not implemented since we only expect valid
HTML5, e.g. the output of Firefox' Element.innerHTML.
Use case: TranslateLocally uses Qt's HTML representation of rich text.
That HTML uses self-closing tags like `<meta .../>` and `<br/>`.
Implementing a string replace operation that would only match these
elements without parsing HTML is tricky. Fixing it in
bergamot-translator is not.
Implementation: Currently `<img>` is marked as a void tag (an element
which cannot have children or text, and therefore treated differently.
Since void tags normally have no close tag, they are treated as
immediately closed. The HTML parser we use reads `<img/>` as
`<img></img>` which thus causes a problem since now we close an element
that was never open, to begin with.
This fix ignores the `TT_TAG_END` token from the parser when the tag
name is that of a void tag.
Translates a text from source-language to target-language through a
pivot-language. Effectively runs models in series, while having the
following additional benefits compared to when `Service::translate(...)`
would be used repeatedly.
1. Consistency in sentences between source and target. Consistent
creation of the alignment matrix for use in downstream tasks like
tag-translation.
2. Efficient sentence-splitting (does not sentence-split twice, creating
inconsistencies).
3. The `Response` generated can be used as if it were coming through
`translate(...)`, eliminating any need for additional code for clients
in JS or python or C++.
`AsyncService::pivot(...)` is provisioned for C++ multi-threaded setting
and `BlockingService::pivotMultiple(...)` provisioned for blocking
use-case targeted at WebAssembly.
# [BRT]: Test additions, accompanying fixes
For `AsyncService` for a test-case involving of en->es, es->en (same
vocabulary, another one might be more coverage but is too much work).
1. Asserts the Alignment generated after pivoting is a probability
distribution over source tokens given target.
2. Outputs the sentences going from en->en, which should stay consistent
over continuous development to ensure nothing breaks.
3. An accuracy minimum of 70% of token matches from source to target
calibrated on the standard bergamot input text is additionally present,
ensuring that the English tokens at start and end match exactly.
# HTML Pipeline
This PR reworks the HTML translation pipeline to be outside
response-construction via callbacks.
* Convert marian-integration markdown to rst
* Convert native run into a script, include in rst
* Check with CI that the native running example works without fail
* Make stats hits misses atomic to guard when mutex has multiple buckets
* Use compile time switch for cache-stats-collection bound to COMPILE_TESTS cmake variable
* -DENABLE_CACHE_STATS on if COMPILE_TESTS otherwise optional
* Make stats() call without enabling build fatal abort
* Turn logging off by default, allow turning on via config/cmdline
* No need to store config in member variable if things are decided at construction time
- Robust artifact presence check
- Variable name refactoring
- Storing only those artifacts that are required
- Remove commit sha from the names of the Github Releases
- Use BERGAMOT_VERSION file contents for Git Tag names
* Fix test case
This should now be implemented
* Remove FilterEmpty
This path wasn't used anymore anyway, empty tags just got their own spans, and never reached the stack.
* Insert skipped empty source spans into target HTML
Also refactor variable names to better match their contents and be more consistent with each other.
This implementation passes all test cases, finally!
* Fix remaining style changes
* Move HTML formatting to its own section
That code had become exact copies in three different places
* CLI Rework
* Consolidate common tests, template specialize CLI
* Remove remnant cache stuff
* [BRT]: Run BRT with new cli
* Formalizing bridge
* Removing stuff from parsing and moving to TestSuite
* Template includes, everything consolidating at tests
* Inlining readFromStdin
* Removing unnecessary headers
* Checking in template implementation which was missing
* Sane defaults, some catches at BRT
* BRT: Install fixes
* Updating marian-dev to point to main
* Removing the enum indirection, using strings at one place, directly
* Fix typo;
* [BRT] test blocking service via native
* Conservative defaults for workers and cache-mutex buckets in AsyncService
* Create proper barriers for cmdline app
* Build failure fixes
* Moving common, common-impl to a familiar structure
* Binary reorganization: async, blocking, wasm
- async tests AsyncService
- blocking tests BlockingService
- wasm arranges tests for things that are Mozilla requirements. eg:
- bytearray
- multiple sentences in same translate request workflow.
* [brt] updates to adapt to cli rework
* [brt] updates to adapt to cli rework, all working
* Empty commit, sync brt online and run GitHub CI
* Switch for parser to have multiple mode or not
* [brt]: Fix for --bergamot-mode being removed from CLI app
* [brt]: Fix for --bergamot-mode being removed from CLI app
* [brt]: Removing remnant faithful translation test from blocking/
* Fix bug in HasAlignments check
When fixing it to allow empty sentences, it no longer caught misconfigured models. I've added a test that triggers this scenario, and a fix in HasAlignments for it.
* Add more unit tests for xh_scanner
Trying to increase that code coverage to 100%
* Add test for whitespaces around attributes
* Make accessing value(), attr_name() and tag_name() at the wrong time safer
* Fix bug in <style> and <script> parsing
The end tag was never found
* Fix parsing of mix of valueless and quoteless attributes
* Sync list of void tags with Firefox' implementation of outerHTML and innerHTML
Also lets use their name for it: IsVoidTag instead of IsEmptyElement. Empty was a bit ambiguous.
* Bring back support for processing instructions support in xh_scanner
I noticed in https://searchfox.org/mozilla-central/source/dom/base/nsContentUtils.cpp#8961 that these can be produced by innerHTML under some circumstances.
* More permanent link
* Use CamelCase for the internal functions I added
* Rename *_PI to *_PROCESSING_INSTRUCTION
Your IDE will do the typing for you anyway
* Match symbol naming of the rest of code base
CapitalCase for classes, camelCase for functions, snake_case for variables still.
* Missed one 😴
* Change xhscanner's variable case also to camelCase
* Partially fix case variables in html.cpp
* Remove InterpolateAlignment
And some code improvements
* Replace the fixed value buffer with a std::string backing
* Fix tests that had no alignment info
These depended on the linear interpolation that I removed
* Remove arbitrary limits on tag and attribute names
This might also fix a bug caused by the eager lower casing of tag names, which could break <
