Sets a cache to operate for each sentence that a TranslationModel process
caching the corresponding marian::History for a {TranslationModel::Id, marian::Words}
key. Cache is thus shared across multiple TranslationModels bound to the lifetime
of a Service. Cache gracefully downgrades in the case of WebAssembly.
* Updated marian-dev submodule
* Import wasm gemm from a separate wasm module
- The fallback implementation of gemm is currently being imported dynamically
for wasm target
* Updated CI scripts and README to import GEMM from a separate wasm module
* Setting model config to int8shiftAlphaAll in wasm test page
For outbound translation, we require having multiple models in the
inventory at the same time and abstracting the "how-to-translate"
using a model out.
Reorganization: TranslationModel + Service. The new entity which
contains everything required to translate in one direction is
`TranslationModel`. The how-to-translate blocking single-threaded mode
of operation or async multi-threaded mode of operation is decoupled as
`BlockingService` and `AsyncService`. There is a new regression-test
using multiple models in conjunction added, also serving as
a demonstration for using multiple models in Outbound Translation.
WASM: WebAssembly due to the inability to use threads uses
`BlockingService. Bindings are provided with a new API to work with a
Service, and multiple TranslationModels which the client (JS extension)
can inventory and maintain. Ownership of a given `TranslationModel` is
shared while translations using the model are active in the internal
mechanism.
Config-Parsing: So far bergamot-translator has been hijacking marian's
config-parsing mechanisms. However, in order to support multiple models,
it has become impractical to continue this approach and a new
config-parsing that is bergamot specific is provisioned for
command-line applications constituting tests. The original marian
config-parsing tooling is only associated with a subset of
`TranslationModel` now. The new config-parsing for the library manages
workers and other common options (tentatively).
There is a known issue of: Inefficient placing of workspaces, leading to
more memory usage than what's necessary. This is to be fixed trickling
down from marian-dev in a later pull request.
This PR also brings in BRT changes which fix speed-tests that were
broken and also fixes some QE outputs which were different due to not
using shortlist.
Unifies quality estimation with an interface, refactors previously available
quality scores to fit this interface. Adds a new class of model with Logistic
Regression powering the predictions as an implementation of said interface.
QE now provides annotations on words using subwords to word rule-based
algorithms working with space characters.
QualityEstimation
-----------------
Implementations of QE are bound together by a `QualityEstimator`
Interface.
1. The log-probabilities from the machine-translation model re-interpreted
as quality scores are crafted as an implementation of QualityEstimator.
2. A Logistic-Regression based model is added. This class of models is
trained supervised with scores labeled by a human annotator.
Handcrafted features - number of words, log probs from MT model and
statistics over the sequence are used to generate the numeric features.
LogisticRegressor, Matrix (to hold features) are added.
The creation of an instance is switched by the `AlignedMemory` supplied
(be it loaded from the file-system or supplied as a parameter). An empty
AlignedMemory leads to quality scores from NMT while supplying weights
of a trained logistic-regression model in binary format as the contents
lead to an additional pass through the said model to provide more
refined scores.
Both the above now transform subwords into "words" using a heuristic
algorithm, scanning for spaces. This allows the client to work with "words"
to denote quality instead of subwords, as the former is more sensible to
the user.
Testing
-------
1. BRT now has two new test apps to check the QE outputs in text
(covers subword to words) and numbers domain (covers quality scores).
These are tested with en-et models for which QualityEstimation is
available now, on a new input to avoid architecture/compiler issues.
2. Unit test for LogisticRegression model is added.
Docs
----
Doxygen now supports MathJax properly to render explanations for
Logistic Regressions' reductions in place to make computation more
efficient correctly.
Co-authored-by: Felipe C. Dos Santos <felipe.santos.k@gmail.com>
Co-authored-by: Jerin Philip <jerinphilip@live.in>
Adds a clang-tidy run in addition to the existing clang-format checks.
The clang-tidy checks are not enforced, but is potentially useful to
point to during review.
A cmake change has caused vcpkg to fail without much error message,
which is causing windows workflow runs to fail. Details in the following
link:
* https://github.com/microsoft/vcpkg/issues/18718
To fix, we're going with a version bump in vcpkg. Seeing that run-vcpkg
also seems to have gotten an update, updating run-vcpkg from 7.3 to 7.4
Playing with fire: vcpkg master commit
* Change ResponseBuilder to accept callback
Breaks things everywhere, now we follow the compiler to fix and convert
the std::future -> callback.
* More std::future -> callback
* std::future out of service.{h,cpp}
* compile is working, so is callback
* Some reshuffling of args
* Fixing merge error
* Fixing signature conflicts out of merge
* Fixing that test duct-taping future
* Minor adjustment to get that future back
* Add documentation for the new callback function
* Applying clang-format after update
* Using default responseOptions
* Remove future references from documentation
* translateMultiple only for WASM (#177)
* BRT: update to main; fresh-failures hopefully
* Converting test translateFromStdin to use callback
* BRT: Add fresh #native and #wasm tags
* future from promise, fix error
* Adding #native to GitHub CI
Co-authored-by: Nikolay Bogoychev <nheart@gmail.com>
Service now allows loading Sentence-Splitter (non-breaking prefix file) from ByteArray. Behaviour is consistent with the rest of the ByteArray loads (model, shortlist), where first the ByteArray is checked if empty, if not fall back to loading from file-path.
Adds regression test to check if source-sentences in constructed Response match expected behaviour when the non-breaking-prefixes file is provided.
Bonus refactoring to remove an extra layer that existed for no reason.
* Refactored wasm bindings code
- Replaced TranslationModel, TranslationRequest and TranslationResult
with Service, ResponseOptions and Response
- Corresponding documentation changes
- Names of the bindings files changed
- Moved Vector<Response> definition in Response specific bindings
file
* Partial test applications
Previously service-cli was used to generate output and accomplish
regression testing for all of: (1) translated-text (2) alignment tokens
+ scores (3) quality scores (4) indirectly annotation and tokenizations.
The --mode native now only outputs a faithful to source translated text
of the input source on stdin.
Test apps are separated into testing only individual functionalities.
This can help in independently testing ssplit-cpp, quality-scores for
the quality estimation implementation etc.
Separating numbers and text have the advantage of being able to compare
one with tolerance using BLEU (text) and some allowed error-rates
(numbers).
* Removing #mac tag
* Moving test apps to src/tests
* Tests are always on for CI
Unit tests are turned off looking for WASM_COMPATIBLE_SOURCES.
* Fixing WASM_COMPATIBLE_SOURCE -> USE_WASM_COMPATIBLE_SOURCE
* Workaround for now; CMakeLists.txt horrors are starting to bite
* BRT: use bergamot-test instead of bergamot now
* This should fix issues: CMakeLists.txt has so many paths
* Casing to camelCase and removing legacyServiceCli
* removing leftover service-cli declaration, some doc updates
* #pragma once is starting to look easier
* All the more reasons to do #pragma once
* Updating marian-dev with intgemm::kCPU print, resolved from INTGEMM_CPUID
* BRT: Use --gemm-highest-arch instead of python script
* Adding intgemm resolve here, where always(?) have intgemm on?
* intgemm-resolve in default binary directory
* BRT: Update to use intgemm-resolve
* marian-dev: Reset to without --gemm-highest-precision
Co-authored-by: Kenneth Heafield <kpu@users.noreply.github.com>
* Use CMAKE_CURRENT_SOURCE_DIR instead of CMAKE_SOURCE_DIR for project bound version string
* marian-dev cmake fix
* Generate project.h in binary dir
* We don't want people asking about extra spaces
* Matrix is now more organized, Ubuntu 20.04-gcc9.3, Ubuntu-18.04-gcc7.5 is added.
* ccache is extended to MacOS, and brings down CI run times to <5m when
ccache works.
* The compiler hash scripts are gone, ccache already covers most ground
by default. The shell script is unnecessary. Cache works by preprocessor
mode output of running the compiler with -E, which includes the
necessary information. ccache-docs:How the cache works.
* BRT if failed prints the final 20 lines of the test*.log to inspect
what's going wrong without having to artifact download.
* Pull request on any branch triggers workflow.
* Push on main and ci-sandbox triggers workflow.
* Collapsing executables
* Adding new test executable
* Deleting old executable sources
* Updating brt to operate with modes
* cli-framework -> cli
* Updating workflows to check for bergamot instead of bergamot-translator-app
* Adding documentation
* Making fn pure virtual
* Shuffling apps into app namespace, alongside class documentation
* Include app folder in documentation
* BRT update service-cli -> native
* parser.h: service-cli -> native
* Updates to marian-integration.md
* Cleanup: Remove templates, interface proper
* change 4 to 2 cores for build instructions
* service-cli -> native
* Commenting the string constructor explanation
* Not doing halfway interface / inheritance
* Nick hates state, let's try this one
* Revert "Nick hates state, let's try this one"
This reverts commit e56db9f474.
* class -> struct before trying std::function stuff
* oop -> functional?
* Hints on what is happening
* app::ftable -> app::REGISTRY
* We have if-else and functions now.
And we won't have test apps.
* Doc linking to usage examples in brt
* Remove unordered_map
* Documentation updates
* Fix warning
Adds a GitHub workflow that builds documentation from sources through doxygen through sphinx on push to the main branch or on push of any semantic version tags. The built documentation is deployed at https://github.com/browsermt/docs@gh-pages, which is rendered at https://browser.mt/docs/<suffix>, where <suffix> is 'main' or a tag vM.m.p corresponding to a semantic version.
On pull request artifacts are uploaded for reviewers to inspect if need be.
