Allow the self-closing `/>` end for void tags. For non-void tags these
were already "allowed" due to how the HTML parser works, but for
elements where they actually occur, like `<br/>`, they caused a parse
error. Support for them was not implemented since we only expect valid
HTML5, e.g. the output of Firefox' Element.innerHTML.
Use case: TranslateLocally uses Qt's HTML representation of rich text.
That HTML uses self-closing tags like `<meta .../>` and `<br/>`.
Implementing a string replace operation that would only match these
elements without parsing HTML is tricky. Fixing it in
bergamot-translator is not.
Implementation: Currently `<img>` is marked as a void tag (an element
which cannot have children or text, and therefore treated differently.
Since void tags normally have no close tag, they are treated as
immediately closed. The HTML parser we use reads `<img/>` as
`<img></img>` which thus causes a problem since now we close an element
that was never open, to begin with.
This fix ignores the `TT_TAG_END` token from the parser when the tag
name is that of a void tag.
Translates a text from source-language to target-language through a
pivot-language. Effectively runs models in series, while having the
following additional benefits compared to when `Service::translate(...)`
would be used repeatedly.
1. Consistency in sentences between source and target. Consistent
creation of the alignment matrix for use in downstream tasks like
tag-translation.
2. Efficient sentence-splitting (does not sentence-split twice, creating
inconsistencies).
3. The `Response` generated can be used as if it were coming through
`translate(...)`, eliminating any need for additional code for clients
in JS or python or C++.
`AsyncService::pivot(...)` is provisioned for C++ multi-threaded setting
and `BlockingService::pivotMultiple(...)` provisioned for blocking
use-case targeted at WebAssembly.
# [BRT]: Test additions, accompanying fixes
For `AsyncService` for a test-case involving of en->es, es->en (same
vocabulary, another one might be more coverage but is too much work).
1. Asserts the Alignment generated after pivoting is a probability
distribution over source tokens given target.
2. Outputs the sentences going from en->en, which should stay consistent
over continuous development to ensure nothing breaks.
3. An accuracy minimum of 70% of token matches from source to target
calibrated on the standard bergamot input text is additionally present,
ensuring that the English tokens at start and end match exactly.
# HTML Pipeline
This PR reworks the HTML translation pipeline to be outside
response-construction via callbacks.
* Convert marian-integration markdown to rst
* Convert native run into a script, include in rst
* Check with CI that the native running example works without fail
* Make stats hits misses atomic to guard when mutex has multiple buckets
* Use compile time switch for cache-stats-collection bound to COMPILE_TESTS cmake variable
* -DENABLE_CACHE_STATS on if COMPILE_TESTS otherwise optional
* Make stats() call without enabling build fatal abort
* Turn logging off by default, allow turning on via config/cmdline
* No need to store config in member variable if things are decided at construction time
- Robust artifact presence check
- Variable name refactoring
- Storing only those artifacts that are required
- Remove commit sha from the names of the Github Releases
- Use BERGAMOT_VERSION file contents for Git Tag names
* Fix test case
This should now be implemented
* Remove FilterEmpty
This path wasn't used anymore anyway, empty tags just got their own spans, and never reached the stack.
* Insert skipped empty source spans into target HTML
Also refactor variable names to better match their contents and be more consistent with each other.
This implementation passes all test cases, finally!
* Fix remaining style changes
* Move HTML formatting to its own section
That code had become exact copies in three different places
* CLI Rework
* Consolidate common tests, template specialize CLI
* Remove remnant cache stuff
* [BRT]: Run BRT with new cli
* Formalizing bridge
* Removing stuff from parsing and moving to TestSuite
* Template includes, everything consolidating at tests
* Inlining readFromStdin
* Removing unnecessary headers
* Checking in template implementation which was missing
* Sane defaults, some catches at BRT
* BRT: Install fixes
* Updating marian-dev to point to main
* Removing the enum indirection, using strings at one place, directly
* Fix typo;
* [BRT] test blocking service via native
* Conservative defaults for workers and cache-mutex buckets in AsyncService
* Create proper barriers for cmdline app
* Build failure fixes
* Moving common, common-impl to a familiar structure
* Binary reorganization: async, blocking, wasm
- async tests AsyncService
- blocking tests BlockingService
- wasm arranges tests for things that are Mozilla requirements. eg:
- bytearray
- multiple sentences in same translate request workflow.
* [brt] updates to adapt to cli rework
* [brt] updates to adapt to cli rework, all working
* Empty commit, sync brt online and run GitHub CI
* Switch for parser to have multiple mode or not
* [brt]: Fix for --bergamot-mode being removed from CLI app
* [brt]: Fix for --bergamot-mode being removed from CLI app
* [brt]: Removing remnant faithful translation test from blocking/
* Fix bug in HasAlignments check
When fixing it to allow empty sentences, it no longer caught misconfigured models. I've added a test that triggers this scenario, and a fix in HasAlignments for it.
* Add more unit tests for xh_scanner
Trying to increase that code coverage to 100%
* Add test for whitespaces around attributes
* Make accessing value(), attr_name() and tag_name() at the wrong time safer
* Fix bug in <style> and <script> parsing
The end tag was never found
* Fix parsing of mix of valueless and quoteless attributes
* Sync list of void tags with Firefox' implementation of outerHTML and innerHTML
Also lets use their name for it: IsVoidTag instead of IsEmptyElement. Empty was a bit ambiguous.
* Bring back support for processing instructions support in xh_scanner
I noticed in https://searchfox.org/mozilla-central/source/dom/base/nsContentUtils.cpp#8961 that these can be produced by innerHTML under some circumstances.
* More permanent link
* Use CamelCase for the internal functions I added
* Rename *_PI to *_PROCESSING_INSTRUCTION
Your IDE will do the typing for you anyway
* Match symbol naming of the rest of code base
CapitalCase for classes, camelCase for functions, snake_case for variables still.
* Missed one 😴
* Change xhscanner's variable case also to camelCase
* Partially fix case variables in html.cpp
* Remove InterpolateAlignment
And some code improvements
* Replace the fixed value buffer with a std::string backing
* Fix tests that had no alignment info
These depended on the linear interpolation that I removed
* Remove arbitrary limits on tag and attribute names
This might also fix a bug caused by the eager lower casing of tag names, which could break <![CDATA , <style> and <script>
* Remove equals() in favour of operator==()
I trust the compiler can come up with better optimisations than I can.
* Expose std::strings instead of their data
Should save us some std::strlen() calls
* Add & remove headers and no-longer-defined functions from header files
* Remove all string buffers from xh_scanner
It now directly refers to either the input stream or constant strings
* Replace custom string_view with even lighter struct that's only used internally
To the outside world we just expose std::string_view
* Remove __builtin_sub_overflow for MSVC
* ABORT if trying to restore HTML when no alignment info is available
* Add test cases specifically for xh_scanner
Both good for testing regression, and as a little example/reference for what behaviour to expect from it.
* Add --html option to bergamot for tests
This should make it easier to have some integration tests for HTML input
* Add test and fix for empty inputs failing due to alignment check
Co-authored-by: Jerin Philip <jerinphilip@live.in>
* Updated translator configuration in wasm test page
- Added alignment: soft
* Set ResponseOptions::alignment to "true"
- Had to be set for html text translation to work
* Fix out-of-bounds error when determining alignment for whole word
If token at offset 0 was a continuation (which it always is, since the first word of a sentence does not start with a space) it would jump to (unsigned) -1 which is probably out of bounds.
* Don't segfault if alignment info is not available
When alignment info is requested, but model is missing `alignment: soft` you'd get empty alignment info for every target token.
* Partial fix for handling empty elements
This fixes a parse error when dealing with something like `<p>...<br></p>` or `...<br>` where there is no text after the last empty element. This also prevents losing empty elements in the source side of the translation. Empty elements are not yet transferred correctly to the target side.
* Fix formatting
* Enable importing optimized gemm module for wasm
- Updated emscripten generated JS code to
-- import and use the optimized gemm module when available, otherwise
use fallback gemm implementation
* Added logging for gemm implementation being used for wasm target
Sets a cache to operate for each sentence that a TranslationModel process
caching the corresponding marian::History for a {TranslationModel::Id, marian::Words}
key. Cache is thus shared across multiple TranslationModels bound to the lifetime
of a Service. Cache gracefully downgrades in the case of WebAssembly.
* Updated marian-dev submodule
* Import wasm gemm from a separate wasm module
- The fallback implementation of gemm is currently being imported dynamically
for wasm target
* Updated CI scripts and README to import GEMM from a separate wasm module
* Setting model config to int8shiftAlphaAll in wasm test page
For outbound translation, we require having multiple models in the
inventory at the same time and abstracting the "how-to-translate"
using a model out.
Reorganization: TranslationModel + Service. The new entity which
contains everything required to translate in one direction is
`TranslationModel`. The how-to-translate blocking single-threaded mode
of operation or async multi-threaded mode of operation is decoupled as
`BlockingService` and `AsyncService`. There is a new regression-test
using multiple models in conjunction added, also serving as
a demonstration for using multiple models in Outbound Translation.
WASM: WebAssembly due to the inability to use threads uses
`BlockingService. Bindings are provided with a new API to work with a
Service, and multiple TranslationModels which the client (JS extension)
can inventory and maintain. Ownership of a given `TranslationModel` is
shared while translations using the model are active in the internal
mechanism.
Config-Parsing: So far bergamot-translator has been hijacking marian's
config-parsing mechanisms. However, in order to support multiple models,
it has become impractical to continue this approach and a new
config-parsing that is bergamot specific is provisioned for
command-line applications constituting tests. The original marian
config-parsing tooling is only associated with a subset of
`TranslationModel` now. The new config-parsing for the library manages
workers and other common options (tentatively).
There is a known issue of: Inefficient placing of workspaces, leading to
more memory usage than what's necessary. This is to be fixed trickling
down from marian-dev in a later pull request.
This PR also brings in BRT changes which fix speed-tests that were
broken and also fixes some QE outputs which were different due to not
using shortlist.
Unifies quality estimation with an interface, refactors previously available
quality scores to fit this interface. Adds a new class of model with Logistic
Regression powering the predictions as an implementation of said interface.
QE now provides annotations on words using subwords to word rule-based
algorithms working with space characters.
QualityEstimation
-----------------
Implementations of QE are bound together by a `QualityEstimator`
Interface.
1. The log-probabilities from the machine-translation model re-interpreted
as quality scores are crafted as an implementation of QualityEstimator.
2. A Logistic-Regression based model is added. This class of models is
trained supervised with scores labeled by a human annotator.
Handcrafted features - number of words, log probs from MT model and
statistics over the sequence are used to generate the numeric features.
LogisticRegressor, Matrix (to hold features) are added.
The creation of an instance is switched by the `AlignedMemory` supplied
(be it loaded from the file-system or supplied as a parameter). An empty
AlignedMemory leads to quality scores from NMT while supplying weights
of a trained logistic-regression model in binary format as the contents
lead to an additional pass through the said model to provide more
refined scores.
Both the above now transform subwords into "words" using a heuristic
algorithm, scanning for spaces. This allows the client to work with "words"
to denote quality instead of subwords, as the former is more sensible to
the user.
Testing
-------
1. BRT now has two new test apps to check the QE outputs in text
(covers subword to words) and numbers domain (covers quality scores).
These are tested with en-et models for which QualityEstimation is
available now, on a new input to avoid architecture/compiler issues.
2. Unit test for LogisticRegression model is added.
Docs
----
Doxygen now supports MathJax properly to render explanations for
Logistic Regressions' reductions in place to make computation more
efficient correctly.
Co-authored-by: Felipe C. Dos Santos <felipe.santos.k@gmail.com>
Co-authored-by: Jerin Philip <jerinphilip@live.in>