Allow the self-closing `/>` end for void tags. For non-void tags these
were already "allowed" due to how the HTML parser works, but for
elements where they actually occur, like `<br/>`, they caused a parse
error. Support for them was not implemented since we only expect valid
HTML5, e.g. the output of Firefox' Element.innerHTML.
Use case: TranslateLocally uses Qt's HTML representation of rich text.
That HTML uses self-closing tags like `<meta .../>` and `<br/>`.
Implementing a string replace operation that would only match these
elements without parsing HTML is tricky. Fixing it in
bergamot-translator is not.
Implementation: Currently `<img>` is marked as a void tag (an element
which cannot have children or text, and therefore treated differently.
Since void tags normally have no close tag, they are treated as
immediately closed. The HTML parser we use reads `<img/>` as
`<img></img>` which thus causes a problem since now we close an element
that was never open, to begin with.
This fix ignores the `TT_TAG_END` token from the parser when the tag
name is that of a void tag.
Translates a text from source-language to target-language through a
pivot-language. Effectively runs models in series, while having the
following additional benefits compared to when `Service::translate(...)`
would be used repeatedly.
1. Consistency in sentences between source and target. Consistent
creation of the alignment matrix for use in downstream tasks like
tag-translation.
2. Efficient sentence-splitting (does not sentence-split twice, creating
inconsistencies).
3. The `Response` generated can be used as if it were coming through
`translate(...)`, eliminating any need for additional code for clients
in JS or python or C++.
`AsyncService::pivot(...)` is provisioned for C++ multi-threaded setting
and `BlockingService::pivotMultiple(...)` provisioned for blocking
use-case targeted at WebAssembly.
# [BRT]: Test additions, accompanying fixes
For `AsyncService` for a test-case involving of en->es, es->en (same
vocabulary, another one might be more coverage but is too much work).
1. Asserts the Alignment generated after pivoting is a probability
distribution over source tokens given target.
2. Outputs the sentences going from en->en, which should stay consistent
over continuous development to ensure nothing breaks.
3. An accuracy minimum of 70% of token matches from source to target
calibrated on the standard bergamot input text is additionally present,
ensuring that the English tokens at start and end match exactly.
# HTML Pipeline
This PR reworks the HTML translation pipeline to be outside
response-construction via callbacks.
* Convert marian-integration markdown to rst
* Convert native run into a script, include in rst
* Check with CI that the native running example works without fail
* Make stats hits misses atomic to guard when mutex has multiple buckets
* Use compile time switch for cache-stats-collection bound to COMPILE_TESTS cmake variable
* -DENABLE_CACHE_STATS on if COMPILE_TESTS otherwise optional
* Make stats() call without enabling build fatal abort
* Turn logging off by default, allow turning on via config/cmdline
* No need to store config in member variable if things are decided at construction time
- Robust artifact presence check
- Variable name refactoring
- Storing only those artifacts that are required
- Remove commit sha from the names of the Github Releases
- Use BERGAMOT_VERSION file contents for Git Tag names
* Fix test case
This should now be implemented
* Remove FilterEmpty
This path wasn't used anymore anyway, empty tags just got their own spans, and never reached the stack.
* Insert skipped empty source spans into target HTML
Also refactor variable names to better match their contents and be more consistent with each other.
This implementation passes all test cases, finally!
* Fix remaining style changes
* Move HTML formatting to its own section
That code had become exact copies in three different places
* CLI Rework
* Consolidate common tests, template specialize CLI
* Remove remnant cache stuff
* [BRT]: Run BRT with new cli
* Formalizing bridge
* Removing stuff from parsing and moving to TestSuite
* Template includes, everything consolidating at tests
* Inlining readFromStdin
* Removing unnecessary headers
* Checking in template implementation which was missing
* Sane defaults, some catches at BRT
* BRT: Install fixes
* Updating marian-dev to point to main
* Removing the enum indirection, using strings at one place, directly
* Fix typo;
* [BRT] test blocking service via native
* Conservative defaults for workers and cache-mutex buckets in AsyncService
* Create proper barriers for cmdline app
* Build failure fixes
* Moving common, common-impl to a familiar structure
* Binary reorganization: async, blocking, wasm
- async tests AsyncService
- blocking tests BlockingService
- wasm arranges tests for things that are Mozilla requirements. eg:
- bytearray
- multiple sentences in same translate request workflow.
* [brt] updates to adapt to cli rework
* [brt] updates to adapt to cli rework, all working
* Empty commit, sync brt online and run GitHub CI
* Switch for parser to have multiple mode or not
* [brt]: Fix for --bergamot-mode being removed from CLI app
* [brt]: Fix for --bergamot-mode being removed from CLI app
* [brt]: Removing remnant faithful translation test from blocking/
* Fix bug in HasAlignments check
When fixing it to allow empty sentences, it no longer caught misconfigured models. I've added a test that triggers this scenario, and a fix in HasAlignments for it.
* Add more unit tests for xh_scanner
Trying to increase that code coverage to 100%
* Add test for whitespaces around attributes
* Make accessing value(), attr_name() and tag_name() at the wrong time safer
* Fix bug in <style> and <script> parsing
The end tag was never found
* Fix parsing of mix of valueless and quoteless attributes
* Sync list of void tags with Firefox' implementation of outerHTML and innerHTML
Also lets use their name for it: IsVoidTag instead of IsEmptyElement. Empty was a bit ambiguous.
* Bring back support for processing instructions support in xh_scanner
I noticed in https://searchfox.org/mozilla-central/source/dom/base/nsContentUtils.cpp#8961 that these can be produced by innerHTML under some circumstances.
* More permanent link
* Use CamelCase for the internal functions I added
* Rename *_PI to *_PROCESSING_INSTRUCTION
Your IDE will do the typing for you anyway
* Match symbol naming of the rest of code base
CapitalCase for classes, camelCase for functions, snake_case for variables still.
* Missed one 😴
* Change xhscanner's variable case also to camelCase
* Partially fix case variables in html.cpp
* Remove InterpolateAlignment
And some code improvements
* Replace the fixed value buffer with a std::string backing
* Fix tests that had no alignment info
These depended on the linear interpolation that I removed
* Remove arbitrary limits on tag and attribute names
This might also fix a bug caused by the eager lower casing of tag names, which could break <
