Provides an additional constructor which takes care of the bundle
loading inside the boundary of the source here, when a configuration
file is supplied from a client like translateLocally or python bindings.
Once the config file is read, we have access to the information required
to construct the MemoryBundle.
- The command-line application supplied from here, app/bergamot is
configured to use the fast-load path now.
- Changes to binary-loading additionally revealed a bug in the
example-run script used in docs and tied to CI and the fix is
included.
- Shortlist is made optional in the memory bundle, making changes to
getModelMemoryFromConfig.
Fixes#304.
Fixes#306.
See also: XapaJIaMnu/translateLocally#82.
* CLI Rework
* Consolidate common tests, template specialize CLI
* Remove remnant cache stuff
* [BRT]: Run BRT with new cli
* Formalizing bridge
* Removing stuff from parsing and moving to TestSuite
* Template includes, everything consolidating at tests
* Inlining readFromStdin
* Removing unnecessary headers
* Checking in template implementation which was missing
* Sane defaults, some catches at BRT
* BRT: Install fixes
* Updating marian-dev to point to main
* Removing the enum indirection, using strings at one place, directly
* Fix typo;
* [BRT] test blocking service via native
* Conservative defaults for workers and cache-mutex buckets in AsyncService
* Create proper barriers for cmdline app
* Build failure fixes
* Moving common, common-impl to a familiar structure
* Binary reorganization: async, blocking, wasm
- async tests AsyncService
- blocking tests BlockingService
- wasm arranges tests for things that are Mozilla requirements. eg:
- bytearray
- multiple sentences in same translate request workflow.
* [brt] updates to adapt to cli rework
* [brt] updates to adapt to cli rework, all working
* Empty commit, sync brt online and run GitHub CI
* Switch for parser to have multiple mode or not
* [brt]: Fix for --bergamot-mode being removed from CLI app
* [brt]: Fix for --bergamot-mode being removed from CLI app
* [brt]: Removing remnant faithful translation test from blocking/
* Remove InterpolateAlignment
And some code improvements
* Replace the fixed value buffer with a std::string backing
* Fix tests that had no alignment info
These depended on the linear interpolation that I removed
* Remove arbitrary limits on tag and attribute names
This might also fix a bug caused by the eager lower casing of tag names, which could break <![CDATA , <style> and <script>
* Remove equals() in favour of operator==()
I trust the compiler can come up with better optimisations than I can.
* Expose std::strings instead of their data
Should save us some std::strlen() calls
* Add & remove headers and no-longer-defined functions from header files
* Remove all string buffers from xh_scanner
It now directly refers to either the input stream or constant strings
* Replace custom string_view with even lighter struct that's only used internally
To the outside world we just expose std::string_view
* Remove __builtin_sub_overflow for MSVC
* ABORT if trying to restore HTML when no alignment info is available
* Add test cases specifically for xh_scanner
Both good for testing regression, and as a little example/reference for what behaviour to expect from it.
* Add --html option to bergamot for tests
This should make it easier to have some integration tests for HTML input
* Add test and fix for empty inputs failing due to alignment check
Co-authored-by: Jerin Philip <jerinphilip@live.in>
For outbound translation, we require having multiple models in the
inventory at the same time and abstracting the "how-to-translate"
using a model out.
Reorganization: TranslationModel + Service. The new entity which
contains everything required to translate in one direction is
`TranslationModel`. The how-to-translate blocking single-threaded mode
of operation or async multi-threaded mode of operation is decoupled as
`BlockingService` and `AsyncService`. There is a new regression-test
using multiple models in conjunction added, also serving as
a demonstration for using multiple models in Outbound Translation.
WASM: WebAssembly due to the inability to use threads uses
`BlockingService. Bindings are provided with a new API to work with a
Service, and multiple TranslationModels which the client (JS extension)
can inventory and maintain. Ownership of a given `TranslationModel` is
shared while translations using the model are active in the internal
mechanism.
Config-Parsing: So far bergamot-translator has been hijacking marian's
config-parsing mechanisms. However, in order to support multiple models,
it has become impractical to continue this approach and a new
config-parsing that is bergamot specific is provisioned for
command-line applications constituting tests. The original marian
config-parsing tooling is only associated with a subset of
`TranslationModel` now. The new config-parsing for the library manages
workers and other common options (tentatively).
There is a known issue of: Inefficient placing of workspaces, leading to
more memory usage than what's necessary. This is to be fixed trickling
down from marian-dev in a later pull request.
This PR also brings in BRT changes which fix speed-tests that were
broken and also fixes some QE outputs which were different due to not
using shortlist.
* Change ResponseBuilder to accept callback
Breaks things everywhere, now we follow the compiler to fix and convert
the std::future -> callback.
* More std::future -> callback
* std::future out of service.{h,cpp}
* compile is working, so is callback
* Some reshuffling of args
* Fixing merge error
* Fixing signature conflicts out of merge
* Fixing that test duct-taping future
* Minor adjustment to get that future back
* Add documentation for the new callback function
* Applying clang-format after update
* Using default responseOptions
* Remove future references from documentation
* translateMultiple only for WASM (#177)
* BRT: update to main; fresh-failures hopefully
* Converting test translateFromStdin to use callback
* BRT: Add fresh #native and #wasm tags
* future from promise, fix error
* Adding #native to GitHub CI
Co-authored-by: Nikolay Bogoychev <nheart@gmail.com>
* Partial test applications
Previously service-cli was used to generate output and accomplish
regression testing for all of: (1) translated-text (2) alignment tokens
+ scores (3) quality scores (4) indirectly annotation and tokenizations.
The --mode native now only outputs a faithful to source translated text
of the input source on stdin.
Test apps are separated into testing only individual functionalities.
This can help in independently testing ssplit-cpp, quality-scores for
the quality estimation implementation etc.
Separating numbers and text have the advantage of being able to compare
one with tolerance using BLEU (text) and some allowed error-rates
(numbers).
* Removing #mac tag
* Moving test apps to src/tests
* Tests are always on for CI
Unit tests are turned off looking for WASM_COMPATIBLE_SOURCES.
* Fixing WASM_COMPATIBLE_SOURCE -> USE_WASM_COMPATIBLE_SOURCE
* Workaround for now; CMakeLists.txt horrors are starting to bite
* BRT: use bergamot-test instead of bergamot now
* This should fix issues: CMakeLists.txt has so many paths
* Casing to camelCase and removing legacyServiceCli
* removing leftover service-cli declaration, some doc updates
* #pragma once is starting to look easier
* All the more reasons to do #pragma once
* Updating marian-dev with intgemm::kCPU print, resolved from INTGEMM_CPUID
* BRT: Use --gemm-highest-arch instead of python script
* Adding intgemm resolve here, where always(?) have intgemm on?
* intgemm-resolve in default binary directory
* BRT: Update to use intgemm-resolve
* marian-dev: Reset to without --gemm-highest-precision
Co-authored-by: Kenneth Heafield <kpu@users.noreply.github.com>
* Collapsing executables
* Adding new test executable
* Deleting old executable sources
* Updating brt to operate with modes
* cli-framework -> cli
* Updating workflows to check for bergamot instead of bergamot-translator-app
* Adding documentation
* Making fn pure virtual
* Shuffling apps into app namespace, alongside class documentation
* Include app folder in documentation
* BRT update service-cli -> native
* parser.h: service-cli -> native
* Updates to marian-integration.md
* Cleanup: Remove templates, interface proper
* change 4 to 2 cores for build instructions
* service-cli -> native
* Commenting the string constructor explanation
* Not doing halfway interface / inheritance
* Nick hates state, let's try this one
* Revert "Nick hates state, let's try this one"
This reverts commit e56db9f474.
* class -> struct before trying std::function stuff
* oop -> functional?
* Hints on what is happening
* app::ftable -> app::REGISTRY
* We have if-else and functions now.
And we won't have test apps.
* Doc linking to usage examples in brt
* Remove unordered_map
* Documentation updates
* Fix warning
* first attempt to enable vocabs pass as byte arrays
* pass vocabs bytes as AlignedMemory
* add vocabIndices to avoid double loading
* small fix on parameter names and documentation
* fix windows build plus tiny update on documentation
* update marian-dev submodule
* move validate model bytearray in BatchTranslator
* small refactors on validateBinaryModel()
* switch vocab memories to std::vector<marian::Ptr<AlignedMemory>>
* update marian-dev submodule
* replace marian::Ptr to std::shared_ptr for vocab memories
* add note for vocab memories
* Adding bytearray option
* collapse intermediate for bytearray apps
* Removing service-cli-bytearray
* Removing the bergamot bytearray app
* Bumping updates to brt collapsing apps
* Reasonable defaults and hard check when cmd enabled
* Update documentation for flags
* Bump brt with MKL check and skip
* Bumping BRT with MKL_FOUND instead of USE_MKL
* Bumping BRT with no mkl enforce
* Bumping BRT with ssse3 output
* Let's try disabling OpenBLAS
* Trying to disable apple accelerate
* Using WASM compatible BLAS can enable intgemm
* Adding a CMake -L to see what exactly is the diff
* Revert "Let's try disabling OpenBLAS"
This reverts commit 9a6b9bc53b.
* Revert "Using WASM compatible BLAS can enable intgemm"
This reverts commit 936a592e18.
* Restricting mac tests through tags and on GitHub CI
* Using only check-bytearray
* Bumping BRT with change of default behaviour
* Safe transfer of bindings through typedefs
* Removing Translation* files and bringing in counterparts
* Remove previously commented out code
* Removing commented out include
* Absorb Translation* documentation
Co-authored-by: abhi-agg <66322306+abhi-agg@users.noreply.github.com>
* Removes vocabs and propogates fixes for breaks
* Prettify diff: Undoing comment shuffles due to merge conflict edits
* 20% of time actual work, 80% prettifying diff
* Histories members -> poof!
We however have Histories in constructor, which we will remove out of the way
soon.
Co-authored-by: Kenneth Heafield <kpu@users.noreply.github.com>
Contains "hack" that must go immediately by editing TranslationModel, to come in following commit.
* add shortlist_memory and update service-cli-bytearray test
* update marian-dev
* address review comments
* fix ccompliation and tests failures and further address review comments
* small update on marian-dev (based on browsermt/marian-dev PR#28)
* update marian-dev with upstream
* code refactoring according to review
* fix marian-dev submodule conflicts
* switch MemoryGift to AlignedVector
* copy aligned.h from kpu/intgemm for AlignedVector
* changes based on memory ownership and AlignedVector
* fix BatchTranslator inits
* small fixes according to review comments
* update submodule marian-dev to master
* update submodule marian-dev with upstream
Co-authored-by: Kenneth Heafield <kpu@users.noreply.github.com>
* Updated marian-dev submodule
- cmake changes required after the submodule update
* Added workflows for building custom marian on mac and ubuntu
* Renamed cmake option
- Renamed USE_WASM_COMPATIBLE_SOURCES to USE_WASM_COMPATIBLE_SOURCE
- Use proper compile defnitions
* Draft adjustments to API
* Adjustments to docs
* Let's call the word + sentence ranges annotations
* Editing confusing comment on size()
* Fixing compilation for template adjustments for SentenceRanges
* string_view template hacks
This commit shifts AnnotatedBlob into a templated type and gets the
troubled part to compile. All to manage absl::string_view and
std::string_view.
Objective: marian::bergamot stays C++ 11 to pluck and put in marian
code, bergamot-translator somehow flexes C++17. Simplify development in
one place.
* Fixing the wiring: Gets source to build
Runtime errors exist, but AnnotatedBlobs are consistent.
* Bugfix: Matching old-state after factoring AnnotatedBlob in
* Removing vocabs_ from Response.
(For the umpteenth time).
* Alignment API ready in marian::bergamot::Response
* Wiring alignments upto TranslationResult
* Adjustment to get alignments; bergamot-translator-app has alignments available
* Accessing words instead of Ids
This code sets up access of word string_views from annotations instead
of printing Ids. However, we have segfault. This is likely due to
targetRanges not being set, pending from
https://github.com/browsermt/bergamot-translator/issues/25.
Could also be a rogue EOS token which we're filtering for in string_view
annotations, but not so in alignments.
* Switching to browsermt/marian-dev@jp/decode-string-view for targetTokenRanges
* Target word byte range annotations available
Issues corresponding to #25 should be resolved. There is still a
segfault. Could be due to EOS. Pending investigation.
* Bugfix: Tokens for alignments are now through.
Was not EOS.
* browsermt/marian-dev@master
ByteRange changes work downstream and has been merged to master.
Updating submodule to point to master.
* Style and documentation enhancements: response.cpp
* Style and documentation enhancements: TranslationResult.h
* Descriptions for SentenceRanges templating
* Switching to marian-dev@wasm-sync
* AnnotatedBlob can be copy-ctord/copy-assigned
* TranslationResult: Empty ctor + WASM Bindings
Allows empty construction of TranslationResult. Using this empty
constructor, WASM bindings are adjusted. Unsure of the results, maybe
@abhi-agg can test.
* Cosmetic: SentenceRangesT -> Annotation
- SentenceRangesT is renamed to AnnotationT;
- Further comments to explain heavily templated files.
* Response: Cleaning up unused members and adding docs
* Adding quality scores - attempt
* Stub QualityScores
This adjustment adds capability to get "scores", which should
potentially indicate how confident (at least relative in a
target-sentence) should be. This enables writing the code forward for
TranslationResult, and an example quality-score people can be pointed
at.
- These are not between [0,1] yet.
- In addition, guards to check out-of-bounds access have been placed so
illegal accesses are caught early on during development.
* Removing token debug statements
* Reworking Annotation without templates
https://github.com/mozilla/bergamot-translator/issues/8 provides
ByteRanges.
- This ByteRange data-type is used in Annotation and converted
to marian::string_view(=absl::string-view) on demand.
- Since Annotation[using ByteRange] is not bound to anything else, it
can be unit tested. A unit test is added (originally to test
independently for integration after).
- Annotation with ByteRange is now propogated across marian::bergamot
and functionality matched to how it was previously working.
This eliminates the string-view conversion and template code.
* Nit: Removing std::endl flushes
* Bring TranslationResult and Response closer
Helps https://github.com/browsermt/bergamot-translator/issues/53.
In preparation , the data-export types for Quality and Alignment are
pushed down to Response from TranslationResult and computed during
construction. This brings TranslationResult closer to Response, paving
way to avoid having two TranslationResults.
histories_ only remain for marian-decoder replacement usage, which can
be removed in a separate PR.
* Clean up hacks originally added for a unit-test to compile
* Moving Annotation functions to cpp and documenting header file
* Shifting alignments, qualityScore testing capability into main-mts
* Restore Unified API files to previous state
* Adaptations to fix Response with Quality, Alignments to connect to old Unified API
* Missing reset on TranslationResultBindings
* Cleaning up Response documentation to reflect newer code
* Minor adjustments to get build back after main sync
* Marian seems to make available Catch somehow
* Disable COMPILE_BERGAMOT_TESTS for WASM
* Add COMPILE_BERGAMOT_TESTS as a CMakeDependent option
* Use the COMPILE_TESTS flag instead to skip macos.yml
* Trigger unit-tests on GitHub runners for Annotation
* Reordering enable_testing() to before inclusion of test directory
* doc constructs required to operate with alignments
Documents with doxygen compatible documentation for Response,
AnnotatedBlob, Annotation, ByteRange.
Incorporates doxygen compatible documentation for
* Updates ByteRange consistent with general C++
Also little documentation enhancements in the process.
* Updating marian-dev@9337105
* Copy-paste documentation because lazy
* Turn off autoformat and manually edit to fix style changes
* AnnotatedBlob -> AnnotatedText; blob -> text
* text.text in test app renamed
* text of text -> blob of text in places of documentation
* Merging two Services
* Moving stop() logic to destructor
* We have WITH_PTHREADS back
* string based constructor on Service
* Removing now empty service_base.* files
* Hiding away pcqueue_ construction
Ugliest ifdefs I have done in my life.
* Another ifdef to hide pcqueue header file
* Missing semicolons in WITH_PTHREADS path
* Fixing async_translate residue argument from copy
* Adding comments
* Initialize batchtranslator only at one place
To reduce tax for bytebuffer loads, initialize batchtranslator only at
one place.
* \#ifdef WITH_PTHREADS -> #ifndef WASM_HIDE_THREADS
Sane platform (non WASM) is default. This truly only hide-threads from
compilation path and not switch unswitch pthreads (-lpthread).
* Review comments: Rearranging destructor, fix wrong comment
* Move loadVocabularies to service.cpp and put in anonymous namespace
* Prettifying diff: Removing unwanted empty lines
* Indicate in comments multithreaded has numWorkers translators
* Typo fix: bergamot_translator -> bergamot-translator
* Safety guards to avoid pcqueue illegal init
* Add WASM_HIDE_THREADS as a global WASM_COMPILE_FLAG
* Compile Defs: WASM_HIDE_THREADS -> __EMSCRIPTEN__
* Removing dead CMakeLists.txt code following __EMSCRIPTEN__
* Compile defs: __EMSCRIPTEN__ -> WASM
* Switch to wasm branch for this example
* Load marian model from a byte array
* Sanitise executable names
* Change marian branch
* Update marian branch that loads binary models
* Example of loading model as a byte array
* Add the byte array loading files
* Die on misaligned memory
* Remove the unused argument
* Allow loading without a ptr parameter so that we don't break emc workflow
Through inheritance, a non-threaded and multithreaded Service are
created, both derived of the same ServiceBase class which holds the
common elements.
In preparation to solve SIGSEGV in #41. First inspections gave aborts in
thread part, and repeated SIGSEGV's in lock-policy's of shared_pointers
even in non-threaded paths.
Solving this first, to avoid ifdef or tricky paths. The non-threaded
implementation is not included in WASM builds at all, by separating out
the single-threaded logic. DRY is achieved through inheritance and
operator overloading.
To avoid confusion, this commit renames
marian::bergamot::TranslationResult -> marian::bergamot::Response.
Usages of marian::bergamot::TranslationResults are updated across the
source to be consistent with the change and get source back working.
marian-TranslationResult has more guards in place. Switching to a
construction on demand model for sentenceMappings. These changes
propogate to bergamot translation results.
Integration broke with the change in marian's internals, which are
updated accordingly to get back functionality.
Changes revealed a few bugs, which are fixed:
- ConfigParser already discovered in wasm-integration
(a06530e92b).
- Lambda captures and undefined values in DeviceId
Updates marian-dev and ssplit submodules to point to the upstream
commits which implements the following:
- marian-dev: encodeWithByteRanges(...) to get source token byte-ranges
- ssplit: Has a trivial sentencesplitter functionality implemented, and
now is faster to benchmark with marian-decoder.
This enables a marian-decoder replacement written through ssplit in this
source to be benchmarked constantly with existing marian-decoder.
Nits: Removes logging introduced for multiple workers, and respective
log statements.
Requirement for string_view is the original source string be transferred
all the way from input to service to back to TranslationResult. This
constraint was violated in several places by means of existence of a
copy-constructor. The issue is fixed by deleting copy and assignment
constructors in marian::bergamot::TranslationResult and
UnifiedAPI::TranslationResult, which demonstrated a few occurances of
the same. Replaced the same with move semantics. In addition, future is
set and get using move semantics at the moment. Default
move-constructor didn't seem to be working, so they're made explicit for
TranslationResults.
This commit additionally packs a few deletions and improvements made to
improve structure (textops.cpp, batcher.cpp) along the process of
inspecting and fixing the garbled outputs. They are choose to be kept,
in the interest of time, against a prettified atomic commit engineering.
Combinations of the following commits in jp/string-view-bug
[acfc92 78a588 12d91b 00a277 919e2f 9d3a46 b7e39b 18f67b bf667c]
Using std::string for config. Now capable of launching marian translator
through API interface. There's a sketchy workaround to convert a string
config to marian::Options, with an added note.
Only the bergamot-translator library should be linked to main target
Any other library (marian ${MARIAN_CUDA_LIB} ${EXT_LIBS} ssplit
pcrecpp.a pcre.a) should be linked to bergamot-translator target inside
src/translator folder.
