* Improved script that patches wasm artifacts to enable wormhole
- Made the regex pattern ignore multiple whitespaces b/w words of
the matching pattern
* Fix for loading EN->DE vocabularies in wasm test page
- Loading vocabularies for EN->DE was failing because of
the new structure of bergamot-models
* Safe transfer of bindings through typedefs
* Removing Translation* files and bringing in counterparts
* Remove previously commented out code
* Removing commented out include
* Absorb Translation* documentation
Co-authored-by: abhi-agg <66322306+abhi-agg@users.noreply.github.com>
* Update marian-dev to the newest mac version
* Attempt windows workflow
* force workflow rerun
* Separate id
* Attempt 3 at github action
* Marian dev submodule now compiles with apple clang
* Updated ssplit version to something more recent
* Attempt to fix compile on wasm
* Do not compile subproject tests
* Fix emscripten compilation on Mac
* 99% on the way to windows compile
* Try with a different generator
* Build release not debug
* Revert CMakeLists.txt hacks
* Fix sse2 compilation failure
* MSVC settings for WIN32
* Add nodefaultlib LIBCMT
* Do not compile ssplit.cpp as it contains sys/mman.h
* Revert ab56b9aa4f
* Update paths
* Set the build type to release if not set previously
* Attempt to build release with the windows workflow
* Attempt 5 at VS studio release build
* Attempt 6 at getting release build on MSVC generator
* The windows build is debug at the moment...
* fix ssplit for ubuntu 16.04
* Fix compilation with clang
* Compile on ubuntu16.04
* Explain what is going on
* Updated ssplit and workflow
* Bindings to load model and shortlist files as bytes
* Modified wasm test page for byte based loading of files
* Updates wasm README for byte loading based usage of TranslationModel
* Control validating the config options via a boolean flag
- parseOptions() function now validates the parsed options
based on the validate argument
* Minor syntactic fix
- AlignedMemory is AlignedVector<char> now instead of AlignedVector<const void*>
- This solves the issue of allocating 8x of the actual required memory for
loading files as bytes
Adds regression-tests to the workflow for native minimal/custom marian and full builds.
Co-authored-by: abhi-agg <66322306+abhi-agg@users.noreply.github.com>
* Changing Annotation to adhere to [begin, end)
* Stronger unit tests on sentences + num words, num sentences
* Hotfix with empty string view from EOS
* No more absolving empty-sentence; Added tests now defined behaviour
* Uncommenting important section in unit test
* Ensure empty string view default, beginning at end so marker points
* Further strengthen and comment unit-tests, mark exactly where empty sentence is happening
* Review comments: Dummy sentence + docs
- What should be a simple fast accessor is turning into compute.
Normally the way to deal with this, for better or worse, is to put 0 at
the beginning of sentenceEndIds_. (Putting 0 at the beginning of
sentenceEndIds_)
- Indices into what? Mentioned to be flatByteRanges_.
* Documentation updates
* More changes to docs
Co-authored-by: abhi-agg <66322306+abhi-agg@users.noreply.github.com>
* Removes vocabs and propogates fixes for breaks
* Prettify diff: Undoing comment shuffles due to merge conflict edits
* 20% of time actual work, 80% prettifying diff
* Histories members -> poof!
We however have Histories in constructor, which we will remove out of the way
soon.
Co-authored-by: Kenneth Heafield <kpu@users.noreply.github.com>
Contains "hack" that must go immediately by editing TranslationModel, to come in following commit.
* add shortlist_memory and update service-cli-bytearray test
* update marian-dev
* address review comments
* fix ccompliation and tests failures and further address review comments
* small update on marian-dev (based on browsermt/marian-dev PR#28)
* update marian-dev with upstream
* code refactoring according to review
* fix marian-dev submodule conflicts
* switch MemoryGift to AlignedVector
* copy aligned.h from kpu/intgemm for AlignedVector
* changes based on memory ownership and AlignedVector
* fix BatchTranslator inits
* small fixes according to review comments
* update submodule marian-dev to master
* update submodule marian-dev with upstream
Co-authored-by: Kenneth Heafield <kpu@users.noreply.github.com>
* Updated marian-dev submodule
- cmake changes required after the submodule update
* Added workflows for building custom marian on mac and ubuntu
* Renamed cmake option
- Renamed USE_WASM_COMPATIBLE_SOURCES to USE_WASM_COMPATIBLE_SOURCE
- Use proper compile defnitions
* Draft adjustments to API
* Adjustments to docs
* Let's call the word + sentence ranges annotations
* Editing confusing comment on size()
* Fixing compilation for template adjustments for SentenceRanges
* string_view template hacks
This commit shifts AnnotatedBlob into a templated type and gets the
troubled part to compile. All to manage absl::string_view and
std::string_view.
Objective: marian::bergamot stays C++ 11 to pluck and put in marian
code, bergamot-translator somehow flexes C++17. Simplify development in
one place.
* Fixing the wiring: Gets source to build
Runtime errors exist, but AnnotatedBlobs are consistent.
* Bugfix: Matching old-state after factoring AnnotatedBlob in
* Removing vocabs_ from Response.
(For the umpteenth time).
* Alignment API ready in marian::bergamot::Response
* Wiring alignments upto TranslationResult
* Adjustment to get alignments; bergamot-translator-app has alignments available
* Accessing words instead of Ids
This code sets up access of word string_views from annotations instead
of printing Ids. However, we have segfault. This is likely due to
targetRanges not being set, pending from
https://github.com/browsermt/bergamot-translator/issues/25.
Could also be a rogue EOS token which we're filtering for in string_view
annotations, but not so in alignments.
* Switching to browsermt/marian-dev@jp/decode-string-view for targetTokenRanges
* Target word byte range annotations available
Issues corresponding to #25 should be resolved. There is still a
segfault. Could be due to EOS. Pending investigation.
* Bugfix: Tokens for alignments are now through.
Was not EOS.
* browsermt/marian-dev@master
ByteRange changes work downstream and has been merged to master.
Updating submodule to point to master.
* Style and documentation enhancements: response.cpp
* Style and documentation enhancements: TranslationResult.h
* Descriptions for SentenceRanges templating
* Switching to marian-dev@wasm-sync
* AnnotatedBlob can be copy-ctord/copy-assigned
* TranslationResult: Empty ctor + WASM Bindings
Allows empty construction of TranslationResult. Using this empty
constructor, WASM bindings are adjusted. Unsure of the results, maybe
@abhi-agg can test.
* Cosmetic: SentenceRangesT -> Annotation
- SentenceRangesT is renamed to AnnotationT;
- Further comments to explain heavily templated files.
* Response: Cleaning up unused members and adding docs
* Adding quality scores - attempt
* Stub QualityScores
This adjustment adds capability to get "scores", which should
potentially indicate how confident (at least relative in a
target-sentence) should be. This enables writing the code forward for
TranslationResult, and an example quality-score people can be pointed
at.
- These are not between [0,1] yet.
- In addition, guards to check out-of-bounds access have been placed so
illegal accesses are caught early on during development.
* Removing token debug statements
* Reworking Annotation without templates
https://github.com/mozilla/bergamot-translator/issues/8 provides
ByteRanges.
- This ByteRange data-type is used in Annotation and converted
to marian::string_view(=absl::string-view) on demand.
- Since Annotation[using ByteRange] is not bound to anything else, it
can be unit tested. A unit test is added (originally to test
independently for integration after).
- Annotation with ByteRange is now propogated across marian::bergamot
and functionality matched to how it was previously working.
This eliminates the string-view conversion and template code.
* Nit: Removing std::endl flushes
* Bring TranslationResult and Response closer
Helps https://github.com/browsermt/bergamot-translator/issues/53.
In preparation , the data-export types for Quality and Alignment are
pushed down to Response from TranslationResult and computed during
construction. This brings TranslationResult closer to Response, paving
way to avoid having two TranslationResults.
histories_ only remain for marian-decoder replacement usage, which can
be removed in a separate PR.
* Clean up hacks originally added for a unit-test to compile
* Moving Annotation functions to cpp and documenting header file
* Shifting alignments, qualityScore testing capability into main-mts
* Restore Unified API files to previous state
* Adaptations to fix Response with Quality, Alignments to connect to old Unified API
* Missing reset on TranslationResultBindings
* Cleaning up Response documentation to reflect newer code
* Minor adjustments to get build back after main sync
* Marian seems to make available Catch somehow
* Disable COMPILE_BERGAMOT_TESTS for WASM
* Add COMPILE_BERGAMOT_TESTS as a CMakeDependent option
* Use the COMPILE_TESTS flag instead to skip macos.yml
* Trigger unit-tests on GitHub runners for Annotation
* Reordering enable_testing() to before inclusion of test directory
* doc constructs required to operate with alignments
Documents with doxygen compatible documentation for Response,
AnnotatedBlob, Annotation, ByteRange.
Incorporates doxygen compatible documentation for
* Updates ByteRange consistent with general C++
Also little documentation enhancements in the process.
* Updating marian-dev@9337105
* Copy-paste documentation because lazy
* Turn off autoformat and manually edit to fix style changes
* AnnotatedBlob -> AnnotatedText; blob -> text
* text.text in test app renamed
* text of text -> blob of text in places of documentation
* A script to patch the wasm artifacts to use wormhole via
APIs that instantiate WASM module
* Updated README
* Load just production ready models
* Shallow clone bergamot-models repo since it has such a large history
* Improved wasm test_page
- test page can load all 5 language pairs
- Use intgemm.alpha* models
* Refactor the code that patches wasm artifacts to enable wormhole
Co-authored-by: Andre Natal <anatal@gmail.com>
Co-authored-by: Motin <motin@motin.eu>
* Merging two Services
* Moving stop() logic to destructor
* We have WITH_PTHREADS back
* string based constructor on Service
* Removing now empty service_base.* files
* Hiding away pcqueue_ construction
Ugliest ifdefs I have done in my life.
* Another ifdef to hide pcqueue header file
* Missing semicolons in WITH_PTHREADS path
* Fixing async_translate residue argument from copy
* Adding comments
* Initialize batchtranslator only at one place
To reduce tax for bytebuffer loads, initialize batchtranslator only at
one place.
* \#ifdef WITH_PTHREADS -> #ifndef WASM_HIDE_THREADS
Sane platform (non WASM) is default. This truly only hide-threads from
compilation path and not switch unswitch pthreads (-lpthread).
* Review comments: Rearranging destructor, fix wrong comment
* Move loadVocabularies to service.cpp and put in anonymous namespace
* Prettifying diff: Removing unwanted empty lines
* Indicate in comments multithreaded has numWorkers translators
* Typo fix: bergamot_translator -> bergamot-translator
* Safety guards to avoid pcqueue illegal init
* Add WASM_HIDE_THREADS as a global WASM_COMPILE_FLAG
* Compile Defs: WASM_HIDE_THREADS -> __EMSCRIPTEN__
* Removing dead CMakeLists.txt code following __EMSCRIPTEN__
* Compile defs: __EMSCRIPTEN__ -> WASM
* Switch to wasm branch for this example
* Load marian model from a byte array
* Sanitise executable names
* Change marian branch
* Update marian branch that loads binary models
* Example of loading model as a byte array
* Add the byte array loading files
* Die on misaligned memory
* Remove the unused argument
* Allow loading without a ptr parameter so that we don't break emc workflow
- test page can load all 5 language pairs
- Use intgemm.aplha* models
- start_server.sh script automatically enable simdwormhole via
APIs that instantiate WASM module
