* Updated marian-dev submodule
- cmake changes required after the submodule update
* Added workflows for building custom marian on mac and ubuntu
* Renamed cmake option
- Renamed USE_WASM_COMPATIBLE_SOURCES to USE_WASM_COMPATIBLE_SOURCE
- Use proper compile defnitions
* Switch to wasm branch for this example
* Load marian model from a byte array
* Sanitise executable names
* Change marian branch
* Update marian branch that loads binary models
* Example of loading model as a byte array
* Add the byte array loading files
* Die on misaligned memory
* Remove the unused argument
* Allow loading without a ptr parameter so that we don't break emc workflow
Updates marian-dev and ssplit submodules to point to the upstream
commits which implements the following:
- marian-dev: encodeWithByteRanges(...) to get source token byte-ranges
- ssplit: Has a trivial sentencesplitter functionality implemented, and
now is faster to benchmark with marian-decoder.
This enables a marian-decoder replacement written through ssplit in this
source to be benchmarked constantly with existing marian-decoder.
Nits: Removes logging introduced for multiple workers, and respective
log statements.
- Added abhi-agg/ssplit-cpp
- Added its wasm branch in bergamot-translator
- Native builds of bergamot-translator are successful
-- Sentence splitting is NOT WORKING
-- Only translation is working
Enables Mac and Ubuntu CPU only builds through GitHub CI. CI scripts are
copied from marian-dev with necessary changes.
3rd-party/marian-dev is modified to meet C++17 requirements modifying
for half_float.
CMakeLists have been modified with the necessary includes to add
browsermt/mts@nuke files to the bergamot-translator library. In
addition, adds the ssplit dependency, corresponding includes.
Intel MKL fails on compilation, unable to find libraries. To solve this
3rd_party/CMakeLists.txt is modified with @ug's fixes to propogate
variables (EXT_LIBS, etc) at a library level.
Modifications to SentencePiece are necessary to provide token level
string_views. This commit changes marian to an alternate branch which
has the feature incorporated.
