* CI Changes to add tiny regression tests
* Adding an inspect cache step
* Removing ccache, pursue in another
* Incorporating Nick's changes through submodule merge
* Submodule now points to master
* Restoring ccache enabled workflow file
* Restoring ccache enabled CMakeLists
* cache -> ccache typo fix
* Moving CCACHE setup to GitHub runner file
* Find also uses CCACHE dir
* Updating CMakeLists not to override env
* Cache compiler binary's contents
* Changing a few names to trigger new build; Testing cache looks fun
* USE_CCACHE=on, -L for inspection
* Adding a ccache_cmd, but will only use in next commit
* Using ccache_cmd
* Removing "
* Adding compiler hash script
* Bunch of absolute paths
* GITHUB_WORKSPACE typo
* Nah, I'll keep -L and trigger another build
* Trying something with compiler hash on cache key backup as well
* builtin, bash it seems
* Empty commit #1
* Move ccache stats to after compile
* Reshuffling ccache vars
* No comments
* Updates to Github output set syntax
* Empty Commit 1
* Empty Commit 2
* Empty commit 3
* /bin/bash -> bash; ccache_cmd for consistency
* Adding ccache -s before and after build
* Adding comments to compiler-hash script
* Let's build cached and non-cached variants together for comparison
* Fixing quotes, /bin/bash -> bash
* Minor var/env adjustment
* Adding ccache -z before the job
* Reverting CMakeLists.txt without CCACHE
* Switching to CMAKE_LANG_COMPILER_LAUNCHER instead of CMakeLists.txt rule
* 5G -> 1G cache size
* 1G -> 2G; Hyperparameter tuning
- Removed PACKAGE_DIR cmake option
- Removed Workerfs, FORCE_FILESYSTEM=1 in wasm builds
-- File system support is not needed any more (since model,
shortlist and vocabs are being passed as bytes now)
* Change WASM_COMPATIBLE_SOURCE=OFF by default
The default was WASN_COMPATIBLE_SOURCE=ON COMPILE_WASM=OFF which is a
testing configuration, not a sensible default for native or wasm.
* Always USE_WASM_COMPATIBLE_SOURCE with COMPILE_WASM
* Set CMP0077 to fix variable handling
* first attempt to enable vocabs pass as byte arrays
* pass vocabs bytes as AlignedMemory
* add vocabIndices to avoid double loading
* small fix on parameter names and documentation
* fix windows build plus tiny update on documentation
* update marian-dev submodule
* move validate model bytearray in BatchTranslator
* small refactors on validateBinaryModel()
* switch vocab memories to std::vector<marian::Ptr<AlignedMemory>>
* update marian-dev submodule
* replace marian::Ptr to std::shared_ptr for vocab memories
* add note for vocab memories
* First draft of faithful translation
* Comments explaining pre and post
* Comments on response_builder
* Updating bergamot-translator-tests with new outputs
* Cosmetic changes in response target text construction
* Replacing &(x[0]) -> x.data() to avoid illegal indices
* Removing nullptr given both branches init pointer with legal values
* pre, post -> gap(i) addressing review comments
Functions which were pre and post before are subsumed by gap(i), and the
algorithm in ResponseBuilder adjusted to fix.
`x = nullptr` is back, should be harmless.
* Updating brt with paragraph outputs
* Bumping brt with updated outputs, buffer text at begin as well
* Bumping BRT with sync after bytearray collapse merge
* Pointing BRT to main after merge
Co-authored-by: Nikolay Bogoychev <nheart@gmail.com>
* Adding bytearray option
* collapse intermediate for bytearray apps
* Removing service-cli-bytearray
* Removing the bergamot bytearray app
* Bumping updates to brt collapsing apps
* Reasonable defaults and hard check when cmd enabled
* Update documentation for flags
* Bump brt with MKL check and skip
* Bumping BRT with MKL_FOUND instead of USE_MKL
* Bumping BRT with no mkl enforce
* Bumping BRT with ssse3 output
* Let's try disabling OpenBLAS
* Trying to disable apple accelerate
* Using WASM compatible BLAS can enable intgemm
* Adding a CMake -L to see what exactly is the diff
* Revert "Let's try disabling OpenBLAS"
This reverts commit 9a6b9bc53b.
* Revert "Using WASM compatible BLAS can enable intgemm"
This reverts commit 936a592e18.
* Restricting mac tests through tags and on GitHub CI
* Using only check-bytearray
* Bumping BRT with change of default behaviour
* Improved script that patches wasm artifacts to enable wormhole
- Made the regex pattern ignore multiple whitespaces b/w words of
the matching pattern
* Fix for loading EN->DE vocabularies in wasm test page
- Loading vocabularies for EN->DE was failing because of
the new structure of bergamot-models
* Safe transfer of bindings through typedefs
* Removing Translation* files and bringing in counterparts
* Remove previously commented out code
* Removing commented out include
* Absorb Translation* documentation
Co-authored-by: abhi-agg <66322306+abhi-agg@users.noreply.github.com>
* Update marian-dev to the newest mac version
* Attempt windows workflow
* force workflow rerun
* Separate id
* Attempt 3 at github action
* Marian dev submodule now compiles with apple clang
* Updated ssplit version to something more recent
* Attempt to fix compile on wasm
* Do not compile subproject tests
* Fix emscripten compilation on Mac
* 99% on the way to windows compile
* Try with a different generator
* Build release not debug
* Revert CMakeLists.txt hacks
* Fix sse2 compilation failure
* MSVC settings for WIN32
* Add nodefaultlib LIBCMT
* Do not compile ssplit.cpp as it contains sys/mman.h
* Revert ab56b9aa4f
* Update paths
* Set the build type to release if not set previously
* Attempt to build release with the windows workflow
* Attempt 5 at VS studio release build
* Attempt 6 at getting release build on MSVC generator
* The windows build is debug at the moment...
* fix ssplit for ubuntu 16.04
* Fix compilation with clang
* Compile on ubuntu16.04
* Explain what is going on
* Updated ssplit and workflow
* Bindings to load model and shortlist files as bytes
* Modified wasm test page for byte based loading of files
* Updates wasm README for byte loading based usage of TranslationModel
* Control validating the config options via a boolean flag
- parseOptions() function now validates the parsed options
based on the validate argument
* Minor syntactic fix
- AlignedMemory is AlignedVector<char> now instead of AlignedVector<const void*>
- This solves the issue of allocating 8x of the actual required memory for
loading files as bytes
Adds regression-tests to the workflow for native minimal/custom marian and full builds.
Co-authored-by: abhi-agg <66322306+abhi-agg@users.noreply.github.com>
* Changing Annotation to adhere to [begin, end)
* Stronger unit tests on sentences + num words, num sentences
* Hotfix with empty string view from EOS
* No more absolving empty-sentence; Added tests now defined behaviour
* Uncommenting important section in unit test
* Ensure empty string view default, beginning at end so marker points
* Further strengthen and comment unit-tests, mark exactly where empty sentence is happening
* Review comments: Dummy sentence + docs
- What should be a simple fast accessor is turning into compute.
Normally the way to deal with this, for better or worse, is to put 0 at
the beginning of sentenceEndIds_. (Putting 0 at the beginning of
sentenceEndIds_)
- Indices into what? Mentioned to be flatByteRanges_.
* Documentation updates
* More changes to docs
Co-authored-by: abhi-agg <66322306+abhi-agg@users.noreply.github.com>
* Removes vocabs and propogates fixes for breaks
* Prettify diff: Undoing comment shuffles due to merge conflict edits
* 20% of time actual work, 80% prettifying diff
* Histories members -> poof!
We however have Histories in constructor, which we will remove out of the way
soon.
Co-authored-by: Kenneth Heafield <kpu@users.noreply.github.com>
Contains "hack" that must go immediately by editing TranslationModel, to come in following commit.
* add shortlist_memory and update service-cli-bytearray test
* update marian-dev
* address review comments
* fix ccompliation and tests failures and further address review comments
* small update on marian-dev (based on browsermt/marian-dev PR#28)
* update marian-dev with upstream
* code refactoring according to review
* fix marian-dev submodule conflicts
* switch MemoryGift to AlignedVector
* copy aligned.h from kpu/intgemm for AlignedVector
* changes based on memory ownership and AlignedVector
* fix BatchTranslator inits
* small fixes according to review comments
* update submodule marian-dev to master
* update submodule marian-dev with upstream
Co-authored-by: Kenneth Heafield <kpu@users.noreply.github.com>
* Updated marian-dev submodule
- cmake changes required after the submodule update
* Added workflows for building custom marian on mac and ubuntu
* Renamed cmake option
- Renamed USE_WASM_COMPATIBLE_SOURCES to USE_WASM_COMPATIBLE_SOURCE
- Use proper compile defnitions
