Jerin Philip
e1b74bccab
Reverting moot COMPILE_WASM guards in app folder
2021-02-26 11:42:23 +00:00
Jerin Philip
7c14b737a4
Improving abort error message.
2021-02-26 11:41:10 +00:00
Jerin Philip
9ead41d879
Adds documentation, makes enqueue() private
2021-02-26 10:33:07 +00:00
Jerin Philip
66e3b4493e
LoadVocabularies inlined in service_base.h
...
To fix WASM Mac builds on CI. loadVocabularies function is now inlines
and available through service_base.h, from where it seems to propogate
to all places of use.
2021-02-26 00:43:36 +00:00
Jerin Philip
dad3a4088c
Marking enqueue as override()
2021-02-26 00:39:38 +00:00
Jerin Philip
2c57f4b498
Adding ci-sandbox branch for GitHub CI test-use
2021-02-26 00:11:58 +00:00
Jerin Philip
570865e799
Getting rid of the ifdef from BatchTranslator as well
2021-02-25 23:50:57 +00:00
Jerin Philip
cd01d7552a
ServiceBase -> [NonThreadedService, Service]
...
Through inheritance, a non-threaded and multithreaded Service are
created, both derived of the same ServiceBase class which holds the
common elements.
In preparation to solve SIGSEGV in #41 . First inspections gave aborts in
thread part, and repeated SIGSEGV's in lock-policy's of shared_pointers
even in non-threaded paths.
Solving this first, to avoid ifdef or tricky paths. The non-threaded
implementation is not included in WASM builds at all, by separating out
the single-threaded logic. DRY is achieved through inheritance and
operator overloading.
2021-02-25 23:11:09 +00:00
abhi-agg
dd3dc6f932
Merge pull request #43 from browsermt/ci-test
...
CI Fixes: vanilla bergamot-translator
2021-02-25 17:35:59 +01:00
Jerin Philip
51da121057
CI Fixes: vanilla bergamot-translator builds
...
* Updating vanilla workflows with -DUSE_WASM_COMPATIBLE_MARIAN=off
* Boost: Replacing with OS Boost for Ubuntu Builds
2021-02-25 16:29:49 +00:00
Abhishek Aggarwal
18b4c7a016
Improved README
...
- Removed 'wasm-integration' branch from wasm build instructions
- Improved native build instructions
2021-02-25 15:22:52 +01:00
Abhishek Aggarwal
2538fb6007
Added workflows for compilation with custom marian
...
- Custom marian means only those marian features that
are required for wasm
- Added workflow for native builds
- Added workflow for wasm builds
2021-02-25 13:12:16 +01:00
Abhishek Aggarwal
c2b1c6eab4
Use system installed PCRE2 for builds using full blown marian
...
- USE_INTERNAL_PCRE2 is ON for custom marian builds while OFF
for full marian builds
2021-02-24 20:02:48 +01:00
Abhishek Aggarwal
b845ed3693
Update marian submodule
...
- Fixes the compilation while building with full blown marian
2021-02-24 19:54:38 +01:00
abhi-agg
31cdf861fa
Merge pull request #40 from browsermt/wasm-integration
...
Wasm integration
2021-02-23 18:47:27 +01:00
Abhishek Aggarwal
eb5284fb20
Removed erroneous '?' in example text in wasm test page
2021-02-23 18:43:51 +01:00
Abhishek Aggarwal
4369a56f90
Enable building marian executables for vanilla marian builds
...
- COMPILE_LIBRARY_ONLY is set to ON only for wasm compatible marian
builds
2021-02-23 18:15:33 +01:00
abhi-agg
c28687fffb
Merge pull request #38 from browsermt/wasm-integration
...
wasm-integration -> integration
2021-02-23 16:34:22 +01:00
abhi-agg
c0efc21c6e
Merge pull request #30 from browsermt/jp/absorb-batch-translator
...
Sync thread adjustments with wasm-integration
2021-02-23 16:32:26 +01:00
Abhishek Aggarwal
415d16bd1d
Single cmake option to enable/disable wasm compatible marian compilation
...
- USE_WASM_COMPATIBLE_MARIAN=off will start using vanilla Marian
i.e. with full threading support, with exceptions, with MKL
- Changed the relevant documentation
2021-02-23 16:15:05 +01:00
Abhishek Aggarwal
458176c050
Enable building pcre2 from sources for ssplit submodule
...
- USE_INTERNAL_PCRE2 is set to ON
- Sentence splitting is working (tested it via wasm test page)
2021-02-22 18:51:48 +01:00
Abhishek Aggarwal
462a850d8a
Changed Sentences to Paragraphs in test page of WASM
...
- Sentence Splitter works now => No more sentence splitting in
test code
- Changed example to include some paragraphs
2021-02-22 18:48:59 +01:00
Abhishek Aggarwal
fa4a1ed67d
Adapted model config in test example of bergamot
...
- Replaced deprecated names with new names
mini-batch-words and max-length-break
- Set cpu-threads to 0
2021-02-22 18:29:26 +01:00
Abhishek Aggarwal
5dcbb721fa
Update ssplit submodule to master branch
...
- This submodule brings pcre2 lib compiled from sources
2021-02-22 18:03:53 +01:00
Jerin Philip
fd9e79a817
Merge remote-tracking branch 'origin/wasm-integration' into jp/absorb-batch-translator
2021-02-22 16:33:46 +00:00
Motin
51f702ea6c
Remove Docker-based builds since they are no more reproducible than metal builds. Fixes https://github.com/browsermt/bergamot-translator/issues/31
2021-02-18 15:02:44 +02:00
Abhishek Aggarwal
79571bada5
Improved wasm/README
...
- Clarified that the Demo and API usage section assumes
bergamot models were packaged into wasm binary
- Formatting changes
2021-02-18 12:48:45 +01:00
Abhishek Aggarwal
c2371dd771
Replaced "build-wasm-docker" with "build-wasm"
...
- Now things are consistent with the top level README
instructions that suggest to build in "build-wasm"
folder
2021-02-18 12:20:06 +01:00
Jerin Philip
fbff7389d1
Temporary: Switch to abhi-agg/ssplit-cpp@wasm
2021-02-18 11:20:01 +00:00
Jerin Philip
ca9aa64926
Switch to work with ssplit-cpp both pcre2 and pcrecpp
2021-02-18 11:07:31 +00:00
Abhishek Aggarwal
b75e72e65d
Added more explanation for FILES_TO_PACKAGE in README
2021-02-18 10:42:06 +01:00
Jerin Philip
d249dcbfaa
Build doc updated with wasm-branch compatible command
2021-02-17 21:15:35 +00:00
Jerin Philip
b9d081dd45
Temporary: Updating marian-dev to wasm branch
2021-02-17 19:51:57 +00:00
Abhishek Aggarwal
9feebe5cb2
Allow using relative paths for packaging files
...
- PACKAGE_DIR cmake option can now accept relative paths
2021-02-17 20:06:04 +01:00
Jerin Philip
d72343567c
BatchTranslator doesn't do thread_, residue from merge removed
2021-02-17 16:41:04 +00:00
Jerin Philip
70b57ee3e7
Redundant parser include fixed
2021-02-17 16:38:47 +00:00
Jerin Philip
7b10c35483
Hard abort if multithread path launched without multithread-support
2021-02-17 13:50:42 +00:00
Jerin Philip
47b9db0c45
Documentation formatting/syntax fix
2021-02-17 13:35:10 +00:00
Jerin Philip
72848ba0f6
Fixes UEdin builds after wasm-integration merge
...
A bug which crept in during manual merge is now fixed. PCItem -> Batch
on a PCQueue.
docs/marian-integration.md provides instructions to compile successfully
for multithread.
2021-02-17 13:28:58 +00:00
Abhishek Aggarwal
b86f8a7dc2
Improved README
...
- Clears up the spaghetti of model packaging
- Usage instructions
- Formatting changes
2021-02-17 14:21:51 +01:00
Jerin Philip
d005f73cb9
Reverting changes to PCQueue
2021-02-17 13:10:39 +00:00
Jerin Philip
10dcb8f548
Merge remote-tracking branch 'origin/wasm-integration' into jp/absorb-batch-translator
...
Merging wasm-integration. Single thread codepath seems functional.
Multithreading is broken.
2021-02-17 13:08:58 +00:00
Jerin Philip
44a44fa156
CMake build with submodule recursive clones
2021-02-17 11:48:00 +00:00
Jerin Philip
c205c82585
Updates to README with option changes
2021-02-17 01:12:30 +00:00
Jerin Philip
fba44bec8f
Improving Batcher error message with new option names
2021-02-17 01:05:20 +00:00
Jerin Philip
69201ba44c
Unify options with marian
...
Service specific options are renamed to align with marian-option naming
as follows:
1. max-input-sentence-tokens -> max-length-break (There's a
max-length-crop in marian, this is the same, except breaks into
multiple sentences than truncate/crop).
2. max-input-tokens -> mini-batch-words.
2021-02-17 00:54:30 +00:00
Jerin Philip
0296a38cd4
Bunch of integers on containers to size_ts
2021-02-17 00:45:19 +00:00
Jerin Philip
d7556bc168
SentenceRanges: Class to work with string_views
...
Adds SentenceRanges in sentence_ranges.{h,cpp} and propogates use of the
class into the rest of the pipeline.
SentenceRanges previously a vector<vector<...>> is now converted into a
flat single vector<string_view>. Annotations marking sentence boundaries
are additionally stored in the class, enabling sentence string_view
access through methods.
2021-02-17 00:31:44 +00:00
Jerin Philip
9c907ea605
another int to size_t
2021-02-16 20:04:30 +00:00
Jerin Philip
4c8b655ac5
Batch cleanup
...
Moves Batch into batch.{h,cpp}.
- Id_ no longer used due to overflow concerns. (#27 )
- size_t for places where signed integer is not preferred.
- Adjustments to response.{h,cpp}
2021-02-16 19:46:40 +00:00