browsermt/bergamot-translator
1
1
Fork 0
mirror of https://github.com/browsermt/bergamot-translator.git synced 2024-09-17 16:47:18 +03:00
Code Issues Packages Projects Releases Wiki Activity
175 Commits 9 Branches 11 Tags 5.6 MiB
e1b74bccab
Commit Graph

175 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Jerin Philip
e1b74bccab Reverting moot COMPILE_WASM guards in app folder 2021-02-26 11:42:23 +00:00
Jerin Philip
7c14b737a4 Improving abort error message. 2021-02-26 11:41:10 +00:00
Jerin Philip
9ead41d879 Adds documentation, makes enqueue() private 2021-02-26 10:33:07 +00:00
Jerin Philip
66e3b4493e LoadVocabularies inlined in service_base.h 
To fix WASM Mac builds on CI. loadVocabularies function is now inlines
and available through service_base.h, from where it seems to propogate
to all places of use.
 2021-02-26 00:43:36 +00:00
Jerin Philip
dad3a4088c Marking enqueue as override() 2021-02-26 00:39:38 +00:00
Jerin Philip
2c57f4b498 Adding ci-sandbox branch for GitHub CI test-use 2021-02-26 00:11:58 +00:00
Jerin Philip
570865e799 Getting rid of the ifdef from BatchTranslator as well 2021-02-25 23:50:57 +00:00
Jerin Philip
cd01d7552a ServiceBase -> [NonThreadedService, Service] 
Through inheritance, a non-threaded and multithreaded Service are
created, both derived of the same ServiceBase class which holds the
common elements.

In preparation to solve SIGSEGV in #41. First inspections gave aborts in
thread part, and repeated SIGSEGV's in lock-policy's of shared_pointers
even in non-threaded paths.

Solving this first, to avoid ifdef or tricky paths. The non-threaded
implementation is not included in WASM builds at all, by separating out
the single-threaded logic. DRY is achieved through inheritance and
operator overloading.
 2021-02-25 23:11:09 +00:00
abhi-agg
dd3dc6f932
Merge pull request #43 from browsermt/ci-test 
CI Fixes: vanilla bergamot-translator
 2021-02-25 17:35:59 +01:00
Jerin Philip
51da121057 CI Fixes: vanilla bergamot-translator builds 
* Updating vanilla workflows with -DUSE_WASM_COMPATIBLE_MARIAN=off
* Boost: Replacing with OS Boost for Ubuntu Builds
 2021-02-25 16:29:49 +00:00
Abhishek Aggarwal
18b4c7a016 Improved README 
- Removed 'wasm-integration' branch from wasm build instructions
 - Improved native build instructions
 2021-02-25 15:22:52 +01:00
Abhishek Aggarwal
2538fb6007 Added workflows for compilation with custom marian 
- Custom marian means only those marian features that
   are required for wasm

 - Added workflow for native builds
 - Added workflow for wasm builds
 2021-02-25 13:12:16 +01:00
Abhishek Aggarwal
c2b1c6eab4 Use system installed PCRE2 for builds using full blown marian 
- USE_INTERNAL_PCRE2 is ON for custom marian builds while OFF
   for full marian builds
 2021-02-24 20:02:48 +01:00
Abhishek Aggarwal
b845ed3693 Update marian submodule 
- Fixes the compilation while building with full blown marian
 2021-02-24 19:54:38 +01:00
abhi-agg
31cdf861fa
Merge pull request #40 from browsermt/wasm-integration 
Wasm integration
 2021-02-23 18:47:27 +01:00
Abhishek Aggarwal
eb5284fb20 Removed erroneous '?' in example text in wasm test page 2021-02-23 18:43:51 +01:00
Abhishek Aggarwal
4369a56f90 Enable building marian executables for vanilla marian builds 
- COMPILE_LIBRARY_ONLY is set to ON only for wasm compatible marian
   builds
 2021-02-23 18:15:33 +01:00
abhi-agg
c28687fffb
Merge pull request #38 from browsermt/wasm-integration 
wasm-integration -> integration
 2021-02-23 16:34:22 +01:00
abhi-agg
c0efc21c6e
Merge pull request #30 from browsermt/jp/absorb-batch-translator 
Sync thread adjustments with wasm-integration
 2021-02-23 16:32:26 +01:00
Abhishek Aggarwal
415d16bd1d Single cmake option to enable/disable wasm compatible marian compilation 
- USE_WASM_COMPATIBLE_MARIAN=off will start using vanilla Marian
   i.e. with full threading support, with exceptions, with MKL

 - Changed the relevant documentation
 2021-02-23 16:15:05 +01:00
Abhishek Aggarwal
458176c050 Enable building pcre2 from sources for ssplit submodule 
- USE_INTERNAL_PCRE2 is set to ON
 - Sentence splitting is working (tested it via wasm test page)
 2021-02-22 18:51:48 +01:00
Abhishek Aggarwal
462a850d8a Changed Sentences to Paragraphs in test page of WASM 
- Sentence Splitter works now => No more sentence splitting in
   test code
 - Changed example to include some paragraphs
 2021-02-22 18:48:59 +01:00
Abhishek Aggarwal
fa4a1ed67d Adapted model config in test example of bergamot 
- Replaced deprecated names with new names
      mini-batch-words and max-length-break
 - Set cpu-threads to 0
 2021-02-22 18:29:26 +01:00
Abhishek Aggarwal
5dcbb721fa Update ssplit submodule to master branch 
- This submodule brings pcre2 lib compiled from sources
 2021-02-22 18:03:53 +01:00
Jerin Philip
fd9e79a817 Merge remote-tracking branch 'origin/wasm-integration' into jp/absorb-batch-translator 2021-02-22 16:33:46 +00:00
Motin
51f702ea6c Remove Docker-based builds since they are no more reproducible than metal builds. Fixes https://github.com/browsermt/bergamot-translator/issues/31 2021-02-18 15:02:44 +02:00
Abhishek Aggarwal
79571bada5 Improved wasm/README 
- Clarified that the Demo and API usage section assumes
   bergamot models were packaged into wasm binary
 - Formatting changes
 2021-02-18 12:48:45 +01:00
Abhishek Aggarwal
c2371dd771 Replaced "build-wasm-docker" with "build-wasm" 
- Now things are consistent with the top level README
   instructions that suggest to build in "build-wasm"
   folder
 2021-02-18 12:20:06 +01:00
Jerin Philip
fbff7389d1 Temporary: Switch to abhi-agg/ssplit-cpp@wasm 2021-02-18 11:20:01 +00:00
Jerin Philip
ca9aa64926 Switch to work with ssplit-cpp both pcre2 and pcrecpp 2021-02-18 11:07:31 +00:00
Abhishek Aggarwal
b75e72e65d Added more explanation for FILES_TO_PACKAGE in README 2021-02-18 10:42:06 +01:00
Jerin Philip
d249dcbfaa Build doc updated with wasm-branch compatible command 2021-02-17 21:15:35 +00:00
Jerin Philip
b9d081dd45 Temporary: Updating marian-dev to wasm branch 2021-02-17 19:51:57 +00:00
Abhishek Aggarwal
9feebe5cb2 Allow using relative paths for packaging files 
- PACKAGE_DIR cmake option can now accept relative paths
 2021-02-17 20:06:04 +01:00
Jerin Philip
d72343567c BatchTranslator doesn't do thread_, residue from merge removed 2021-02-17 16:41:04 +00:00
Jerin Philip
70b57ee3e7 Redundant parser include fixed 2021-02-17 16:38:47 +00:00
Jerin Philip
7b10c35483 Hard abort if multithread path launched without multithread-support 2021-02-17 13:50:42 +00:00
Jerin Philip
47b9db0c45 Documentation formatting/syntax fix 2021-02-17 13:35:10 +00:00
Jerin Philip
72848ba0f6 Fixes UEdin builds after wasm-integration merge 
A bug which crept in during manual merge is now fixed. PCItem -> Batch
on a PCQueue.

docs/marian-integration.md provides instructions to compile successfully
for multithread.
 2021-02-17 13:28:58 +00:00
Abhishek Aggarwal
b86f8a7dc2 Improved README 
- Clears up the spaghetti of model packaging
 - Usage instructions
 - Formatting changes
 2021-02-17 14:21:51 +01:00
Jerin Philip
d005f73cb9 Reverting changes to PCQueue 2021-02-17 13:10:39 +00:00
Jerin Philip
10dcb8f548 Merge remote-tracking branch 'origin/wasm-integration' into jp/absorb-batch-translator 
Merging wasm-integration. Single thread codepath seems functional.
Multithreading is broken.
 2021-02-17 13:08:58 +00:00
Jerin Philip
44a44fa156 CMake build with submodule recursive clones 2021-02-17 11:48:00 +00:00
Jerin Philip
c205c82585 Updates to README with option changes 2021-02-17 01:12:30 +00:00
Jerin Philip
fba44bec8f Improving Batcher error message with new option names 2021-02-17 01:05:20 +00:00
Jerin Philip
69201ba44c Unify options with marian 
Service specific options are renamed to align with marian-option naming
as follows:

1. max-input-sentence-tokens -> max-length-break (There's a
   max-length-crop in marian, this is the same, except breaks into
   multiple sentences than truncate/crop).
2. max-input-tokens -> mini-batch-words.
 2021-02-17 00:54:30 +00:00
Jerin Philip
0296a38cd4 Bunch of integers on containers to size_ts 2021-02-17 00:45:19 +00:00
Jerin Philip
d7556bc168 SentenceRanges: Class to work with string_views 
Adds SentenceRanges in sentence_ranges.{h,cpp} and propogates use of the
class into the rest of the pipeline.

SentenceRanges previously a vector<vector<...>> is now converted into a
flat single vector<string_view>. Annotations marking sentence boundaries
are additionally stored in the class, enabling sentence string_view
access through methods.
 2021-02-17 00:31:44 +00:00
Jerin Philip
9c907ea605 another int to size_t 2021-02-16 20:04:30 +00:00
Jerin Philip
4c8b655ac5 Batch cleanup 
Moves Batch into batch.{h,cpp}.

- Id_ no longer used due to overflow concerns. (#27)
- size_t for places where signed integer is not preferred.
- Adjustments to response.{h,cpp}
 2021-02-16 19:46:40 +00:00