Lysandre
|
05829b4c1c
|
Revert "Fix SPM conversions"
This reverts commit e1ffe39764 .
|
2021-05-03 20:43:21 -04:00 |
|
Lysandre
|
319bf0445d
|
Fix SPM conversions
|
2021-05-03 20:43:20 -04:00 |
|
Torsten Scholak
|
41bf8695a9
|
make critical stability improvements
|
2021-05-03 20:40:41 -04:00 |
|
Torsten Scholak
|
5ab5ca03f8
|
add .stack-work
|
2021-04-27 17:48:03 -04:00 |
|
Austin Huang
|
fa47324b44
|
Merge pull request #3 from hasktorch/jsonConfig
add creation from json config string
|
2021-04-27 05:36:05 -04:00 |
|
Torsten Scholak
|
2ed4cf479f
|
add creation from json config string
|
2021-04-26 09:01:38 -04:00 |
|
austinvhuang
|
92f021449d
|
copy shared libraries instead of using symlinks (stack git repo imports fail for symlinks to directories outside of the project directory)
|
2021-04-25 22:19:19 -04:00 |
|
austinvhuang
|
87f985db8e
|
change execution permission for get_bart.sh
|
2021-04-25 20:20:17 -04:00 |
|
Austin Huang
|
dda628c7f7
|
Merge pull request #2 from hasktorch/spec
add createTokenizerFromConfig, saveTokenizerToConfig, and decode. imp…
|
2021-04-25 20:16:45 -04:00 |
|
Torsten Scholak
|
e7db8affc7
|
add safe withTokenizerFromConfig wrapper
|
2021-04-25 18:44:40 -04:00 |
|
Torsten Scholak
|
d10c60aff8
|
add createTokenizerFromConfig, saveTokenizerToConfig, and decode. improve marshalling. add tests for bart, roberta, and t5
|
2021-04-25 18:27:15 -04:00 |
|
austinvhuang
|
ee086b60dc
|
fix lib symlink
|
2021-04-24 23:24:16 -04:00 |
|
austinvhuang
|
94f930ca23
|
revert f661557768
|
2021-04-24 22:49:43 -04:00 |
|
austinvhuang
|
f051353199
|
update makefile
|
2021-04-19 21:53:16 -04:00 |
|
Austin Huang
|
51585f16ac
|
Merge pull request #1 from hasktorch/nix
Nix plus upstream update plus wordpiece tokenizer
|
2021-04-18 22:57:23 -04:00 |
|
Torsten Scholak
|
2869f45e64
|
fix roberta preprocessing
|
2021-04-18 11:42:05 -04:00 |
|
Torsten Scholak
|
de02eeea34
|
fix roberta processing
|
2021-04-18 11:34:40 -04:00 |
|
Torsten Scholak
|
e72570c7ae
|
fix order of arguments
|
2021-04-18 11:12:05 -04:00 |
|
Torsten Scholak
|
a2fe5fbe71
|
rename add_special_token
|
2021-04-18 11:05:08 -04:00 |
|
Torsten Scholak
|
a114a9765b
|
fix typo
|
2021-04-18 10:56:01 -04:00 |
|
Torsten Scholak
|
038b602d94
|
add add_special_token
|
2021-04-18 10:51:52 -04:00 |
|
Torsten Scholak
|
ebb2abb497
|
remove comment
|
2021-04-12 23:04:42 -04:00 |
|
Torsten Scholak
|
915cea6e1a
|
add wordpiece tokenizer
|
2021-04-12 23:00:49 -04:00 |
|
Torsten Scholak
|
d96a1ae974
|
patch dylib
|
2021-04-11 19:34:57 -04:00 |
|
Torsten Scholak
|
35abed54f7
|
back to cdylib
|
2021-04-11 14:48:34 -04:00 |
|
Torsten Scholak
|
23f909bb05
|
try a static lib instead
|
2021-04-11 13:59:14 -04:00 |
|
Torsten Scholak
|
7ef68435ab
|
comment extra-lib-dirs
|
2021-04-11 13:13:55 -04:00 |
|
Torsten Scholak
|
405b5a349d
|
rename haskell -> tokenizers_haskell
|
2021-04-11 12:39:34 -04:00 |
|
Torsten Scholak
|
77858fce9d
|
add Cargo.lock files to git
|
2021-04-11 12:17:45 -04:00 |
|
Torsten Scholak
|
6b6f92b123
|
add Cargo.lock to git
|
2021-04-11 12:17:17 -04:00 |
|
Torsten Scholak
|
3c8cf1665e
|
Merge remote-tracking branch 'origin/master' into nix
|
2021-04-11 12:01:15 -04:00 |
|
Torsten Scholak
|
6d8dfe8a19
|
add two targets, one for tokenizers and one for tokenizers-haskell
|
2021-04-11 11:55:41 -04:00 |
|
austinvhuang
|
322be73023
|
fix pointer increment bug
|
2021-04-11 11:28:09 -04:00 |
|
Torsten Scholak
|
6fbcfdd611
|
Merge branch 'nix' of github.com:hasktorch/tokenizers into nix
|
2021-04-11 10:51:53 -04:00 |
|
Torsten Scholak
|
22ff312c1d
|
add workspace
|
2021-04-11 10:51:40 -04:00 |
|
Torsten Scholak
|
f08ee6025b
|
try adding tokenizers-haskell
|
2021-04-11 10:51:40 -04:00 |
|
Torsten Scholak
|
d0ca847ba5
|
add libiconv to buildInputs
|
2021-04-11 10:51:40 -04:00 |
|
Torsten Scholak
|
9842689f1a
|
add shell
|
2021-04-11 10:51:40 -04:00 |
|
Torsten Scholak
|
0665dce421
|
initial nix support
|
2021-04-11 10:51:40 -04:00 |
|
austinvhuang
|
874798777f
|
debugging corrupt token ids (wip)
|
2021-04-11 10:51:39 -04:00 |
|
austinvhuang
|
8edebba50f
|
revert step size
|
2021-04-11 10:51:39 -04:00 |
|
austinvhuang
|
7c2253b5f2
|
debugging memory issues (wip)
|
2021-04-11 10:51:39 -04:00 |
|
austinvhuang
|
2511369b6a
|
fix build
|
2021-04-11 10:51:39 -04:00 |
|
austinvhuang
|
5f321f78b3
|
fix build
|
2021-04-11 10:51:39 -04:00 |
|
austinvhuang
|
388a169bf6
|
package as a library
|
2021-04-11 10:51:39 -04:00 |
|
austinvhuang
|
9f5ea6f603
|
package as exported library, fix up makefiles, wip - getIDs dont look valid
|
2021-04-11 10:51:39 -04:00 |
|
austinvhuang
|
b74bbffedc
|
add target/release dir to repo so symlink works
|
2021-04-11 10:51:39 -04:00 |
|
austinvhuang
|
9ad3f2b920
|
rename haskell lib to tokenizers, t5 stub
|
2021-04-11 10:51:38 -04:00 |
|
austinvhuang
|
8371dda8c4
|
add t5 downloaders
|
2021-04-11 10:51:38 -04:00 |
|
austinvhuang
|
dcba6c0f16
|
id marshalling (WIP)
|
2021-04-11 10:51:38 -04:00 |
|