1562 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Julius Marozas	7bddcf2505	Change rpath	2021-05-19 00:34:16 +03:00
Julius Marozas	045164e95f	Add revision to rayon-cond ... Without rev naersk can't fetch dependencies specified as git repos.	2021-05-19 00:34:16 +03:00
Julius Marozas	66803e9004	Fix nix/rust.nix	2021-05-19 00:34:16 +03:00
Torsten Scholak	e81395dfa0	Merge remote-tracking branch 'huggingface/master'	2021-05-03 20:44:00 -04:00
Lysandre	05829b4c1c	Revert "Fix SPM conversions" ... This reverts commit e1ffe39764.	2021-05-03 20:43:21 -04:00
Lysandre	319bf0445d	Fix SPM conversions	2021-05-03 20:43:20 -04:00
Torsten Scholak	41bf8695a9	make critical stability improvements	2021-05-03 20:40:41 -04:00
Torsten Scholak	5ab5ca03f8	add .stack-work	2021-04-27 17:48:03 -04:00
Austin Huang	fa47324b44	Merge pull request #3 from hasktorch/jsonConfig ... add creation from json config string	2021-04-27 05:36:05 -04:00
Torsten Scholak	2ed4cf479f	add creation from json config string	2021-04-26 09:01:38 -04:00
austinvhuang	92f021449d	copy shared libraries instead of using symlinks (stack git repo imports fail for symlinks to directories outside of the project directory)	2021-04-25 22:19:19 -04:00
austinvhuang	87f985db8e	change execution permission for get_bart.sh	2021-04-25 20:20:17 -04:00
Austin Huang	dda628c7f7	Merge pull request #2 from hasktorch/spec ... add createTokenizerFromConfig, saveTokenizerToConfig, and decode. imp…	2021-04-25 20:16:45 -04:00
Torsten Scholak	e7db8affc7	add safe withTokenizerFromConfig wrapper	2021-04-25 18:44:40 -04:00
Torsten Scholak	d10c60aff8	add createTokenizerFromConfig, saveTokenizerToConfig, and decode. improve marshalling. add tests for bart, roberta, and t5	2021-04-25 18:27:15 -04:00
austinvhuang	ee086b60dc	fix lib symlink	2021-04-24 23:24:16 -04:00
austinvhuang	94f930ca23	revert f661557768	2021-04-24 22:49:43 -04:00
Lysandre	e999a7b5f9	Revert "Fix SPM conversions" ... This reverts commit e1ffe39764.	2021-04-21 18:09:58 -04:00
Lysandre	e1ffe39764	Fix SPM conversions	2021-04-21 18:09:49 -04:00
austinvhuang	f051353199	update makefile	2021-04-19 21:53:16 -04:00
Austin Huang	51585f16ac	Merge pull request #1 from hasktorch/nix ... Nix plus upstream update plus wordpiece tokenizer	2021-04-18 22:57:23 -04:00
Torsten Scholak	2869f45e64	fix roberta preprocessing	2021-04-18 11:42:05 -04:00
Torsten Scholak	de02eeea34	fix roberta processing	2021-04-18 11:34:40 -04:00
Torsten Scholak	e72570c7ae	fix order of arguments	2021-04-18 11:12:05 -04:00
Torsten Scholak	a2fe5fbe71	rename add_special_token	2021-04-18 11:05:08 -04:00
Torsten Scholak	a114a9765b	fix typo	2021-04-18 10:56:01 -04:00
Torsten Scholak	038b602d94	add add_special_token	2021-04-18 10:51:52 -04:00
Torsten Scholak	ebb2abb497	remove comment	2021-04-12 23:04:42 -04:00
Torsten Scholak	915cea6e1a	add wordpiece tokenizer	2021-04-12 23:00:49 -04:00
Torsten Scholak	d96a1ae974	patch dylib	2021-04-11 19:34:57 -04:00
Torsten Scholak	35abed54f7	back to cdylib	2021-04-11 14:48:34 -04:00
Torsten Scholak	23f909bb05	try a static lib instead	2021-04-11 13:59:14 -04:00
Torsten Scholak	7ef68435ab	comment extra-lib-dirs	2021-04-11 13:13:55 -04:00
Torsten Scholak	405b5a349d	rename haskell -> tokenizers_haskell	2021-04-11 12:39:34 -04:00
Torsten Scholak	77858fce9d	add Cargo.lock files to git	2021-04-11 12:17:45 -04:00
Torsten Scholak	6b6f92b123	add Cargo.lock to git	2021-04-11 12:17:17 -04:00
Torsten Scholak	3c8cf1665e	Merge remote-tracking branch 'origin/master' into nix	2021-04-11 12:01:15 -04:00
Torsten Scholak	6d8dfe8a19	add two targets, one for tokenizers and one for tokenizers-haskell	2021-04-11 11:55:41 -04:00
austinvhuang	322be73023	fix pointer increment bug	2021-04-11 11:28:09 -04:00
Torsten Scholak	6fbcfdd611	Merge branch 'nix' of github.com:hasktorch/tokenizers into nix	2021-04-11 10:51:53 -04:00
Torsten Scholak	22ff312c1d	add workspace	2021-04-11 10:51:40 -04:00
Torsten Scholak	f08ee6025b	try adding tokenizers-haskell	2021-04-11 10:51:40 -04:00
Torsten Scholak	d0ca847ba5	add libiconv to buildInputs	2021-04-11 10:51:40 -04:00
Torsten Scholak	9842689f1a	add shell	2021-04-11 10:51:40 -04:00
Torsten Scholak	0665dce421	initial nix support	2021-04-11 10:51:40 -04:00
austinvhuang	874798777f	debugging corrupt token ids (wip)	2021-04-11 10:51:39 -04:00
austinvhuang	8edebba50f	revert step size	2021-04-11 10:51:39 -04:00
austinvhuang	7c2253b5f2	debugging memory issues (wip)	2021-04-11 10:51:39 -04:00
austinvhuang	2511369b6a	fix build	2021-04-11 10:51:39 -04:00
austinvhuang	5f321f78b3	fix build	2021-04-11 10:51:39 -04:00