15 Commits

Author	SHA1	Message	Date
Torsten Scholak	5ab5ca03f8	add .stack-work	2021-04-27 17:48:03 -04:00
Torsten Scholak	6b6f92b123	add Cargo.lock to git	2021-04-11 12:17:17 -04:00
Tal Perry	8916b6bb27	Add a visualization utility to render tokens and annotations in a notebook (#508) ... * Draft functionality of visualization * Added comments to make code more intelligble * polish the styles * Ensure colors are stable and comment the css * Code clean up * Made visualizer importable and added some docs * Fix styling * implement comments from PR * Fixed the regex for UNK tokens and examples in notebook * Converted docs to google format * Added a notebook showing multiple languages and tokenizers * Added visual indication of chars that are tokenized with >1 token * Reorganize things a bit and fix import * Update docs Co-authored-by: Anthony MOI <m.anthony.moi@gmail.com>	2020-12-04 10:25:56 -05:00
Nicolas Patry	81bb4f6da3	Actually adding docs.	2020-11-02 17:07:27 -05:00
Pierric Cistac	88354b0e40	gitignore vscode workspace files	2020-06-15 18:08:41 -04:00
Anthony MOI	36e3c28a23	Ignore .vim folder	2020-05-01 17:11:54 -04:00
Anthony MOI	023566fbbb	Python - Add some tests utils	2020-04-01 17:25:55 -04:00
Anthony MOI	a80d0a302f	Update .gitignores for data folder	2020-03-16 10:36:41 -04:00
thomwolf	6d0e3ba8f1	fix imports	2020-01-07 18:54:21 -05:00
Anthony MOI	70a17ce330	Ignore some more python files	2020-01-06 15:23:21 -05:00
Anthony MOI	5b8cd00d21	Ignore Python package files	2020-01-06 13:04:45 -05:00
Evan Pete Walsh	9f1421a04b	remove Cargo.lock (#7)	2019-12-23 21:22:42 -08:00
Anthony MOI	989e9b03ca	Ignore some python files	2019-11-27 12:22:01 -05:00
Anthony MOI	d7ba6802df	Update gitignore	2019-11-25 15:35:54 -05:00
Anthony MOI	5f57ee9f0e	Global gitignore	2019-11-01 14:55:08 -04:00