tokenizers/.gitignore

.DS_Store
*~

.vim
.env
target
.idea

/data
tokenizers/data
bindings/python/tests/data
docs/build/
docs/make.bat

__pycache__
pip-wheel-metadata
*.egg-info
*.so
/bindings/python/examples/.ipynb_checkpoints
/bindings/python/build
/bindings/python/dist

.vscode
*.code-workspace

.stack-work
Global gitignore 2019-11-01 21:55:08 +03:00			`.DS_Store`
			`*~`

Ignore .vim folder 2020-04-27 22:03:08 +03:00			`.vim`
Global gitignore 2019-11-01 21:55:08 +03:00			`.env`
			`target`
Add a visualization utility to render tokens and annotations in a notebook (#508) * Draft functionality of visualization * Added comments to make code more intelligble * polish the styles * Ensure colors are stable and comment the css * Code clean up * Made visualizer importable and added some docs * Fix styling * implement comments from PR * Fixed the regex for UNK tokens and examples in notebook * Converted docs to google format * Added a notebook showing multiple languages and tokenizers * Added visual indication of chars that are tokenized with >1 token * Reorganize things a bit and fix import * Update docs Co-authored-by: Anthony MOI <m.anthony.moi@gmail.com> 2020-12-04 18:25:56 +03:00			`.idea`
remove Cargo.lock (#7) 2019-12-24 08:22:42 +03:00
Update gitignore 2019-11-25 23:35:54 +03:00			`/data`
Update .gitignores for data folder 2020-03-15 20:44:34 +03:00			`tokenizers/data`
Python - Add some tests utils 2020-03-30 23:56:43 +03:00			`bindings/python/tests/data`
Actually adding docs. 2020-09-25 22:46:41 +03:00			`docs/build/`
			`docs/make.bat`
Ignore some python files 2019-11-27 20:22:01 +03:00
			`__pycache__`
Ignore Python package files 2020-01-06 21:04:45 +03:00			`pip-wheel-metadata`
			`*.egg-info`
Ignore some python files 2019-11-27 20:22:01 +03:00			`*.so`
Add a visualization utility to render tokens and annotations in a notebook (#508) * Draft functionality of visualization * Added comments to make code more intelligble * polish the styles * Ensure colors are stable and comment the css * Code clean up * Made visualizer importable and added some docs * Fix styling * implement comments from PR * Fixed the regex for UNK tokens and examples in notebook * Converted docs to google format * Added a notebook showing multiple languages and tokenizers * Added visual indication of chars that are tokenized with >1 token * Reorganize things a bit and fix import * Update docs Co-authored-by: Anthony MOI <m.anthony.moi@gmail.com> 2020-12-04 18:25:56 +03:00			`/bindings/python/examples/.ipynb_checkpoints`
Ignore some more python files 2020-01-06 23:23:21 +03:00			`/bindings/python/build`
			`/bindings/python/dist`
fix imports 2020-01-08 00:55:50 +03:00
Update .gitignores for data folder 2020-03-15 20:44:34 +03:00			`.vscode`
gitignore vscode workspace files 2020-06-16 01:08:14 +03:00			`*.code-workspace`
add .stack-work 2021-04-28 00:48:03 +03:00
			`.stack-work`