Abdulrhman Alkhodiry
|
33b2944298
|
Update dependencies and fix convert_model (#458)
* feat: Update dependencies in Cargo.toml
Update the dependencies in Cargo.toml to their latest versions:
- rust_tokenizers: 8.1.1
- tch: 0.16.0 (with features = ["download-libtorch"])
- serde_json: 1
- serde: 1 (with features = ["derive"])
- ordered-float: 4.2.0
- uuid: 1 (with features = ["v4"])
- thiserror: 1
- half: 2
- regex: 1.6
- cached-path: 0.6 (with default-features = false and optional = true)
- dirs: 5 (optional = true)
- lazy_static: 1 (optional = true)
- ort: 1.16.3 (optional = true, default-features = false, features = ["half"])
- ndarray: 0.15 (optional = true)
- tokenizers: 0.19.1 (optional = true, default-features = false, features = ["onig"])
```
* chore: Update .gitignore and requirements.txt, and improve convert_model.py
Update .gitignore to exclude the /models/ and /.venv/ directories, and the convert_model.log file.
Remove the requirements.txt file.
In convert_model.py:
- Add a new function, `zipfile_factory`, to handle zip file creation.
- Update the logger configuration to log debug messages to a file named `convert_model.log`.
* delete duplicate requirements file
* update CI req file path
* missing requests dependency
---------
Co-authored-by: Abdulrhman Alkhodiry <aalkhodiry@jahez.net>
Co-authored-by: Guillaume Becquin <guillaume.becquin@gmail.com>
|
2024-06-30 08:41:10 +01:00 |
|
guillaume-be
|
107fb214d9
|
Support wildcard for source files in convert utilities (#399)
* Support wildcard for source files
* - Support offloading at end of every file for conversion
|
2023-07-02 08:36:21 +01:00 |
|
guillaume-be
|
a74d023583
|
Add bfloat16 tensor loading support (#396)
|
2023-06-25 09:21:52 +01:00 |
|
Joseph Hajduk
|
2bff63b2ee
|
updated tch-rs to 0.13.0 (#380)
* updated tch-rs to 0.13.0
find replaced of_slice to from_slice as per
008fff6cc0/CHANGELOG.md
* fixed formatting
* Add download feature and update CI
* add build script, update CI
* updated chanelog, readme, convert script
* fixed wrong position for build script
* added libtorch download to dependencies download test script
* args reordering
---------
Co-authored-by: josephhajduk <joseph@solidys.dev>
Co-authored-by: Guillaume Becquin <guillaume.becquin@gmail.com>
|
2023-05-21 18:41:18 +01:00 |
|
Panakotta00
|
431c3ef11b
|
convert_model.py add support for multiple binary files (#347)
|
2023-04-06 21:32:51 +01:00 |
|
Romain Leroux
|
c448862185
|
Add GPT-J support (#285) (#288)
* Add GPT-J support (#285)
* Improve GPT-J implementation
* Improve GPT-J tests
* Adapt GPT-J to latest master branch
* Specify how to convert GPT-J weights instead of providing them
|
2023-02-15 19:10:47 +00:00 |
|
guillaume-be
|
d7e9c03694
|
Long t5 implementation (#333)
* LongT5 config implementation
* LongT5 WiP: utility functions 1
* LongT5 WiP: utility functions (2)
* LongT5 WiP: utility functions (3)
* LongT5 WiP: utility functions (4)
* made T5 FF activations generic, expose T5 modules to crate
* Longt% local attention WIP
* LongT5 local attention
* LongT5 global attention WIP
* LongT5 global attention
* LongT5 attention modules (WIP)
* align LongT5 position bias with T5
* Addition of LongT5Block
* LongT5Stack WiP
* LongT5Stack implementation
* LongT5Model implementation
* LongT5ForConditionalGeneration implementation
* Addition of LongT5Generator, inclusion in pipelines
* LongT5 attention fixes
* Fix MIN/MAX dtype computation, mask for longt5
* Updated min/max and infinity computation across models
* GlobalTransient attention fixes
* Updated changelog, readme, tests, clippy
|
2023-02-12 16:18:20 +00:00 |
|
Romain Leroux
|
4d8a298586
|
Add sbert implementation for inference (#250)
* Add sbert implementation for inference
* Fix clippy warnings
* Refactor sentence embeddings into a dedicated pipeline
* Add output_attentions and output_hidden_states to T5Config
* Add sbert implementation for inference
* Fix clippy warnings
* Refactor sentence embeddings into a dedicated pipeline
* Add output_attentions and output_hidden_states to T5Config
* Improve sentence_embeddings implementation
* Dedicated tokenizer config for strip_accents and add_prefix_space
* Rename forward to encode_as_tensor
* Remove _conf from Dense layer
* Add sentence embeddings docs
* Addition of remote resources and tests update
* Merge feature branch and fix doctests
* Add SentenceEmbeddingsBuilder<Remote> and improve remote resources
* Use tch::no_grad in sentence embeddings
* Updated changelog, registration of sentence embeddings integration tests
Co-authored-by: Guillaume Becquin <guillaume.becquin@gmail.com>
|
2022-06-21 20:24:09 +01:00 |
|
Guillaume Becquin
|
60591c0644
|
Updated DeBERTa configuration parsing
|
2021-11-28 12:43:35 +01:00 |
|
guillaume-be
|
4282d7b5c4
|
Reuse gpt2 embeddings (#160)
* Updated GPT2 to re-use embeddings for LM head
* Updated conversion utilities
* Updated changelog
|
2021-06-12 11:11:34 +02:00 |
|
Guillaume B
|
b6db7cacfb
|
MBart validation, weights updated
|
2021-06-05 11:47:56 +02:00 |
|
Guillaume B
|
05295068b8
|
Updated README (addition of benchmarks and file conversion)
|
2021-02-02 17:51:39 +01:00 |
|
Guillaume B
|
f8ed5079c1
|
initial commit for ProphetNet
|
2021-01-07 10:41:50 +01:00 |
|
Guillaume B
|
dca5252f38
|
Addition of generic conversion script
|
2020-12-04 16:35:13 +01:00 |
|