mirror of
https://github.com/guillaume-be/rust-bert.git
synced 2024-09-11 12:55:34 +03:00
17 KiB
17 KiB
Changelog
All notable changes to this project will be documented in this file. The format is based on Keep a Changelog.
[Unreleased]
Added
- (BREAKING) Support for
prefix_allowed_tokens_fn
argument for generation, allowing users to control the generation via custom functions - (BREAKING) Support for
forced_bos_token_id
argument for generation, allowing users to force a given BOS token for generation (useful for MBart/M2M-class models) - Addition of the MBart Language model and support for text generation / direct translation between 50 language
[0.15.1] - 2021-06-01
Fixed
- Fixed conversation model panic for user inputs exceeding the maximum model length (1000 tokens)
- Fixed translation model panic for user inputs exceeding the maximum number of position embeddings
[0.15.0] - 2021-05-16
Added
- Addition of translation language pairs:
- English <-> Chinese (Simplified)
- English <-> Chinese (Traditional)
- English <-> Dutch
- English <-> Swedish
- English <-> Arabic
- English <-> Hebrew
- English <-> Hindi
- Addition of a Part of Speech pipeline. This pipeline allows predicting the POS tag (e.g. Noun, Adjective, Verb) of words in input sentences.
- Addition of a lightweight English Part of Speech tagging pretrained MobileBERT model
- Addition of the Pegasus language model and support for conditional generation
- Addition of a model for Pegasus summarization pretrained on the CNN-DM dataset
- Addition of the GPT-Neo language model and pretrained snapshots (125M, 1.3B and 2.7B parameters). Registration of GPT-Neo as an option for
TextGenerationPipeline
.
Changed
- (BREAKING) Changed
classif_dropout
inBartConfig
to be an optional field. This affects dependencies instantiatingBartConfig
from scratch, or usingclassif_config
for custom model heads. - (BREAKING) Changed token classification pipelines to return a Vec<Vec> instead of a Vec. The token-level predictions are now returned in separate vectors for each input sequence provided as an input (they were previously returned in a flattened vector)
- Simplification of the BART language model code base (also used for Marian and Pegasus language models)
- (BREAKING) Updated to
tch 0.4.1
(based onlibtorch 1.8.1
)
Fixed
- Fixed character indexing error for Question Answering pipeline answers
Removed
- Dependency to
itertools
crate
[0.14.0] - 2021-02-22
Added
- Addition of the Longformer language model, task-specific heads and registration in relevant pipelines
Changed
- (BREAKING) Exposed additional settings for the Question Answering pipeline related to the maximum question, context and answer length. This is not backward compatible if the question answering configuration was created without using the
new
creator. - Simplified the Question answering pipeline to rely on the offsets calculated by the tokenizers instead of a manual alignment. This results in moderate execution speed improvements for this pipeline.
- Updated the padding strategy for the Question answering pipeline. While before all sequences were padded to a fixed
max_length
(defaulting to 384), the padding is now done dynamically based on the length of the inputs. This results in a significant speed improvement for this pipeline.
Fixed
- Fixed a bug for Question Answering for models that were not based on Wordpiece tokenization (including BPE and unigram based tokenizers). The issue was caused by the pre-tokenization step that was stripping the leading whitespace for all tokens. The performance of these models for QA should improve significantly.
[0.13.0] - 2021-02-03
Added
- Addition of the ProphetNet language model, task-specific heads and registration in relevant pipelines
- (BREAKING) Implementation of Diverse Beam Search. This allows the generation of more diverse sequences within the number of beams. Addition of 2 new fields to the
GenerateConfig
that are propagated through all text generation configs (e.g.TranslationConfig
):num_beam_groups
(Option<i64>
), indicating the number of sub-beam groups. This must be a divisor of the number of beams.diversity_penalty
(Option<f64>
), indicating by which amount to penalize common words between beam groups. This will default to 5.5 if not provided. The impact of this diverse beam search is illustrated in the GPT2 integration tests.
Changed
- (BREAKING) Simplified the input and output of encoder/decoder models to avoid needing to take ownership of the possibly cached encoder hidden state, offering a minor performance improvement for text generation tasks. The model output field for encoder hidden states are now optional, and only returned if the encoder hidden states were not provided for the given forward path. This may be a breaking change for low-level dependencies that manipulate directly the encoder/decoder model outputs.
- (BREAKING) Moved the language models implementation of the
PrivateLanguageGenerator
andLanguageGenerator
traits (needed to generate text) to the model modules, cleaning up the generation_utils module. - Updated download utilities crate, now leveraging Tokio 1.0 runtimes.
Fixed
- Updated padding information and addition of position ids for batched GPT2 generation. Prior to this change, inputs that required padding had a lower quality for the text generated.
[0.12.1] - 2021-01-04
Added
- Addition of the MobileBERT language model, task-specific heads and registration in relevant pipelines
Changed
- Made all model configurations
Clone
- Made several base modules of the BERT language model public, and added model output
Struct
for the new publicly exposed, complex types
[0.12.0] - 2020-11-29
Added
- Addition of the Reformer language model, task-specific heads and registration in relevant pipelines
- Pre-trained models for DistilRoBERTa, used as a default for integration tests
Changed
- Updated endpoint of the model resources reflecting changes to the Hugging Face's model hub
- Early stopping turned by default on for translation and summarization
[0.11.0] - 2020-11-02
Added
- Support for additional models for the conversational pipeline
Changed
- Updated the version of Tokenizer crate with consistent visibility
- (BREAKING) move of teh text generation pipeline to its owned pipeline. Shared generation utilities are moved to
generation_utils
- All models, tokenizers and pipelines are now
Send
[0.10.0] - 2020-10-04
Added
- Benchmark scripts for all pipelines
- Addition of the XLNet model and task-specific heads
Changed
- (BREAKING) Changed the download method for resources now a method of the resource itself, and leveraging the cached-path crate.
- (BREAKING) Changed the return type of models to be output
Struct
instead of long tuples. - (BREAKING) Changed the naming of the model main modules from
modelname
tomodel_modelname
to avoid confusion with the top level module name - Extended the range of allowed types for pipelines input, allowing both owned
Vec
and slices, and bothString
and sting slice. - Handling of all activations functions is mow made from a common module and
Struct
[0.9.0] - 2020-09-06
Added
- Zero-shot classification pipeline using a natural language inference model
Changed
- (BREAKING) Updated version of tokenizers crate with added options for lower casing, accent stripping and prefix addition
- Updated BART classification model to allow running their
forward
method without being mutable.
[0.8.0] - 2020-08-25
Added
- (BREAKING) Improved error handling via the addition of
RustBertError
and error propagation throughout the crate.
Changed
- Updated version of tokenizers crate with improved error handling
[0.7.12] - 2020-08-12
Added
- Addition of the reformer language model and its integration for language generation
Changed
- Changed model resources endpoints to leverage updated Hugging Face's model hub
- Updated the beam search processing to use vectorized operations
[0.7.11] - 2020-07-26
Changed
- Generalization of the accepted input for several pipelines to accept both
Vec
and slices, and to accept bothString
and&str
[0.7.10] - 2020-07-08
Added
- Addition of the ALBERT language model and task-specific heads
- Addition of German - English translation models
- Addition of the T5 language model and integration in supported pipelines (translation and summarization)
Changed
- Updated the modules throughout the crate to accept both owned and references to varstore paths.
[0.7.9] - 2020-06-28
Added
- Addition of a multi-turn conversational pipeline based on DialoGPT.
[0.7.8] - 2020-06-23
Fixed
- Code formatting using
rustfmt
[0.7.7] - 2020-06-06
Changed
- Removed the requirement for generation models to be mutable. Models are now all stateless, and no longer store an internal cache (now provided as an input).
- Updated BART model to take past layer states as an input instead of storing in internally.
Fixed
- Fixed sequence classification model logits squeeze causing it to crash for batched inputs.
[0.7.6] - 2020-05-27
Added
- Addition of translation between Russian and English
Fixed
- Fixed a bug causing downloads to be incomplete, and removes the creation of a tokio runtime for the download of resources.
[0.7.5] - 2020-05-25
Added
- Addition of the Marian model, leveraging a shared language model implementation with the BART model.
- Addition of translation capabilities. Supports translation between English and French, Spanish, Portuguese, Italian, Catalan and German, and between German and French.
[0.7.4] - 2020-05-25
Added
- Addition of multi-label classification capabilities for sequence classification via the
predict_mutilabel
function.
[0.7.3] - 2020-05-19
Added
- Generalization of pipelines to allow leveraging multiple model architectures. Leveraging
Enum
unpacking, introducesConfigOption
,TokenizerOption
and pipeline-specific Options. - Addition of generic
SentenceClassificationModel
pipeline. TheSentimentModel
now leverages shared implementation for sentence classification. - Addition of
TokenClassificationModel
pipeline. TheNERModel
now leverages shared implementation for token classification.
Changed
- Major rework of tokenization crate, alignment with updated API
[0.7.2] - 2020-05-03
Fixed
- Minor bug fixes for tokenization
[0.7.1] - 2020-05-03
Added
- Implementation of the Electra model (generator, discriminator, task-specific heads)
- GPT2-medium and GPT2-large models
[0.7.0] - 2020-04-26
Added
- Addition of Resources for handling file dependencies (e.g. vocabularies, model weights, configurations). Resources may be
LocalResources
(pointing to a filesystem location) orRemoteResources
(pointing to a remote endpoint). These resources can be passed to adownload_resource
method that returns the location in the local filesystem for both types of resources, downloading them if necessary. - Resources specifications for all existing architectures, pointing to model files hosted on Hugging Face's model hub.
Changed
- (BREAKING) moved the resources' specification to the
GenerateConfig
forGPT2Generator
. - (BREAKING) creation of pipeline configurations to contain the resources required to build the pipeline, used as an input rather than paths to local files.
- Updated the configuration for the number of target labels to use the
id2label
field instead ofnum_labels
(aligning with changes in standard configuration in the Transformers library). Removednum_labels
from configurations. - Made the
output_attentions
,output_hidden_states
andtorchscript
fields for DistilBERT configuration optional - Fixed the device placement for sinusoidal embeddings for DistilBERT model.
[0.6.2] - 2020-04-07
Changed
- Optimization of the BART model avoiding unnecessary tensor copies for cache manipulation and residual connections.
- Optimization of DistilBERT model when embeddings are provided as an input
[0.6.1] - 2020-04-06
Changed
- Minor optimizations to question answering and sentiment analysis pipelines
- Addition of a cache reset for text generation routines
- Implementation of cache reset for BART language model
[0.6.0] - 2020-04-05
Added
- BART language model
- Implementation of
LanguageModel
andPrivateLanguageModel
for BART - Summarization capabilities
- Tanh activation
Changed
- (BREAKING) Moved the
LMHeadModel
Trait from GPT2 module to the pipelines module - Updated the
LMHeadModel
inputs to includeencoder_outputs
anddecoder_input_ids
to support causal language model (e.g. BART) - (BREAKING) Added methods to the
PrivateLanguageGenerator
to support encoder-decoder models - (BREAKING) changed the type of
Generator
language model to require mutability (BART caching mechanism stores the cache in the model requiring the entire model mutability - changed at a later point) - Optimization of the
get_banned_token
method
Fixed
- Updated the device location of the token update when EOS is not allowed because the minimum sequence length was not reached
- No longer process a given beam hypothesis if it is marked as done
- No longer add beams to a hypothesis if the rank is lower than the number of beams
- Updated final beam update to skip completed hypotheses
[0.5.3] - 2020-03-27
Added
- Documentation throughout the crate
- Creation of a
GenerateConfig
configuration structure to hold generation options
Changed
- Visibility of low-level utilities in the crate
- Updated the generation options to be passed at the text generation model instantiation, rather than at every call to the
generate
method - Updated visibility of generation routines into a public API and private lower level methods
[0.5.2] - 2020-03-17
Changed
- Text generation now takes a
Option<Vec<&str>>
instead of aOption<&str>
. Shorter sequences are left-padded withpad
if available, otherwise witheos
. - Turned-off gradient calculations for generation process
[0.5.1] - 2020-03-16
Fixed
- Beam search completion validation
- Padding sequence for sentences shorter than the maximum length moved to correct device
[0.5.0] - 2020-03-16
Added
- DistilGPT2 pretrained weights for GPT2
LMHeadModel
trait for model supporting text generation, offering an interface between the model specific input/output, and the generic set of inputs/outputs expected for model supporting text generation- Implementation of
LMHeadModel
for GPT2 and GPT - Text generation pipeline, supporting beam search, top-k/top-p decoding, repeated tokens banning, repetition and length penalties as
LanguageGenerator
Trait - Implementation of
LanguageGenerator
for GPT and GPT2 - Examples and tests for language generation
Fixed
- Fixed concatenation dimension for GPT2 past
[0.4.5] - 2020-03-07
Changed
- Updated input type for
QuestionAnsweringModel
'spredict
to be&[QaInput]
instead of a pair of question and context strings. QuestionAnsweringModel now works with a list of inputs and returns a list of predictions, processing inputs as batches.
[0.4.4] - 2020-03-01
Added
- Swish and gelu_new activation functions
- GPT2 language model
- GPT language model
[0.4.3] - 2020-02-25
Added
- Addition of a NER pipeline
- Addition of a QuestionAnswering pipeline
Changed
- Moved
SentimentClassifier
from DistilBERT module to the newly created pipelines - Changed precision of id to label mapping of BERT config from
i32
toi64
- Simplified calculation of sinusoidal embeddings for DistilBERT
[0.4.1] - 2020-02-21
Added
- Addition of RoBERTa language model
- Addition of
BertEmbedding
trait for BERT-like models
Changed
- Updated
BertEmbeddings
to implement the newly createdBertEmbedding
Trait - Updated
BertModel
's embeddings to be of typeimpl BertEmbedding
rather than specific embeddings, allowing to re-use the BERT structure for other models, only replacing the embeddings layer.
Fixed
- Fixed the variable path for BERT models with task-specific heads to allow loading a snapshot from models trained on Transformers.
[0.4.0] - 2020-02-18
Added
- BERT Model and examples
- Addition of
DistilBertForTokenClassification
andDistilBertForQuestionAnswering
model heads - Collection of activation functions (gelu, relu, mish)
- Dropout module
- Custom Linear layer, allowing a creation without bias
- Config trait allowing to deserialize from
json
files
Changed
- (BREAKING) Updated
DistilBertConfig
to use the newly createdConfig
Trait
[0.3.1] - 2020-02-16
Added
- Integration tests
Changed
- Migrated from
rust_transformers
v0.2.0 (deprecated) to `rust_tokenizers v1.0.0
[0.3.0] - 2020-02-13
Added
- Example for DistilBERT masked language modeling
- Download utilities script for DistilBERT (base and SST2)
Changed
- made
label2id
,id2label
,is_decoder
,output_past
anduse_bfloat
configuration fields optional for DistilBertConfig
[0.2.0] - 2020-02-11
Initial release
- Tensor conversion tools from Pytorch to Libtorch format
- DistilBERT model architecture
- Ready-to-use
SentimentClassifier
using a DistilBERT model fine-tuned on SST2