facebookresearch/fairseq
1
1
Fork 0
mirror of https://github.com/facebookresearch/fairseq.git synced 2024-10-26 17:32:57 +03:00
Code Issues Packages Projects Releases Wiki Activity

Fix indexing in TokenBlockDataset

Browse Source 
Summary: Pull Request resolved: https://github.com/pytorch/fairseq/pull/719

Differential Revision: D15258483

Pulled By: myleott

fbshipit-source-id: dd00daa6f1c87264c1196a77dfffc8c876ebde7f
This commit is contained in:
Myle Ott 2019-05-08 06:11:05 -07:00 committed by Facebook Github Bot
parent 0cb45bcb12
commit eddcdf08e1
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
fairseq/data/token_block_dataset.py
View File

@ -70,7 +70,7 @@ class TokenBlockDataset(FairseqDataset):
if not torch.is_tensor(sizes):
sizes = torch.tensor(sizes)
cumsum = torch.cumsum(sizes, dim=0)
self.slice_indices[0, 1] = sizes[0]
self.slice_indices[0] = [0, sizes[0]]
self.slice_indices[1:] = cumsum.unfold(0, 2, 1)
else:
raise ValueError('Invalid break_mode: ' + break_mode)