QuivrHQ/quivr
1
1
Fork 0
mirror of https://github.com/QuivrHQ/quivr.git synced 2024-08-16 18:00:23 +03:00
Code Issues Packages Projects Releases Wiki Activity

feat: Update file loading logic in File model (#2857)

Browse Source
This commit is contained in:
Stan Girard 2024-07-12 16:50:27 +02:00 committed by GitHub
parent 5cc1ae5423
commit 77e1cfb77f
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 2 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
backend/api/quivr_api/models/files.py
View File

@ -42,7 +42,8 @@ class File(BaseModel):
"""
logger.info(f"Computing documents from file {self.file_name}")
loader = loader_class(self.tmp_file_path)
documents = [loader.load()]
documents = []
documents.extend(loader.load())
text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
chunk_size=self.chunk_size, chunk_overlap=self.chunk_overlap