mirror of
https://github.com/QuivrHQ/quivr.git
synced 2024-09-11 14:36:35 +03:00
# Description
Hey,
Here's a breakdown of what I've done:
- Reducing the number of opened fd and memory footprint: Previously, for
each uploaded file, we were opening a temporary NamedTemporaryFile to
write existing content read from Supabase. However, due to the
dependency on `langchain` loader classes, we couldn't use memory buffers
for the loaders. Now, with the changes made, we only open a single
temporary file for each `process_file_and_notify`, cutting down on
excessive file opening, read syscalls, and memory buffer usage. This
could cause stability issues when ingesting and processing large volumes
of documents. Unfortunately, there is still reopening of temporary files
in some code paths but this can be improved further in later work.
- Removing `UploadFile` class from File: The `UploadFile` ( a FastAPI
abstraction over a SpooledTemporaryFile for multipart upload) was
redundant in our `File` setup since we already downloaded the file from
remote storage and read it into memory + wrote the file into a temp
file. By removing this abstraction, we streamline our code and eliminate
unnecessary complexity.
- `async` function Adjustments: I've removed the async labeling from
functions where it wasn't truly asynchronous. For instance, calling
`filter_file` for processing files isn't genuinely async, ass async file
reading isn't actually asynchronous—it [uses a threadpool for reading
the
file](9f16bf5c25/starlette/datastructures.py (L458)
)
. Given that we're already leveraging `celery` for parallelism (one
worker per core), we need to ensure that reading and processing occur in
the same thread, or at least minimize thread spawning. Additionally,
since the rest of the code isn't inherently asynchronous, our bottleneck
lies in CPU operations rather than asynchronous processing.
These changes aim to improve performance and streamline our codebase.
Let me know if you have any questions or suggestions for further
improvements!
## Checklist before requesting a review
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my code
- [x] I have ideally added tests that prove my fix is effective or that
my feature works
---------
Signed-off-by: aminediro <aminediro@github.com>
Co-authored-by: aminediro <aminediro@github.com>
Co-authored-by: Stan Girard <girard.stanislas@gmail.com>
45 lines
1.1 KiB
Python
45 lines
1.1 KiB
Python
# celery_config.py
|
|
import os
|
|
|
|
import dotenv
|
|
from celery import Celery
|
|
|
|
dotenv.load_dotenv()
|
|
|
|
CELERY_BROKER_URL = os.getenv("CELERY_BROKER_URL", "")
|
|
CELERY_BROKER_QUEUE_NAME = os.getenv("CELERY_BROKER_QUEUE_NAME", "quivr")
|
|
|
|
celery = Celery(__name__)
|
|
|
|
if CELERY_BROKER_URL.startswith("sqs"):
|
|
broker_transport_options = {
|
|
CELERY_BROKER_QUEUE_NAME: {
|
|
"my-q": {
|
|
"url": CELERY_BROKER_URL,
|
|
}
|
|
}
|
|
}
|
|
celery = Celery(
|
|
__name__,
|
|
broker=CELERY_BROKER_URL,
|
|
task_serializer="json",
|
|
task_concurrency=4,
|
|
worker_prefetch_multiplier=1,
|
|
broker_transport_options=broker_transport_options,
|
|
)
|
|
celery.conf.task_default_queue = CELERY_BROKER_QUEUE_NAME
|
|
elif CELERY_BROKER_URL.startswith("redis"):
|
|
celery = Celery(
|
|
__name__,
|
|
broker=f"{CELERY_BROKER_URL}",
|
|
backend=f"{CELERY_BROKER_URL}",
|
|
task_concurrency=4,
|
|
worker_prefetch_multiplier=2,
|
|
task_serializer="json",
|
|
)
|
|
else:
|
|
raise ValueError(f"Unsupported broker URL: {CELERY_BROKER_URL}")
|
|
|
|
|
|
celery.autodiscover_tasks(["modules.sync", "modules", "middlewares", "packages"])
|