# Description
Hey,
Here's a breakdown of what I've done:
- Reducing the number of opened fd and memory footprint: Previously, for
each uploaded file, we were opening a temporary NamedTemporaryFile to
write existing content read from Supabase. However, due to the
dependency on `langchain` loader classes, we couldn't use memory buffers
for the loaders. Now, with the changes made, we only open a single
temporary file for each `process_file_and_notify`, cutting down on
excessive file opening, read syscalls, and memory buffer usage. This
could cause stability issues when ingesting and processing large volumes
of documents. Unfortunately, there is still reopening of temporary files
in some code paths but this can be improved further in later work.
- Removing `UploadFile` class from File: The `UploadFile` ( a FastAPI
abstraction over a SpooledTemporaryFile for multipart upload) was
redundant in our `File` setup since we already downloaded the file from
remote storage and read it into memory + wrote the file into a temp
file. By removing this abstraction, we streamline our code and eliminate
unnecessary complexity.
- `async` function Adjustments: I've removed the async labeling from
functions where it wasn't truly asynchronous. For instance, calling
`filter_file` for processing files isn't genuinely async, ass async file
reading isn't actually asynchronous—it [uses a threadpool for reading
the
file](
|
||
---|---|---|
.aws | ||
.devcontainer | ||
.github | ||
.vscode | ||
backend | ||
cms/quivr | ||
docs | ||
frontend | ||
project.inlang | ||
scripts/quivr_team_only | ||
.env.example | ||
.flake8 | ||
.gitignore | ||
CHANGELOG.md | ||
docker-compose.dev.yml | ||
docker-compose.redis.yml | ||
docker-compose.yml | ||
LICENSE | ||
logo.png | ||
Makefile | ||
Pipfile | ||
Pipfile.lock | ||
README.md | ||
renovate.json | ||
vercel.json |
Quivr - Your Second Brain, Empowered by Generative AI
Quivr, your second brain, utilizes the power of GenerativeAI to be your personal assistant ! Think of it as Obsidian, but turbocharged with AI capabilities.
Key Features 🎯
- Fast and Efficient: Designed with speed and efficiency at its core. Quivr ensures rapid access to your data.
- Secure: Your data, your control. Always.
- OS Compatible: Ubuntu 22 or newer.
- File Compatibility: Text, Markdown, PDF, Powerpoint, Excel, CSV, Word, Audio, Video
- Open Source: Freedom is beautiful, and so is Quivr. Open source and free to use.
- Public/Private: Share your brains with your users via a public link, or keep them private.
- Marketplace: Share your brains with the world, or use other people's brains to boost your productivity.
- Offline Mode: Quivr works offline, so you can access your data anytime, anywhere.
Demo Highlights 🎥
https://github.com/quivrhq/quivr/assets/19614572/a6463b73-76c7-4bc0-978d-70562dca71f5
Getting Started 🚀
Follow these instructions to get a copy of the project up and running on your local machine for development and testing purposes.
You can find everything on the documentation.
Prerequisites 📋
Ensure you have the following installed:
- Docker
- Docker Compose
60 seconds Installation 💽
You can find the installation video here.
-
Step 0: Supabase CLI
Follow the instructions here to install the Supabase CLI that is required.
supabase -v # Check that the installation worked
-
Step 1: Clone the repository:
git clone https://github.com/quivrhq/quivr.git && cd quivr
-
Step 2: Copy the
.env.example
filescp .env.example .env
-
Step 3: Update the
.env
filesvim .env # or emacs or vscode or nano
Update OPENAI_API_KEY in the
.env
file.You just need to update the
OPENAI_API_KEY
variable in the.env
file. You can get your API key here. You need to create an account first. And put your credit card information. Don't worry, you won't be charged unless you use the API. You can find more information about the pricing here. -
Step 4: Launch the project
cd backend && supabase start
and then
cd ../ docker compose pull docker compose up
If you have a Mac, go to Docker Desktop > Settings > General and check that the "file sharing implementation" is set to
VirtioFS
.If you are a developer, you can run the project in development mode with the following command:
docker compose -f docker-compose.dev.yml up --build
-
Step 5: Login to the app
You can now sign in to the app with
admin@quivr.app
&admin
. You can access the app at http://localhost:3000/login.You can access Quivr backend API at http://localhost:5050/docs
You can access supabase at http://localhost:54323
Updating Quivr 🚀
-
Step 1: Pull the latest changes
git pull
-
Step 2: Update the migration
supabase migration up
Contributors ✨
Thanks go to these wonderful people:
Contribute 🤝
Did you get a pull request? Open it, and we'll review it as soon as possible. Check out our project board here to see what we're currently focused on, and feel free to bring your fresh ideas to the table!
Partners ❤️
This project would not be possible without the support of our partners. Thank you for your support!
![YCombinator](https://upload.wikimedia.org/wikipedia/commons/thumb/b/b2/Y_Combinator_logo.svg/1200px-Y_Combinator_logo.svg.png)
License 📄
This project is licensed under the Apache 2.0 License - see the LICENSE file for details