# Description
Major PR which, among other things, introduces the possibility of easily
customizing the retrieval workflows. Workflows are based on LangGraph,
and can be customized using a [yaml configuration
file](core/tests/test_llm_endpoint.py), and adding the implementation of
the nodes logic into
[quivr_rag_langgraph.py](1a0c98437a/backend/core/quivr_core/quivr_rag_langgraph.py)
This is a first, simple implementation that will significantly evolve in
the coming weeks to enable more complex workflows (for instance, with
conditional nodes). We also plan to adopt a similar approach for the
ingestion part, i.e. to enable user to easily customize the ingestion
pipeline.
Closes CORE-195, CORE-203, CORE-204
## Checklist before requesting a review
Please delete options that are not relevant.
- [X] My code follows the style guidelines of this project
- [X] I have performed a self-review of my code
- [X] I have commented hard-to-understand areas
- [X] I have ideally added tests that prove my fix is effective or that
my feature works
- [X] New and existing unit tests pass locally with my changes
- [X] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
- Save and load brain to disk:
```python
async def main():
with tempfile.NamedTemporaryFile(mode="w", suffix=".txt") as temp_file:
temp_file.write("Gold is a liquid of blue-like colour.")
temp_file.flush()
brain = await Brain.afrom_files(name="test_brain", file_paths=[temp_file.name])
save_path = await brain.save("/home/amine/.local/quivr")
brain_loaded = Brain.load(save_path)
brain_loaded.print_info()
```
# TODO:
- Loading all chat history
- Loading from other vector stores, PG for example can be great ...
# Description
# Testing backend
## Docker setup
1. Copy `.env.example` to `.env`. Some env variables were added :
EMBEDDING_DIM
2. Apply supabase migratrions :
```sh
supabase stop
supabase db reset
supabase start
```
3. Start backend containers
```
make dev
```
## Local setup
You can also run backend without docker.
1. Install [`rye`](https://rye.astral.sh/guide/installation/). Choose
the managed python version and set the version to 3.11
2. Run the following:
```
cd quivr/backend
rye sync
```
3. Source `.venv` virtual env : `source .venv/bin/activate`
4. Run the backend, make sure you are running redis and supabase
API:
```
LOG_LEVEL=debug uvicorn quivr_api.main:app --log-level debug --reload --host 0.0.0.0 --port 5050 --workers 1
```
Worker:
```
LOG_LEVEL=debug celery -A quivr_worker.celery_worker worker -l info -E --concurrency 1
```
Notifier:
```
LOG_LEVEL=debug python worker/quivr_worker/celery_monitor.py
```
---------
Co-authored-by: chloedia <chloedaems0@gmail.com>
Co-authored-by: aminediro <aminedirhoussi1@gmail.com>
Co-authored-by: Antoine Dewez <44063631+Zewed@users.noreply.github.com>
Co-authored-by: Chloé Daems <73901882+chloedia@users.noreply.github.com>
Co-authored-by: Zewed <dewez.antoine2@gmail.com>
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
---------
Co-authored-by: AmineDiro <aminedirhoussi1@gmail.com>
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
- Moved `quivr-api` parser to `quivr_core.processor.implementations` by
Dynamically creating classes on the fly that inherit from
`ProcessorBase`
- Defined a priority based based system to automagically register the
"important" processor that we can import at runtime
- Wrote extensive tests for the registry
- Added support file extensions
### Next steps
- Find a way to have correct LSP autocomplete on the dynamically
generated processors
- Test that processor are imported correctly based on the installed
packages in environment ( using tox) ?
# Description
- Created registry processor logic for automagically adding processors
to quivr_core based Entrypoints
- Added a langchain_community free `SimpleTxtParser` for the quivr_core
base package
- Added tests
- Added brain_info
- Enriched parsed documents metadata based on quivr_file metadata
used Rich for `Brain.print_info()` to get a better output:
# Description
`quivr-core`
- Generate a fixture to simulate a model with function calling
- Monkey patch `QuivrQARAG` stream
- Tests function
`quivr-api`
- Fixes empty API responses
- Fixes non function calling models
---------
Co-authored-by: Stan Girard <girard.stanislas@gmail.com>
# Description
- Defined quivr-core `ChatHistory`
- `ChatHistory` can be iterated over in tuples of
`HumanMessage,AIMessage`
- Brain appends to the chatHistory once response is received
- Brain holds a dict of chats and defines the default chat (TODO: define
a system of selecting the chats)
- Wrote test
- Updated `QuivrQARAG` to use `ChatHistory` as input
# Description
- Introduce `LLMEndpoint` class: wrapper around a `BaseChatLLM` to load
OpenAI compatible models
- Add `brain.search(...)` function to retrieve
- Test with test-coverage:
- Added ability to load brain `from langchain.Document`
- Configured mypy and poetry lock in `.pre-commit.yaml`
# Test coverage
---------
Co-authored-by: aminediro <aminedirhoussi@gmail.com>
* 🗃️ update and add fields in brains table
* ✨ update endpoints for more brain attribute
* ✨ new set as default brain endpoint
* 🔥 remove update brain with file
* ✏️ fix wrong auto imports
* 🐛 fix max tokens for model in front
* 🚑 post instead of put to set default brain
* 🚑 update brain creation endpoint with new fields
* ♻️ use Single Responsibility Principle on brains_subscription
* ✨ new brain subscription endpoints for invited user
* 📝 add documentation to endpoints
* 🎨 add base_frontend_url to send custom url for brain share
* ✏️ brains instead of brain in url
* ✨ use origin in header for frontend url in subscription email
* 🚚 move and remove unused code
* ✨ new subscription API for BE endpoints in frontend
* ✨ new addBrain to add a shared brain in frontend
* 🥚 new hook for brain invitations
* ✨ new page for brain invitation
* ✨ change frontend url to copy for brain subscription
* ✏️ call RBAC with wrapper function
* ✏️ last typos
