Commit Graph

59 Commits

Author SHA1 Message Date
Stan Girard
4d730448af
chore: packages (#2527)
Update python packages
2024-05-01 11:20:15 -07:00
Stan Girard
bdb115ad0a
feat(profiler): Add pyinstrument package and update Makefile and backend code (#2512)
This pull request adds the pyinstrument package and updates the Makefile
and backend code. The pyinstrument package is used for profiling and the
Makefile and backend code have been modified to support profiling.
2024-04-28 06:10:21 -07:00
Stan Girard
f656dbcb42
feat(reranker): Add flashrank and contextual compression retriever (#2480)
This pull request adds the flashrank and contextual compression
retriever to the codebase. The flashrank reranker model is used for
compression, and the contextual compression retriever combines the base
compressor and base retriever to improve document retrieval.
2024-04-24 10:44:31 -07:00
Stan Girard
309666f3fd
feat: Update Docker build commands and dependencies (#2441)
This pull request updates the Docker build commands and dependencies.
2024-04-17 14:37:32 -07:00
Stan Girard
b62297341f
feat(assistants): Add PDF generation functionality and nice emails (#2436)
This pull request adds PDF generation functionality and improves the
formatting of emails. It includes a new module for generating PDFs using
the fpdf2 library and updates the email templates to use markdown
formatting for better readability.

<!--
ELLIPSIS_HIDDEN
-->
----

| <a href="https://ellipsis.dev" target="_blank"><img
src="https://avatars.githubusercontent.com/u/80834858?s=400&u=31e596315b0d8f7465b3ee670f25cea677299c96&v=4"
alt="Ellipsis" width="30px" height="30px"/></a> | 🚀 This PR
description was created by [Ellipsis](https://www.ellipsis.dev) for
commit ccecff77a5. |
|--------|--------|

### Summary:
This PR introduces PDF generation functionality, improves email
formatting, sanitizes filenames, updates various files and dependencies,
and updates `docker-compose.dev.yml`.

**Key points**:
- Added PDF generation functionality using the `fpdf2` library
- Improved email formatting
- Sanitized filenames using the `unidecode` library
- Updated `backend/modules/assistant/ito/ito.py` to generate PDFs and
sanitize filenames
- Added `pdf_generator.py` in `backend/modules/assistant/ito/utils/` for
PDF generation
- Updated `Pipfile` and `Pipfile.lock` in
`backend/modules/assistant/ito/utils/` to include `unidecode`
- Updated `requirements.txt` with new dependencies
- Updated `docker-compose.dev.yml`


----
Generated with ❤️ by [ellipsis.dev](https://www.ellipsis.dev)

<!--
ELLIPSIS_HIDDEN
-->
2024-04-17 06:00:31 -07:00
Stan Girard
861931f231 tests: added new way of testing less prone to errors 2024-04-15 18:22:21 +02:00
Stan Girard
38589d32cf
feat(backend): add RAG evaluation using Ragas (#2429)
# Description

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.

## Checklist before requesting a review

Please delete options that are not relevant.

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged

## Screenshots (if appropriate):


<!--
ELLIPSIS_HIDDEN
-->


----

| <a href="https://ellipsis.dev" target="_blank"><img
src="https://avatars.githubusercontent.com/u/80834858?s=400&u=31e596315b0d8f7465b3ee670f25cea677299c96&v=4"
alt="Ellipsis" width="30px" height="30px"/></a> | 🚀 This PR
description was created by [Ellipsis](https://www.ellipsis.dev) for
commit 5350e0a071. |
|--------|--------|

### Summary:
This PR adds a new script for evaluating the RAG model using the Ragas
library, with results saved as a JSON file and printed to the console.

**Key points**:
- New script `run_evaluation.py` added to
`backend/tests/ragas_evaluation/`.
- The script processes documents, generates replies using a QuivrRAG
chain, and evaluates the replies using specified metrics.
- Results are saved as a JSON file and printed to the console.
- The script can be run from the command line with various options.


----
Generated with ❤️ by [ellipsis.dev](https://www.ellipsis.dev)



<!--
ELLIPSIS_HIDDEN
-->

---------

Co-authored-by: Damien Mourot <damien.mourot@gmail.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2024-04-15 06:53:48 -07:00
Stan Girard
c773677b46
feat: llamaparse & diff agent (#2427)
This pull request updates the dependencies in the Pipfile to their
latest versions.
2024-04-12 09:06:50 -07:00
Damien Mourot
d42323d34a
feat(docker image): Docker image Optimized for CPU-only env (#2368)
# Description

Optimize the docker image for cpu-only environment. (PyTorch in cpu
mode, cuda not installed etc) This should reduce the backend docker
image size by ~2

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.

## Checklist before requesting a review

Please delete options that are not relevant.

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged

## Screenshots (if appropriate):
2024-03-20 23:38:07 -07:00
Stan Girard
6754829d8b
chore: update packages backend (#2339)
# Description

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.

## Checklist before requesting a review

Please delete options that are not relevant.

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged

## Screenshots (if appropriate):
2024-03-11 19:23:39 -07:00
Stan Girard
29e762612c
feat: Update to newest version of litellm (#2307)
This pull request includes a feature update to the newest version of
litellm. This update brings in the latest enhancements and bug fixes to
improve the functionality and performance of the software.
2024-03-06 11:03:47 -08:00
Stan Girard
ba5ef60362
feat(integration): implementation (#2191)
# Description

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.

## Checklist before requesting a review

Please delete options that are not relevant.

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged

## Screenshots (if appropriate):
2024-02-14 20:07:53 -08:00
Stan Girard
08e015af6c
feat(lcel): migrated to lcel and pydantic (#2185)
# Description

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.

## Checklist before requesting a review

Please delete options that are not relevant.

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged

## Screenshots (if appropriate):

---------

Co-authored-by: Antoine Dewez <44063631+Zewed@users.noreply.github.com>
2024-02-14 14:01:35 -08:00
Stan Girard
2ba3bc1f07
feat: 🎸 ocr (#2187)
added ocr

# Description

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.

## Checklist before requesting a review

Please delete options that are not relevant.

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged

## Screenshots (if appropriate):
2024-02-12 19:56:20 -08:00
Stan Girard
4d91d1cadc
feat(integrations): integration with Notion in the backend (#2123)
moved to brains

# Description

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.

## Checklist before requesting a review

Please delete options that are not relevant.

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged

## Screenshots (if appropriate):

---------

Co-authored-by: Antoine Dewez <44063631+Zewed@users.noreply.github.com>
2024-02-05 21:02:46 -08:00
Stan Girard
d801450341 feat: 🎸 apm
implemented datadog
2024-01-27 12:50:21 -08:00
Stan Girard
dfdb294c50
feat: 🎸 api (#2078)
adding metadata to api

# Description

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.

## Checklist before requesting a review

Please delete options that are not relevant.

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged

## Screenshots (if appropriate):
2024-01-25 15:56:46 -08:00
Stan Girard
6ac7962ec7
feat: 🎸 posthog (#1978)
user settings to premium

# Description

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.

## Checklist before requesting a review

Please delete options that are not relevant.

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged

## Screenshots (if appropriate):
2024-01-04 13:57:13 +01:00
Mamadou DICKO
7e6209a30b
feat: add generate_answer function to support non streamed response for api brain (#1847)
Issue: https://github.com/StanGirard/quivr/issues/1841

Demo:


https://github.com/StanGirard/quivr/assets/63923024/56dfa85c-6c2e-4e55-8eda-4723e58ced1d
2023-12-07 14:52:37 +01:00
Stan Girard
8ba376e0a1
feat(requirements): update aws version (#1819)
# Description

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.

## Checklist before requesting a review

Please delete options that are not relevant.

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged

## Screenshots (if appropriate):
2023-12-05 13:49:34 +01:00
Stan Girard
e1cde0fcb4
Feat/local llm bug fix (#1758)
# Description

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.

## Checklist before requesting a review

Please delete options that are not relevant.

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged

## Screenshots (if appropriate):
2023-11-29 19:17:16 +01:00
Mamadou DICKO
07618877a9
fix: revert implement local llms (#1749) 2023-11-29 13:19:22 +01:00
Mamadou DICKO
d5b45cbad1
chore: downgrade versions (#1748) 2023-11-29 13:18:52 +01:00
Stan Girard
7e4f711f07
feat: implement local llms (#1745)
upgraded

# Description

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.

## Checklist before requesting a review

Please delete options that are not relevant.

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged

## Screenshots (if appropriate):
2023-11-29 00:07:04 +01:00
Stan Girard
56a5c81fe4
fix: 🐛 sentry (#1716)
reduce sampling

# Description

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.

## Checklist before requesting a review

Please delete options that are not relevant.

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged

## Screenshots (if appropriate):
2023-11-27 10:08:00 +01:00
Pascal Gula
8bbe6e7054
Adds pytesseract, tesseract and poopler-utils (#1648)
To enable the ingestion of copy protected PDF via OCR instead of text
extraction

# Description

Copy protected PDF can't be properly imported via the standard langchain
loader.

See the following errors:

```
2023-11-15 14:16:31,927 [INFO] models.files: Computing documents from file Cradle to Cradle Criteria for the built environmen.pdf
[nltk_data] Downloading package punkt to
[nltk_data]     /home/pascal_gula_luccid_ai/nltk_data...
[nltk_data]   Unzipping tokenizers/punkt.zip.
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /home/pascal_gula_luccid_ai/nltk_data...
[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.
Error processing file: detectron2 is not installed, pytesseract is not installed and the text of the PDF is not extractable. To process this file, install detectron2, install pytesseract, or remove copy protection from the PDF.
```

```
2023-11-15 15:04:14,624 [INFO] models.files: Computing documents from file Cradle to Cradle Criteria for the built environmen.pdf
Error processing file: Unable to get page count. Is poppler installed and in PATH?
```

```
023-11-15 15:59:11,886 [INFO] models.files: Computing documents from file Cradle to Cradle Criteria for the built environmen.pdf
Error processing file: tesseract is not installed or it's not in your PATH. See README file for more information.
```


## Checklist before requesting a review

Please delete options that are not relevant.

- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my code
- [x] I have commented hard-to-understand areas
- [x] I have ideally added tests that prove my fix is effective or that
my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] Any dependent changes have been merged

## Screenshots (if appropriate):

None

Co-authored-by: Stan Girard <girard.stanislas@gmail.com>
2023-11-22 17:26:11 +01:00
Stan Girard
744eea6d43
feat: 🎸 docker reduced size by 2 (#1653)
reduced size by 2

# Description

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.

## Checklist before requesting a review

Please delete options that are not relevant.

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged

## Screenshots (if appropriate):
2023-11-18 19:23:56 +01:00
Stan Girard
5a3f284785
test(all): added (#1624)
# Description

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.

## Checklist before requesting a review

Please delete options that are not relevant.

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged

## Screenshots (if appropriate):
2023-11-14 08:36:44 +01:00
Stan Girard
6017fa2e9c test(contact): added contact route with mock 2023-11-13 17:58:11 +01:00
Stan Girard
6bc9dd1894
ci: 🎡 tests (#1615)
fixed

# Description

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.

## Checklist before requesting a review

Please delete options that are not relevant.

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged

## Screenshots (if appropriate):
2023-11-13 13:53:25 +01:00
Mamadou DICKO
db5a6e4b9b
feat: allow users to chat with apis (#1612)
You can now create a brain which can fetch data from external APIs with
or without authentification

- POST query example with authentification 


https://github.com/StanGirard/quivr/assets/63923024/15013ba9-dedb-4f24-9e06-49daad9de7f3


- Get query example with authentification and search params



https://github.com/StanGirard/quivr/assets/63923024/1763875d-a8e9-4478-b07c-e99ca7337942


- Get query without authentification and search params



https://github.com/StanGirard/quivr/assets/63923024/f2742963-790d-4cb2-864a-8173979b650a
2023-11-09 16:58:51 +01:00
Mamadou DICKO
c47548d3cd
feat: setup premium feature backend (#1467)
Issue: https://github.com/StanGirard/quivr/issues/1468
2023-10-23 18:19:04 +02:00
Stan Girard
e62c3e0579 feat(litellm): adding huggingface compatibility mistral 2023-10-03 17:18:50 +02:00
Stan Girard
160588cfae feat(litellm): improved 2023-10-03 10:12:44 +02:00
Stan Girard
ead1ae86fc
feat(user_settings): increased (#1291) 2023-09-30 22:32:53 +02:00
Stan Girard
d8e188788f
fix(gpt-3.5-instruct): bug and new version of node (#1228) 2023-09-20 16:16:50 +02:00
Stan Girard
1d33fbd3eb
feat(file-system): added queue and filesystem (#1159)
* feat(queue): added

* feat(crawling): added queue

* fix(crawler): fixed github

* feat(docker): simplified docker compose

* feat(celery): added worker

* feat(files): now uploaded

* feat(files): missing routes

* feat(delete): added

* feat(storage): added policy and migrations

* feat(sqs): implemented

* feat(redis): added queue name variable

* fix(task): updated

* style(env): emoved unused env

* ci(tests): removed broken tests
2023-09-14 11:56:59 +02:00
Ishaan Jaff
02964c4077
feat(liteLLM): Add support for Azure OpenAI, Palm, Claude-2, Llama2, CodeLlama (100+LLMs) (#1097)
* v0 litellm

* bump versions
2023-09-05 17:38:19 +02:00
Zineb El Bachiri
3821502c6d
add xlsx and xls parser (#997) 2023-08-21 12:56:48 +02:00
Stan Girard
d0370ab499
feat(refacto): changed a bit of things to make better dx (#984) 2023-08-19 13:32:16 +02:00
Matt
e61f437ce8
Feat/backend core (#656) 2023-07-17 07:57:27 +01:00
Matt
8fbb4b2d91
fix: gpt4all (#595)
* fix: gpt4all

* fix: pyright

* Update backend/llm/openai.py

* fix: remove backend tag

* fix: typing

* feat: qa_base class

* fix: pyright

* fix: model_path not found
2023-07-11 20:15:56 +02:00
Zineb El Bachiri
f837a6e9b9
Feat/shareable brains send link be (#599)
* 🗃️ new table for invitations to subscribe to brain

*  new BrainSubscription class

*  new subscription router

* 👽️ add RESEND_API_KEY to .env in BE

* 📦 add 'resend' lib to requirements

* ♻️ fix some stanGPT
2023-07-11 18:20:31 +02:00
Mamadou DICKO
9e9f531c99
Feat/static analysis (#582)
* feat: add static analysis

* chore: update Makefile add static analysis script

* chore: add vscode extensions recommandations
2023-07-10 14:27:49 +02:00
Stan Girard
fbd1e17018
feat(sentry): added sentry (#443) 2023-07-01 21:12:13 +02:00
Mamadou DICKO
59fe7b089b
feat(chat): use openai function for answer (#354)
* feat(chat): use openai function for answer (backend)

* feat(chat): use openai function for answer (frontend)

* chore: refacto BrainPicking

* feat: update chat creation logic

* feat: simplify chat system logic

* feat: set default method to gpt-3.5-turbo-0613

* feat: use user own openai key

* feat(chat): slightly improve prompts

* feat: add global error interceptor

* feat: remove unused endpoints

* docs: update chat system doc

* chore(linter): add unused import remove config

* feat: improve dx

* feat: improve OpenAiFunctionBasedAnswerGenerator prompt
2023-06-22 17:50:06 +02:00
Matt
83fde0aeea
feat: private llm (#360)
* feat: private llm

* Update backend/vectorstore/supabase.py

* Update backend/vectorstore/supabase.py
2023-06-22 09:45:35 +01:00
Mamadou DICKO
e1a740472f
Feat: chat name edit (#343)
* feat(chat): add name update

* chore(linting): add flake8

* feat: add chat name edit
2023-06-20 09:54:23 +02:00
Stan Girard
f4e85db187 fix(llm): using wrong llm probably because of breaking change in langchain 2023-06-14 22:15:48 +02:00
Matt
33f49ee289
feat: user can create api keys (#329)
* feat: user can create api keys

* fix: linting on build

* Update backend/routes/api_key_routes.py

* chore: rename and refactor AuthBearer

* chore: add types
2023-06-14 21:21:13 +02:00