Commit Graph

42 Commits

Author SHA1 Message Date
Stan Girard
fb0658a43d fix(docker): compose 2024-09-10 18:13:16 +02:00
Stan Girard
380cf82706
feat: quivr core 0.1 (#2970)
# Description


# Testing backend 

## Docker setup
1. Copy `.env.example` to `.env`. Some env variables were added :
EMBEDDING_DIM
2. Apply supabase migratrions : 
```sh
supabase stop
supabase db reset
supabase start
```
3. Start backend containers
```
make dev
```
## Local setup 
You can also run backend without docker.
1. Install [`rye`](https://rye.astral.sh/guide/installation/). Choose
the managed python version and set the version to 3.11
2. Run the following: 
```
cd quivr/backend
rye sync
```
3. Source `.venv` virtual env : `source .venv/bin/activate`
4. Run the backend, make sure you are running redis and supabase
API: 
```
LOG_LEVEL=debug uvicorn quivr_api.main:app --log-level debug --reload --host 0.0.0.0 --port 5050 --workers 1
```
Worker: 
```
LOG_LEVEL=debug celery -A quivr_worker.celery_worker worker -l info -E --concurrency 1
```
Notifier: 
```
LOG_LEVEL=debug python worker/quivr_worker/celery_monitor.py
```

---------

Co-authored-by: chloedia <chloedaems0@gmail.com>
Co-authored-by: aminediro <aminedirhoussi1@gmail.com>
Co-authored-by: Antoine Dewez <44063631+Zewed@users.noreply.github.com>
Co-authored-by: Chloé Daems <73901882+chloedia@users.noreply.github.com>
Co-authored-by: Zewed <dewez.antoine2@gmail.com>
2024-09-02 10:20:53 +02:00
Stan Girard
4ef5f30aa9
feat(models): all models by default (#2983)
# Description

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.

## Checklist before requesting a review

Please delete options that are not relevant.

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged

## Screenshots (if appropriate):
2024-08-08 15:36:31 -07:00
AmineDiro
a90a01ccf2
feat: quivr api send notification (#2865)
# Description

- Quivr notification service. listens to celery events and updates
knowledge status.
- Process file notification will wait for retried tasks
2024-07-15 06:37:42 -07:00
Chloé Daems
913217f682
feat: add megaparse (#2785)
Add megaparse as the main parser

---------

Co-authored-by: Stan Girard <girard.stanislas@gmail.com>
2024-07-04 02:23:15 -07:00
AmineDiro
2e75de4039
feat(backend): quivr-monorepo and quivr-core package (#2765)
# Description

closes #2722.

- Creates `quivr-monorepo` 
- Separates `quivr-core`
- Update dockerfiles and docker-compose

---------

Co-authored-by: aminediro <aminediro@github.com>
2024-06-27 03:51:01 -07:00
AmineDiro
ca93cb9062
refacto(backend): poetry package manager and chat route refactoring (#2684)
# Description
- Added package manager
- Added precommit checks
- Rewrote dependency injection of Services and Repositories
- Integrate async SQL alchemy engine
- Migrate Chat  repository to SQLModel 
- Migrated ChatHistory repository to SQLModel
- User SQLModel
- Unit test methodology with db rollback
- Unit tests ChatRepository
- Test ChatService get_history
- Brain entity SQL Model
- Promp SQLModel
- Rewrite chat/{chat_id}/question route
- updated docker files and docker compose in dev and production

Added `quivr_core` subpackages:
- Refactored KnowledgebrainQa
- Added Rag service to interface with non-rag dependencies

---------

Co-authored-by: aminediro <aminediro@github.com>
2024-06-26 00:58:55 -07:00
Stan Girard
9b4142d0ba
fix: NEXT_PUBLIC_AUTH_MODES in docker-compose.yml (#2572)
This pull request fixes the value of NEXT_PUBLIC_AUTH_MODES in the
docker-compose.yml file. The previous value was incorrect and has been
updated to the correct value.
2024-05-10 01:25:29 -07:00
Stan Girard
75a37491fb Add network configuration to docker-compose.yml 2024-04-23 00:46:47 +02:00
Stan Girard
777a6d80ea
Feat/docs category brains agents (#2469)
# Description

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.

## Checklist before requesting a review

Please delete options that are not relevant.

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged

## Screenshots (if appropriate):
2024-04-22 15:33:36 -07:00
Stan Girard
19c11944bc
feat: implement elasticache (#2234)
# Description

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.

## Checklist before requesting a review

Please delete options that are not relevant.

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged

## Screenshots (if appropriate):
2024-02-20 19:52:14 -08:00
Stan Girard
a1dbce559c Update image tags in AWS workflow and docker-compose.yml 2024-02-07 14:15:24 -08:00
Stan Girard
d562eb5e5e Update GitHub URLs 2024-02-07 13:59:48 -08:00
Stan Girard
7395dd0b44 Update URLs and dependencies 2024-02-07 13:57:44 -08:00
Stan Girard
15ba223f9c chore(docker): increased number of replicas 2024-01-20 13:51:03 -08:00
Stan Girard
b2d1d10212 feat: 🎸 supabase
migrated to supabase cli
2024-01-07 16:50:38 +01:00
renovate[bot]
fe373fdd10
chore(deps): pin dependencies (#1975)
[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [actions/checkout](https://togithub.com/actions/checkout) | action |
pinDigest | -> `b4ffde6` |
| [actions/setup-node](https://togithub.com/actions/setup-node) | action
| pinDigest | -> `b39b52d` |
| [actions/setup-python](https://togithub.com/actions/setup-python) |
action | pinDigest | -> `65d7f2d` |
|
[aws-actions/amazon-ecr-login](https://togithub.com/aws-actions/amazon-ecr-login)
| action | pinDigest | -> `2fc7ace` |
|
[aws-actions/amazon-ecs-deploy-task-definition](https://togithub.com/aws-actions/amazon-ecs-deploy-task-definition)
| action | pinDigest | -> `df96430` |
|
[aws-actions/amazon-ecs-render-task-definition](https://togithub.com/aws-actions/amazon-ecs-render-task-definition)
| action | pinDigest | -> `4225e0b` |
|
[aws-actions/configure-aws-credentials](https://togithub.com/aws-actions/configure-aws-credentials)
| action | pinDigest | -> `5fd3084` |
| darthsim/imgproxy |  | pinDigest |  -> `0facd35` |
|
[docker/build-push-action](https://togithub.com/docker/build-push-action)
| action | pinDigest | -> `0a97817` |
|
[docker/build-push-action](https://togithub.com/docker/build-push-action)
| action | pinDigest | -> `4a13e50` |
| [docker/login-action](https://togithub.com/docker/login-action) |
action | pinDigest | -> `465a078` |
| [docker/login-action](https://togithub.com/docker/login-action) |
action | pinDigest | -> `343f7c4` |
|
[docker/setup-buildx-action](https://togithub.com/docker/setup-buildx-action)
| action | pinDigest | -> `885d146` |
|
[docker/setup-buildx-action](https://togithub.com/docker/setup-buildx-action)
| action | pinDigest | -> `f95db51` |
|
[docker/setup-qemu-action](https://togithub.com/docker/setup-qemu-action)
| action | pinDigest | -> `6882732` |
|
[google-github-actions/release-please-action](https://togithub.com/google-github-actions/release-please-action)
| action | pinDigest | -> `db8f2c6` |
| kong |  | pinDigest |  -> `1b53405` |
| [pavelzw/pytest-action](https://togithub.com/pavelzw/pytest-action) |
action | pinDigest | -> `510c5e9` |
| postgrest/postgrest |  | pinDigest |  -> `23b2dab` |
| python | final | pinDigest |  -> `0c1fbb2` |
| redis |  | pinDigest |  -> `a7cee7c` |
| supabase/edge-runtime |  | pinDigest |  -> `4e02aac` |
| supabase/gotrue |  | pinDigest |  -> `b503f1f` |
| supabase/logflare |  | pinDigest |  -> `e693c78` |
| supabase/postgres |  | pinDigest |  -> `fb8387f` |
| supabase/postgres-meta |  | pinDigest |  -> `31a107d` |
| supabase/realtime |  | pinDigest |  -> `634a59e` |
| supabase/storage-api |  | pinDigest |  -> `2cd146f` |
| supabase/studio |  | pinDigest |  -> `393669f` |
| timberio/vector |  | pinDigest |  -> `4bc04ac` |

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

👻 **Immortal**: This PR will be recreated if closed unmerged. Get
[config help](https://togithub.com/renovatebot/renovate/discussions) if
that's undesired.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/StanGirard/quivr).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4xMDMuMSIsInVwZGF0ZWRJblZlciI6IjM3LjEwMy4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2024-01-04 14:00:18 +01:00
Stan Girard
5619f2e602 feat(sentry): added on celery 2023-12-06 12:20:46 +01:00
Stan Girard
475f8ddf42 fix(docker-compose): mixed up prod and dev 2023-12-05 00:26:06 +01:00
Stan Girard
1ea647b147
feat(prebuilt): prebuild backend image for faster compilation (#1815)
# Description

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.

## Checklist before requesting a review

Please delete options that are not relevant.

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged

## Screenshots (if appropriate):
2023-12-05 00:01:14 +01:00
Stan Girard
4c89812832 feat(supabase): upgraded to latest version 2023-12-03 20:26:26 +01:00
Stan Girard
10617727ab feat(dev): added devmode easy 2023-12-02 17:08:21 +01:00
Stan Girard
0aeb497bfc fix(docker): compose pull policy 2023-12-02 14:56:05 +01:00
Stan Girard
9e0bc0749f
feat: 🎸 install (#1784)
now made easy

# Description

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.

## Checklist before requesting a review

Please delete options that are not relevant.

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged

## Screenshots (if appropriate):
2023-12-02 14:50:57 +01:00
Stan Girard
87138635aa
feat(install): it now takes 30 seconds to install Quivr (#1780)
# Description

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.

## Checklist before requesting a review

Please delete options that are not relevant.

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged

## Screenshots (if appropriate):
2023-12-02 00:26:06 +01:00
Stan Girard
744eea6d43
feat: 🎸 docker reduced size by 2 (#1653)
reduced size by 2

# Description

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.

## Checklist before requesting a review

Please delete options that are not relevant.

- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged

## Screenshots (if appropriate):
2023-11-18 19:23:56 +01:00
Stan Girard
46af5b7a1b fix(docker): fixed 2023-11-14 10:03:32 +01:00
Stan Girard
e266351589 feat: 🎸 docker
added image name common
2023-11-13 10:14:24 +01:00
Mamadou DICKO
a9d2145fe6
feat: add a cron to remove onboarding more than 7 days (#1397)
https://github.com/StanGirard/quivr/issues/1385
2023-10-16 16:11:34 +02:00
Stan Girard
223d3d9102
feat(frontend): responsiveness (#1174)
* feat: 🎸 responsiveness chat

* feat: 🎸 responsive

added responsive menu

* feat: 🎸 responsive

brain management

* fix: 🐛 docker

prod
2023-09-15 01:09:26 +02:00
Stan Girard
f46ad68aa4 feat(broker): fixed url 2023-09-14 17:23:30 +02:00
Stan Girard
1d33fbd3eb
feat(file-system): added queue and filesystem (#1159)
* feat(queue): added

* feat(crawling): added queue

* fix(crawler): fixed github

* feat(docker): simplified docker compose

* feat(celery): added worker

* feat(files): now uploaded

* feat(files): missing routes

* feat(delete): added

* feat(storage): added policy and migrations

* feat(sqs): implemented

* feat(redis): added queue name variable

* fix(task): updated

* style(env): emoved unused env

* ci(tests): removed broken tests
2023-09-14 11:56:59 +02:00
Stan Girard
322ee318be
feat: the good user management (#1158)
* feat(user_management): added user management

* feat(user_management): added user management

* feat(user_management): removed print

* feat: use tanstack query for user data fecthing

* feat: add getUser to sdk

* feat: improve user page ux use tanstack query

* feat: fetch models from backend on brains settings page

* feat: update model selection on chat page

* feat: update tests

---------

Co-authored-by: mamadoudicko <mamadoudicko100@gmail.com>
2023-09-13 13:47:12 +02:00
Stan Girard
d0370ab499
feat(refacto): changed a bit of things to make better dx (#984) 2023-08-19 13:32:16 +02:00
Stan Girard
7281fd905a
feat(microservices): split into 4 quivr to better handle long services (#972)
* feat(verbose): removed

* feat(microservices): split quivr into micro services

* test(main): fixed
2023-08-18 10:18:29 +02:00
Matt
2f5088547a test: fix ci 2023-07-18 15:34:08 +01:00
Matt
e61f437ce8
Feat/backend core (#656) 2023-07-17 07:57:27 +01:00
Stan Girard
a3ca7ecb37
Back/refacto files (#240)
* feat(docker): added docker for prod

* feat(refacto): moved to modules
2023-06-03 23:12:42 +02:00
Stan Girard
72c92b1a54
VertexAI Google Cloud Palm2 Support (#226)
* feat(bard): added

* docs(readme): update

* chore(print): removed
2023-06-01 16:01:27 +02:00
shaun
a1693d94b2 Better envs 2023-05-21 21:18:55 -07:00
Stan Girard
c96436ce4b fix(docler): silent volumes 2023-05-21 02:52:22 +02:00
Stan Girard
f952d7a269
New Webapp migration (#56)
* feat(v2): loaders added

* feature: Add scroll animations

* feature: upload ui

* feature: upload multiple files

* fix: Same file name and size remove

* feat(crawler): added

* feat(parsers): v2 added more

* feat(v2): audio now working

* feat(v2): all loaders

* feat(v2): explorer

* chore: add links

* feat(api): added status in return message

* refactor(website): remove old code

* feat(upload): return type for messages

* feature: redirect to upload if ENV=local

* fix(chat): fixed some issues

* feature: respect response type

* loading state

* feature: Loading stat

* feat(v2): added explore and chat pages

* feature: modal settings

* style: Chat UI

* feature: scroll to bottom when chatting

* feature: smooth scroll in chat

* feature(anim): Slide chat in

* feature: markdown chat

* feat(explorer): list

* feat(doc): added document item

* feat(explore): added modal

* Add clarification on Project API keys and web interface for migration scripts to Readme (#58)

* fix(demo): changed link

* add support to uploading zip file (#62)

* Catch UnicodeEncodeError exception (#64)

* feature: fixed chatbar

* fix(loaders): missing argument

* fix: layout

* fix: One whole chatbox

* fix: Scroll into view

* fix(build): vercel issues

* chore(streamlit): moved to own file

* refactor(api): moved to backend folder

* feat(docker): added docker compose

* Fix a bug where langchain memories were not being cleaned (#71)

* Update README.md (#70)

* chore(streamlit): moved to own file

* refactor(api): moved to backend folder

* docs(readme): updated for new version

* docs(readme): added old readme

* docs(readme): update copy dot env file

* docs(readme): cleanup

---------

Co-authored-by: iMADi-ARCH <nandanaditya985@gmail.com>
Co-authored-by: Matt LeBel <github@lebel.io>
Co-authored-by: Evan Carlson <45178375+EvanCarlson@users.noreply.github.com>
Co-authored-by: Mustafa Hasan Khan <65130881+mustafahasankhan@users.noreply.github.com>
Co-authored-by: zhulixi <48713110+zlxxlz1026@users.noreply.github.com>
Co-authored-by: Stanisław Tuszyński <stanislaw@tuszynski.me>
2023-05-21 01:20:55 +02:00