# Description
`quivr-core`
- Generate a fixture to simulate a model with function calling
- Monkey patch `QuivrQARAG` stream
- Tests function
`quivr-api`
- Fixes empty API responses
- Fixes non function calling models
---------
Co-authored-by: Stan Girard <girard.stanislas@gmail.com>
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
This pull request adds a new field called `bulk_id` to the
`CreateNotification` and `Notification` models. The `bulk_id` field is
an optional UUID that can be used for bulk operations.
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
---------
Co-authored-by: Stan Girard <girard.stanislas@gmail.com>
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
This pull request updates the versions of quivr-core and chainlit in the
requirements.txt file. The quivr-core version is changed from 0.0.8 to
0.0.8[base], and the chainlit version is changed from 1.1.306 to
1.1.306.
The commit adds a new Quivr chatbot example to the repository. The
example demonstrates how to create a simple chatbot using Quivr and
Chainlit. Users can upload a text file and ask questions about its
content. The commit includes the necessary files, installation
instructions, and usage guidelines.
# Description
- Defined quivr-core `ChatHistory`
- `ChatHistory` can be iterated over in tuples of
`HumanMessage,AIMessage`
- Brain appends to the chatHistory once response is received
- Brain holds a dict of chats and defines the default chat (TODO: define
a system of selecting the chats)
- Wrote test
- Updated `QuivrQARAG` to use `ChatHistory` as input
This pull request updates the pre-commit hooks in the
`.pre-commit-config.yaml` file to the latest versions. By doing so, we
ensure that our codebase benefits from the latest bug fixes and
improvements provided by the pre-commit community.
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
This commit adds a new GitHub Actions workflow file
`backend-core-tests.yml` to the repository. This workflow is triggered
on push and pull request events, specifically for changes made to the
`backend/core` directory. The workflow runs tests for the backend/core
module using a Tika server as a service. It checks the health of the
Tika server before running the tests. The workflow sets up Python,
installs dependencies using Poetry, and runs the tests using pytest.
This commit is necessary to automate the testing process for the
backend/core module and ensure that the tests are run consistently and
reliably.
Co-authored-by: aminediro <aminedirhoussi@gmail.com>
# Description
- Introduce `LLMEndpoint` class: wrapper around a `BaseChatLLM` to load
OpenAI compatible models
- Add `brain.search(...)` function to retrieve
- Test with test-coverage:
- Added ability to load brain `from langchain.Document`
- Configured mypy and poetry lock in `.pre-commit.yaml`
# Test coverage
---------
Co-authored-by: aminediro <aminedirhoussi@gmail.com>
This commit updates the GitHub Actions workflows for the backend services (quivr-com-backend, quivr-com, raise-frontend, raise, theodo-backend, and theodo-frontend) to trigger on tag pushes instead of branch pushes. This change allows for more controlled deployments and ensures that only tagged versions are deployed to the respective environments.
This commit adds the release-please-core workflow and configuration files for the python package in the backend. The release-please workflow is triggered on push to the main branch and on manual workflow dispatch. It sets up Python, installs Poetry, configures Poetry, and runs release-please for the backend/core package. The release-please configuration file specifies the release type, package name, bump patch for minor pre-major, changelog notes type, include v in tag, tag separator, and component.
The code changes handle the case when the access token is not present in
the result of acquiring the token. It now logs an error message with the
specific reason for the failure and raises an HTTPException with a
corresponding detail message. This improves the error handling and
provides more information to the user.
Note: The commit message follows the convention used in the recent user
commits and adheres to the established format.
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
closes#2782.
Changes `sqlalchemy` connection pooling config :
- **pool_pre_ping=True** : pessimistic disconnect handling.
> - It is critical to note that the pre-ping approach does not
accommodate **for connections dropped in the middle of transactions or
other SQL operations** ! But this should only happen if we lose the
database either due to network of DB server restart.
- **pool_size=10**, with no db side pooling for now, if 6 uvicorn
process workers are spawned (or 6 instances of the backed) we have a
pool of 60 processes connecting to the database.
- **pool_recycle=1800** : Recycles the pool every 30min
Added additional session config :
- expire_on_commit=False, When True, all instances will be fully expired
after each commit, so that all attribute/object access subsequent to a
completed transaction will load from the most recent database state.
- autoflush=False, query operations will issue a Session.flush() call to
this Session before proceeding. We are calling `session.commit` (which
flushes) on each repository method so ne need to reflush on subsequent
access
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
This pull request updates the docker-compose files to specify the
platform for the backend services as linux/amd64. This ensures that the
services are built and run specifically for the amd64 architecture.
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
Co-authored-by: Stan Girard <stan@quivr.app>
# Description
closes#2756
- Fixed torchvision cpu version in poetry
- Moved test dependencies to dev group
Co-authored-by: aminediro <aminediro@github.com>
# Description
- Added package manager
- Added precommit checks
- Rewrote dependency injection of Services and Repositories
- Integrate async SQL alchemy engine
- Migrate Chat repository to SQLModel
- Migrated ChatHistory repository to SQLModel
- User SQLModel
- Unit test methodology with db rollback
- Unit tests ChatRepository
- Test ChatService get_history
- Brain entity SQL Model
- Promp SQLModel
- Rewrite chat/{chat_id}/question route
- updated docker files and docker compose in dev and production
Added `quivr_core` subpackages:
- Refactored KnowledgebrainQa
- Added Rag service to interface with non-rag dependencies
---------
Co-authored-by: aminediro <aminediro@github.com>
Fixes#2718
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
---------
Co-authored-by: Stan Girard <stan@quivr.app>
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
Co-authored-by: Stan Girard <stan@quivr.app>
This commit adds timezone conversion to the premium user check in the
`check_if_is_premium_user` function. It imports the `pytz` library and
retrieves the current time in the "Europe/Paris" timezone. This ensures
that the comparison of the current period end with the current time is
accurate, taking into account the correct timezone. The Paris time is
logged for debugging purposes.
Co-authored-by: Stan Girard <stan@quivr.app>
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
---------
Co-authored-by: Stan Girard <stan@quivr.app>
"This pull request adds a new celery task called
`check_if_is_premium_user` that checks if a user is a premium user based
on their subscription status. The task retrieves the list of active
subscriptions and the list of customers from the Supabase database. It
then matches the subscriptions with the customers and updates the user
settings with the corresponding premium features if a match is found. If
a user is not found or their subscription is expired, the user settings
are deleted. This task will run periodically to keep the user settings
up to date with the subscription status.
---------
Co-authored-by: Stan Girard <stan@quivr.app>
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
---------
Co-authored-by: Stan Girard <stan@quivr.app>
This pull request adds error handling for the syncing process in the
tasks.py file. Previously, if an error occurred during the syncing
process, the program would crash. With this change, any exceptions that
occur during syncing will be caught and logged, allowing the program to
continue running. This improves the stability and reliability of the
syncing functionality.
Co-authored-by: Stan Girard <stan@quivr.app>
This commit adds a new utility function `remove_special_characters` to
the `normalize.py` module in the `sync/utils` directory. The function
removes special characters from file names by normalizing the input
string and using regular expressions to remove non-alphanumeric
characters.
The function is then used in the `list_files.py` module in the
`sync/utils` directory to normalize the names of files retrieved from
Google Drive and Azure Drive. This ensures that the file names are free
of special characters, improving consistency and compatibility with
other parts of the system.
Co-authored-by: Stan Girard <stan@quivr.app>
The code changes in `google_sync_routes.py` update the authorization URL
for Google authentication. The `prompt` parameter is added with the
value "consent" to ensure that users are prompted to grant consent when
authorizing the application. This improves the user experience and
ensures that the necessary permissions are obtained.
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
This pull request updates the license to include enterprise features.
The license now specifies that certain folders or files within the
distribution may be subject to different license terms and conditions,
which will be specified in a separate LICENSE file or within the file
itself. Additionally, a new Quivr Enterprise license has been added,
which restricts the use of the software to production environments only
if the user has a valid Quivr Enterprise license for the correct number
of user seats. The license also clarifies that modifications and patches
to the software can only be used with a valid Quivr Enterprise license.
Finally, the license includes a disclaimer of warranties and limitations
of liability.
This commit adds a new optional boolean field, force_sync, to the
SyncsActiveUpdateInput class in the sync module. The force_sync field
allows users to manually trigger a sync operation, bypassing the regular
sync interval. By default, force_sync is set to False.
The force_sync field is used in the GoogleSyncUtils and AzureSyncUtils
classes to determine whether to perform a sync operation even if the
regular sync interval has not elapsed. If force_sync is True, the
last_synced timestamp is updated and the sync operation is executed.
This enhancement provides more flexibility and control over the
synchronization process, allowing users to manually trigger sync
operations when needed.
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
This pull request adds support for recursive folder retrieval in the
`get_files_folder_user_sync` method. Previously, the method only
retrieved files from the specified folder, but now it can also retrieve
files from all subfolders recursively. This enhancement improves the
functionality and flexibility of the method, allowing for more
comprehensive file retrieval in sync operations.
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
This pull request includes updates to the `docker-compose.dev.yml` and
`Dockerfile.dev` files. The changes aim to improve performance and fix
bugs. The updates include:
- Removing unnecessary workers configuration in the
`docker-compose.dev.yml` file.
- Updating the base image in the `Dockerfile.dev` to use a slim version.
- Adjusting the schedule for a specific task in the code.
- Modifying the time interval for retrieving active syncs.
- Changing the loader class for processing PowerPoint files.
- Refactoring the file existence check logic.
- Adding debug logs for file existence check and file removal.
- Adjusting the file synchronization logic.
These changes are intended to enhance the performance and stability of
the application.
# Description
Hey,
Here's a breakdown of what I've done:
- Reducing the number of opened fd and memory footprint: Previously, for
each uploaded file, we were opening a temporary NamedTemporaryFile to
write existing content read from Supabase. However, due to the
dependency on `langchain` loader classes, we couldn't use memory buffers
for the loaders. Now, with the changes made, we only open a single
temporary file for each `process_file_and_notify`, cutting down on
excessive file opening, read syscalls, and memory buffer usage. This
could cause stability issues when ingesting and processing large volumes
of documents. Unfortunately, there is still reopening of temporary files
in some code paths but this can be improved further in later work.
- Removing `UploadFile` class from File: The `UploadFile` ( a FastAPI
abstraction over a SpooledTemporaryFile for multipart upload) was
redundant in our `File` setup since we already downloaded the file from
remote storage and read it into memory + wrote the file into a temp
file. By removing this abstraction, we streamline our code and eliminate
unnecessary complexity.
- `async` function Adjustments: I've removed the async labeling from
functions where it wasn't truly asynchronous. For instance, calling
`filter_file` for processing files isn't genuinely async, ass async file
reading isn't actually asynchronous—it [uses a threadpool for reading
the
file](9f16bf5c25/starlette/datastructures.py (L458))
. Given that we're already leveraging `celery` for parallelism (one
worker per core), we need to ensure that reading and processing occur in
the same thread, or at least minimize thread spawning. Additionally,
since the rest of the code isn't inherently asynchronous, our bottleneck
lies in CPU operations rather than asynchronous processing.
These changes aim to improve performance and streamline our codebase.
Let me know if you have any questions or suggestions for further
improvements!
## Checklist before requesting a review
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my code
- [x] I have ideally added tests that prove my fix is effective or that
my feature works
---------
Signed-off-by: aminediro <aminediro@github.com>
Co-authored-by: aminediro <aminediro@github.com>
Co-authored-by: Stan Girard <girard.stanislas@gmail.com>
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
This pull request updates the parsing instructions in the `common.py`
file for the `llamaparse` feature. The previous parsing instruction for
transforming checkboxes into text has been modified to also extract
tables and transform them into key-value pairs. Additionally, the
instruction now allows for duplicate keys if needed. The example
instructions have also been updated to provide clearer examples for both
tables and checkboxes.
# Description
Change the prompt of the thoughts feature to have more steps.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
---------
Co-authored-by: chloedia <chloedaems0@gmail.com>
This pull request refactors the generate_answer and generate_stream
functions in order to improve code readability and maintainability. It
also adds new fields to the cited_answer model and updates the system
message template.
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
This pull request fixes the sender email address in the
`resend_invitation_email` function in the `resend_invitation_email.py`
file. The `from` field has been changed to `sender` to ensure that the
correct email address is used when sending the invitation email.
This pull request updates the ChatLiteLLM model to "gpt-4o" and adds a
row-level security (RLS) optimization for notifications. It also
includes a new SQL script to drop and create a policy for allowing user
access to all notifications.
This pull request adds support for the gpt-4o model to the existing
codebase. It includes changes to the BrainConfig, openAiFreeModels,
defineMaxTokens, model_compatible_with_function_calling, create_graph,
main, and process_assistant functions.
This pull request fixes the import statements for OllamaEmbeddings in
multiple files. The import statements are updated to use the correct
package name "langchain_community.embeddings" instead of
"langchain.embeddings.ollama". This ensures that the code can be
compiled and executed without any import errors.
This pull request adds comprehensive docstrings to the Brain classes
within the `backend/modules/brain/integrations` directory, enhancing
code documentation and readability. The changes include:
- **BigBrain (`Big/Brain.py`)**: Adds a class-level docstring explaining
the purpose and functionality of the BigBrain class, along with
method-level docstrings detailing the operations performed by each
method.
- **ClaudeBrain (`Claude/Brain.py`)**: Introduces a class-level
docstring that describes the ClaudeBrain class's integration with the
Claude model for conversational AI capabilities, and method-level
docstrings that clarify the purpose of each method.
- **GPT4Brain (`GPT4/Brain.py`)**: Updates include a detailed
class-level docstring outlining the GPT4Brain's integration with GPT-4
for real-time answers and tool support, along with method-level
docstrings explaining the functionality of each method.
- **NotionBrain (`Notion/Brain.py`)**: Adds a class-level docstring that
describes the NotionBrain's role in leveraging Notion data for
knowledge-based responses.
- **ProxyBrain (`Proxy/Brain.py`)**: Incorporates a class-level
docstring explaining the ProxyBrain's function as a dynamic language
model selector and method-level docstrings detailing the operations of
each method.
These additions ensure that each Brain class and its methods are
well-documented, providing clear insights into their purposes and
functionalities.
---
For more details, open the [Copilot Workspace
session](https://copilot-workspace.githubnext.com/QuivrHQ/quivr?shareId=b4e301ad-828e-4424-95ec-6e378d5d3849).
Updates the GPT-4 documentation and the `GPT4Brain` class to include
detailed information about the tools available for GPT4Brain and their
use cases.
- **Documentation (`docs/brains/gpt4.mdx`):**
- Adds a new section titled "Tools Available for GPT4Brain" that
describes specific tools: WebSearchTool, ImageGeneratorTool,
URLReaderTool, and EmailSenderTool.
- Provides use cases for each tool, demonstrating how they can be
utilized within GPT4Brain for various scenarios, such as generating
images, reading content from URLs, and sending emails.
- **Code (`backend/modules/brain/integrations/GPT4/Brain.py`):**
- Updates the class documentation to include information about the tools
available for GPT4Brain and outlines use cases for WebSearchTool,
ImageGeneratorTool, URLReaderTool, and EmailSenderTool.
- Maintains the existing functionality of the `GPT4Brain` class,
ensuring compatibility with the newly documented tools and use cases.
---
For more details, open the [Copilot Workspace
session](https://copilot-workspace.githubnext.com/QuivrHQ/quivr?shareId=2c2c1666-e5fb-4a06-bb08-ca967f4fe276).
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
This pull request fixes the value of NEXT_PUBLIC_AUTH_MODES in the
docker-compose.yml file. The previous value was incorrect and has been
updated to the correct value.
This pull request adds a new feature to generate images using the OpenAI
DALL-E model. The `ImageGeneratorTool` class is implemented to handle
the image generation functionality.
This pull request adds a GitHub Actions workflow for building and
pushing Docker images to Amazon ECR. The workflow is triggered on every
push to the main branch and includes steps for configuring AWS
credentials, logging in to Amazon ECR, GitHub Container Registry, and
Docker Hub, setting up Docker Buildx, creating a Docker cache storage
backend, and building, tagging, and pushing the Docker image to Amazon
ECR.
This pull request adds the Playwright library for web crawling. It
includes the necessary dependencies and updates the code to use
Playwright for crawling websites.
# Description
Delete the replacement of non ASCII characters into spaces
## Checklist before requesting a review
Please delete options that are not relevant.
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [x] New and existing unit tests pass locally with my changes
- [x] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
This pull request adds a new config parameter to the
`conversational_qa_chain` function. The config parameter allows for
passing metadata, specifically the conversation ID, to the function.
This change ensures that the conversation ID is included in the metadata
when invoking the `conversational_qa_chain` function.
This pull request adds the ProxyBrain integration to the project. The
ProxyBrain class is responsible for handling conversational QA and
generating answers based on the provided chat history and question.
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
---------
Co-authored-by: Stan Girard <girard.stanislas@gmail.com>
This pull request fixes the parsing instruction in the common.py file.
The result_type has been corrected to "markdown" and the
parsing_instruction has been updated to handle checkboxes, tables, and
other elements that are hard to parse in a meaningful way.
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
This pull request adds Llama Parse integration for complex document
parsing in Quivr. Llama Parse is a tool from Llama Index that allows you
to read complex documents in Quivr. It provides an API key that needs to
be added to the `.env` file as `LLAMA_CLOUD_API_KEY`. Once configured,
you can use the Llama Parse tool to read `pdf`, `docx`, and `doc` files
in Quivr.
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
This pull request adds caching for the Supabase client and database
instances in order to improve performance and reduce unnecessary API
calls. The `get_supabase_client()` and `get_supabase_db()` functions now
check if the instances have already been created and return the cached
instances if available. This avoids creating new instances for every
function call, resulting in faster execution times.
This pull request adds the pyinstrument package and updates the Makefile
and backend code. The pyinstrument package is used for profiling and the
Makefile and backend code have been modified to support profiling.
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
This pull request adds a ci-migration script that sets the project ID
from an environment variable and runs the supabase link and supabase db
push commands. This script will be used for continuous integration
purposes.
This pull request removes the citation metadata from the generate_answer
and generate_stream functions. The citation metadata was previously
being added to the streamed_chat_history and metadata dictionaries, but
it is no longer necessary. This change improves the efficiency and
clarity of the code.
This pull request updates the chunk size and overlap parameters in the
File class to improve performance. It also increases the top_n value in
the compressor for both the CohereRerank and FlashrankRerank models.
Additionally, it ensures that the page content is encoded in UTF-8
before processing.
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
This pull request updates the Dockerfile to include the installation of
the Supabase CLI. The Supabase CLI is required for interacting with the
Supabase backend. This update ensures that the Supabase CLI is installed
in the Docker image, allowing developers to easily use the Supabase CLI
within their Docker environment.
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
… to the DB API
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
This pull request updates the chunk overlap value in the File class from
300 to 200. This change reduces the overlap between chunks, improving
the performance of chunking operations.
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
This pull request adds the flashrank and contextual compression
retriever to the codebase. The flashrank reranker model is used for
compression, and the contextual compression retriever combines the base
compressor and base retriever to improve document retrieval.
This pull request updates the API documentation to include new sections
on configuring Quivr and contacting the Quivr team. It also removes the
"API Brains" section from the documentation.
This pull request fixes the issue of duplicate sources in the model
response and adds metadata to the response. It removes duplicate sources
with the same name and creates a list of unique sources. Additionally,
it includes the generated URLs and sources in the metadata of the model
response.
This pull request refactors the GPT4Brain and KnowledgeBrainQA classes
to add the functionality of saving non-streaming answers. It includes
changes to the `generate_answer` method and the addition of the
`save_non_streaming_answer` method. This enhancement improves the
overall functionality and performance of the code.
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [ ] My code follows the style guidelines of this project
- [ ] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
# Description
Please include a summary of the changes and the related issue. Please
also include relevant motivation and context.
## Checklist before requesting a review
Please delete options that are not relevant.
- [x] My code follows the style guidelines of this project
- [x] I have performed a self-review of my code
- [ ] I have commented hard-to-understand areas
- [ ] I have ideally added tests that prove my fix is effective or that
my feature works
- [ ] New and existing unit tests pass locally with my changes
- [ ] Any dependent changes have been merged
## Screenshots (if appropriate):
<!--
ELLIPSIS_HIDDEN
-->
----
| 🚀 This description was created by
[Ellipsis](https://www.ellipsis.dev) for commit
997576d577 |
|--------|
### Summary:
This PR involves a significant refactoring of the codebase, with file
and class relocations and renames, and updated import paths, without
introducing new functionality.
**Key points**:
- Significant refactoring of the codebase for improved organization and
clarity.
- File and class relocations and renames.
- Updated import paths to reflect new locations and names.
- No new functionality introduced.
----
Generated with ❤️ by [ellipsis.dev](https://www.ellipsis.dev)
<!--
ELLIPSIS_HIDDEN
-->
