quivr/backend/api/quivr_api/models/crawler.py

import os

from pydantic import BaseModel


class CrawlWebsite(BaseModel):
    url: str
    js: bool = False
    depth: int = int(os.getenv("CRAWL_DEPTH", "1"))
    max_pages: int = 100
    max_time: int = 60
feat: quivr core 0.1 (#2970) # Description # Testing backend ## Docker setup 1. Copy `.env.example` to `.env`. Some env variables were added : EMBEDDING_DIM 2. Apply supabase migratrions : ```sh supabase stop supabase db reset supabase start ``` 3. Start backend containers ``` make dev ``` ## Local setup You can also run backend without docker. 1. Install [`rye`](https://rye.astral.sh/guide/installation/). Choose the managed python version and set the version to 3.11 2. Run the following: ``` cd quivr/backend rye sync ``` 3. Source `.venv` virtual env : `source .venv/bin/activate` 4. Run the backend, make sure you are running redis and supabase API: ``` LOG_LEVEL=debug uvicorn quivr_api.main:app --log-level debug --reload --host 0.0.0.0 --port 5050 --workers 1 ``` Worker: ``` LOG_LEVEL=debug celery -A quivr_worker.celery_worker worker -l info -E --concurrency 1 ``` Notifier: ``` LOG_LEVEL=debug python worker/quivr_worker/celery_monitor.py ``` --------- Co-authored-by: chloedia <chloedaems0@gmail.com> Co-authored-by: aminediro <aminedirhoussi1@gmail.com> Co-authored-by: Antoine Dewez <44063631+Zewed@users.noreply.github.com> Co-authored-by: Chloé Daems <73901882+chloedia@users.noreply.github.com> Co-authored-by: Zewed <dewez.antoine2@gmail.com> 2024-09-02 11:20:53 +03:00			`import os`

			`from pydantic import BaseModel`


			`class CrawlWebsite(BaseModel):`
			`url: str`
			`js: bool = False`
			`depth: int = int(os.getenv("CRAWL_DEPTH", "1"))`
			`max_pages: int = 100`
			`max_time: int = 60`