zed-industries/zed
1
1
Fork 0
mirror of https://github.com/zed-industries/zed.git synced 2024-11-08 07:35:01 +03:00
Code Issues Packages Projects Releases Wiki Activity
11,865 Commits 706 Branches 767 Tags 284 MiB
65e17e212d
Commit Graph

88 Commits

Author SHA1 Message Date
Antonio Scandurra
65e17e212d Eagerly index project on workspace creation if it was indexed before 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-07 18:51:55 +02:00
Antonio Scandurra
a45c8c380f 💄 2023-09-07 15:25:23 +02:00
Antonio Scandurra
757a285852 Keep dropping the documents table if it exists 
This is because we renamed `documents` to `spans`.

Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-07 15:15:16 +02:00
Antonio Scandurra
93b889a93b Merge remote-tracking branch 'origin/main' into semantic-search-watch-worktrees 2023-09-07 15:07:46 +02:00
Antonio Scandurra
3ad1befb11 Remove unneeded logging 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-07 15:07:21 +02:00
KCaverly
265d02a583 update request timeout for open ai embeddings 2023-09-06 15:09:46 -04:00
KCaverly
17237f748c update token_count for OpenAIEmbeddings to accomodate for truncation 2023-09-06 15:09:15 -04:00
Antonio Scandurra
ce62173534 Rename Document to Span 2023-09-06 17:03:08 +02:00
Antonio Scandurra
de0f53b39f Ensure SemanticIndex::search waits for indexing to complete 2023-09-06 11:40:59 +02:00
Antonio Scandurra
c802680084 Clip ranges returned by SemanticIndex::search 
The files may have changed since the last time they were parsed, so the
ranges returned by `SemanticIndex::search` may be out of bounds.
 2023-09-06 09:41:51 +02:00
Antonio Scandurra
95b72a73ad Re-index project when a worktree is registered 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-05 17:17:58 +02:00
Antonio Scandurra
3c70b127bd Simplify SemanticIndex::index_project 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-05 16:54:48 +02:00
Antonio Scandurra
6b1dc63fc0 Retrieve embeddings based on pending files 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-05 16:16:12 +02:00
Antonio Scandurra
7b5a41dda2 Move retrieval of embeddings from the db into reindex_changed_files 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-05 16:09:24 +02:00
Antonio Scandurra
d4cff68475 🎨 2023-09-05 15:52:36 +02:00
KCaverly
8dbc0fe033 update pragma settings for improved database performance 2023-09-01 17:07:20 -04:00
KCaverly
54235f4fb1 updated embeddings background delay to 5 minutes 
Co-authored-by: Max <max@zed.dev>
 2023-09-01 13:04:09 -04:00
KCaverly
e86964eb5d optimize insert file in vector database 
Co-authored-by: Max <max@zed.dev>
 2023-09-01 13:01:37 -04:00
KCaverly
524533cfb2 flush embeddings queue when no files are parsed for 250 milliseconds 
Co-authored-by: Antonio <antonio@zed.dev>
 2023-09-01 11:24:08 -04:00
KCaverly
c4db914f0a move embeddings queue to use single hashmap for all changed paths 
Co-authored-by: Antonio <me@as-cii.com>
 2023-09-01 08:59:25 -04:00
KCaverly
afa59abbcd WIP: work towards wiring up a embeddings_for_digest hashmap that is stored for all indexed files 2023-08-31 16:42:39 -04:00
KCaverly
50cfb067e7 fill embeddings with database values and skip during embeddings queue 2023-08-31 13:19:17 -04:00
Antonio Scandurra
220533ff1a WIP 2023-08-31 18:00:57 +02:00
Antonio Scandurra
2503d54d19 Rename Sha1 to DocumentDigest 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-08-31 18:00:36 +02:00
Antonio Scandurra
3001a46f69 Reify Embedding/Sha1 structs that can be (de)serialized from SQL 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-08-31 17:55:43 +02:00
Antonio Scandurra
c763e728d1 Write to and read from the database in a transactional way 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-08-31 16:59:54 +02:00
Antonio Scandurra
35440be98e Abstract away how database transactions are executed 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-08-31 16:54:11 +02:00
KCaverly
7d4d6c871b fix bug for truncation ensuring no valid inputs are sent to openai 2023-08-30 17:42:16 -04:00
KCaverly
5abad58b0d moved semantic index to use embeddings queue to batch and managed for atomic database writes 
Co-authored-by: Max <max@zed.dev>
 2023-08-30 16:58:45 -04:00
KCaverly
76ce52df4e move queuing to embedding_queue functionality and update embedding provider to include trait items for max tokens per batch" 
Co-authored-by: Max <max@zed.dev>
 2023-08-30 16:01:28 -04:00
KCaverly
9781047156 move truncation to parsing step leveraging the EmbeddingProvider trait 2023-08-30 12:13:26 -04:00
KCaverly
76caea80f7 add should_truncate to embedding providers 2023-08-30 11:58:45 -04:00
KCaverly
e377ada1a9 added token count to documents during parsing 2023-08-30 11:05:46 -04:00
KCaverly
a7e6a65deb reindex files in the background after they have not been edited for 10 minutes 
Co-authored-by: Max <max@zed.dev>
 2023-08-29 17:14:44 -04:00
KCaverly
4f8b95cf0d add proper handling for open ai rate limit delays 2023-08-29 15:44:51 -04:00
Max Brunsfeld
404f76739c Format let-else statements 2023-08-25 10:11:32 -07:00
KCaverly
ee97bc54cf cleaned up warnings 2023-08-25 10:38:01 +02:00
KCaverly
131950f670 add handling for Added file events to semantic index 2023-08-24 18:40:08 +02:00
KCaverly
a892a51ec3 update initialize project call to accomodate for test scenarios 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 13:46:43 +02:00
KCaverly
a1519e4c38 move semantic search project intialization to a subscribe event for workspace created 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 13:14:19 +02:00
KCaverly
e8e7b294d8 add delete files operation for remaining files in database not included in current worktree 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 12:49:20 +02:00
KCaverly
afe0e74868 remove worktree_file_mtimes in state as it is no longer used 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 12:42:41 +02:00
KCaverly
aa07872a24 accomodate for duplicate entries in indexing queue 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 12:36:43 +02:00
KCaverly
3f9f742530 update rate limiting embeddings strategy to delay less 2023-08-24 11:45:52 +02:00
KCaverly
e42b9e910e fix async calls on project updated entries to ensure that all files are updating appropriately 2023-08-23 22:28:30 +02:00
KCaverly
09fd99b1e3 moved semantic_index project intialization to queue and channel method 2023-08-23 15:09:15 +02:00
KCaverly
328b7e523c reorganized to stop the race 2023-08-22 15:01:21 +02:00
KCaverly
aabdfa210f working on initialization + index breakup 2023-08-22 14:45:27 +02:00
KCaverly
ced2b2aec3 reworked ProjectState to include additional context 2023-08-22 11:58:48 +02:00
KCaverly
3d89cd10a4 added sha1 encoding for each document 2023-08-21 16:35:57 +02:00