zed-industries/zed
1
1
Fork 0
mirror of https://github.com/zed-industries/zed.git synced 2024-09-19 18:41:56 +03:00
Code Issues Packages Projects Releases Wiki Activity
11,672 Commits 672 Branches 720 Tags 177 MiB
3ad1befb11
Commit Graph

82 Commits

Author SHA1 Message Date
Antonio Scandurra
3ad1befb11 Remove unneeded logging 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-07 15:07:21 +02:00
Antonio Scandurra
ce62173534 Rename Document to Span 2023-09-06 17:03:08 +02:00
Antonio Scandurra
de0f53b39f Ensure SemanticIndex::search waits for indexing to complete 2023-09-06 11:40:59 +02:00
Antonio Scandurra
c802680084 Clip ranges returned by SemanticIndex::search 
The files may have changed since the last time they were parsed, so the
ranges returned by `SemanticIndex::search` may be out of bounds.
 2023-09-06 09:41:51 +02:00
Antonio Scandurra
95b72a73ad Re-index project when a worktree is registered 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-05 17:17:58 +02:00
Antonio Scandurra
3c70b127bd Simplify SemanticIndex::index_project 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-05 16:54:48 +02:00
Antonio Scandurra
6b1dc63fc0 Retrieve embeddings based on pending files 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-05 16:16:12 +02:00
Antonio Scandurra
7b5a41dda2 Move retrieval of embeddings from the db into reindex_changed_files 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-05 16:09:24 +02:00
Antonio Scandurra
d4cff68475 🎨 2023-09-05 15:52:36 +02:00
KCaverly
8dbc0fe033 update pragma settings for improved database performance 2023-09-01 17:07:20 -04:00
KCaverly
54235f4fb1 updated embeddings background delay to 5 minutes 
Co-authored-by: Max <max@zed.dev>
 2023-09-01 13:04:09 -04:00
KCaverly
e86964eb5d optimize insert file in vector database 
Co-authored-by: Max <max@zed.dev>
 2023-09-01 13:01:37 -04:00
KCaverly
524533cfb2 flush embeddings queue when no files are parsed for 250 milliseconds 
Co-authored-by: Antonio <antonio@zed.dev>
 2023-09-01 11:24:08 -04:00
KCaverly
c4db914f0a move embeddings queue to use single hashmap for all changed paths 
Co-authored-by: Antonio <me@as-cii.com>
 2023-09-01 08:59:25 -04:00
KCaverly
afa59abbcd WIP: work towards wiring up a embeddings_for_digest hashmap that is stored for all indexed files 2023-08-31 16:42:39 -04:00
KCaverly
50cfb067e7 fill embeddings with database values and skip during embeddings queue 2023-08-31 13:19:17 -04:00
Antonio Scandurra
220533ff1a WIP 2023-08-31 18:00:57 +02:00
Antonio Scandurra
2503d54d19 Rename Sha1 to DocumentDigest 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-08-31 18:00:36 +02:00
Antonio Scandurra
3001a46f69 Reify Embedding/Sha1 structs that can be (de)serialized from SQL 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-08-31 17:55:43 +02:00
Antonio Scandurra
c763e728d1 Write to and read from the database in a transactional way 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-08-31 16:59:54 +02:00
Antonio Scandurra
35440be98e Abstract away how database transactions are executed 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-08-31 16:54:11 +02:00
KCaverly
7d4d6c871b fix bug for truncation ensuring no valid inputs are sent to openai 2023-08-30 17:42:16 -04:00
KCaverly
5abad58b0d moved semantic index to use embeddings queue to batch and managed for atomic database writes 
Co-authored-by: Max <max@zed.dev>
 2023-08-30 16:58:45 -04:00
KCaverly
76ce52df4e move queuing to embedding_queue functionality and update embedding provider to include trait items for max tokens per batch" 
Co-authored-by: Max <max@zed.dev>
 2023-08-30 16:01:28 -04:00
KCaverly
9781047156 move truncation to parsing step leveraging the EmbeddingProvider trait 2023-08-30 12:13:26 -04:00
KCaverly
76caea80f7 add should_truncate to embedding providers 2023-08-30 11:58:45 -04:00
KCaverly
e377ada1a9 added token count to documents during parsing 2023-08-30 11:05:46 -04:00
KCaverly
a7e6a65deb reindex files in the background after they have not been edited for 10 minutes 
Co-authored-by: Max <max@zed.dev>
 2023-08-29 17:14:44 -04:00
KCaverly
4f8b95cf0d add proper handling for open ai rate limit delays 2023-08-29 15:44:51 -04:00
Max Brunsfeld
404f76739c Format let-else statements 2023-08-25 10:11:32 -07:00
KCaverly
ee97bc54cf cleaned up warnings 2023-08-25 10:38:01 +02:00
KCaverly
131950f670 add handling for Added file events to semantic index 2023-08-24 18:40:08 +02:00
KCaverly
a892a51ec3 update initialize project call to accomodate for test scenarios 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 13:46:43 +02:00
KCaverly
a1519e4c38 move semantic search project intialization to a subscribe event for workspace created 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 13:14:19 +02:00
KCaverly
e8e7b294d8 add delete files operation for remaining files in database not included in current worktree 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 12:49:20 +02:00
KCaverly
afe0e74868 remove worktree_file_mtimes in state as it is no longer used 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 12:42:41 +02:00
KCaverly
aa07872a24 accomodate for duplicate entries in indexing queue 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 12:36:43 +02:00
KCaverly
3f9f742530 update rate limiting embeddings strategy to delay less 2023-08-24 11:45:52 +02:00
KCaverly
e42b9e910e fix async calls on project updated entries to ensure that all files are updating appropriately 2023-08-23 22:28:30 +02:00
KCaverly
09fd99b1e3 moved semantic_index project intialization to queue and channel method 2023-08-23 15:09:15 +02:00
KCaverly
328b7e523c reorganized to stop the race 2023-08-22 15:01:21 +02:00
KCaverly
aabdfa210f working on initialization + index breakup 2023-08-22 14:45:27 +02:00
KCaverly
ced2b2aec3 reworked ProjectState to include additional context 2023-08-22 11:58:48 +02:00
KCaverly
3d89cd10a4 added sha1 encoding for each document 2023-08-21 16:35:57 +02:00
Piotr Osiewicz
c68b518aec chore: fix compiler warning 
Co-authored-by: Kyle <kyle@zed.dev>
 2023-08-21 13:31:45 +02:00
Piotr Osiewicz
61041b0cd1 Do not attempt to reindex a file if previous attempts have failed. 
Add doc comment to JobHandle

Co-authored-by: Kyle <kyle@zed.dev>
 2023-08-21 13:23:11 +02:00
Piotr Osiewicz
1a88444f2f Increment job counter on JobClient::new 
Co-authored-by: Kyle <kyle@zed.dev>
 2023-08-21 13:00:56 +02:00
KCaverly
def215af9f update job handle to ensure file count is consistent 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-21 12:47:43 +02:00
KCaverly
1cae4758cc manage for edge case in which file documents are larger than the allowable limit 2023-08-21 11:29:45 +02:00
Piotr Osiewicz
e2b8e6ce63 chore: fix compiler warnings 2023-08-09 01:16:24 +02:00