zed-industries/zed
1
1
Fork 0
mirror of https://github.com/zed-industries/zed.git synced 2024-11-08 07:35:01 +03:00
Code Issues Packages Projects Releases Wiki Activity
18,223 Commits 706 Branches 767 Tags 284 MiB
a44fc24445
Commit Graph

189 Commits

Author SHA1 Message Date
Antonio Scandurra
eda7e00645 Implement SemanticIndex::status and use it in project search 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-07 19:39:30 +02:00
Antonio Scandurra
65e17e212d Eagerly index project on workspace creation if it was indexed before 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-07 18:51:55 +02:00
Antonio Scandurra
a45c8c380f 💄 2023-09-07 15:25:23 +02:00
Antonio Scandurra
757a285852 Keep dropping the documents table if it exists 
This is because we renamed `documents` to `spans`.

Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-07 15:15:16 +02:00
Antonio Scandurra
93b889a93b Merge remote-tracking branch 'origin/main' into semantic-search-watch-worktrees 2023-09-07 15:07:46 +02:00
Antonio Scandurra
3ad1befb11 Remove unneeded logging 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-07 15:07:21 +02:00
KCaverly
265d02a583 update request timeout for open ai embeddings 2023-09-06 15:09:46 -04:00
KCaverly
17237f748c update token_count for OpenAIEmbeddings to accomodate for truncation 2023-09-06 15:09:15 -04:00
Antonio Scandurra
ce62173534 Rename Document to Span 2023-09-06 17:03:08 +02:00
Antonio Scandurra
de0f53b39f Ensure SemanticIndex::search waits for indexing to complete 2023-09-06 11:40:59 +02:00
Antonio Scandurra
c802680084 Clip ranges returned by SemanticIndex::search 
The files may have changed since the last time they were parsed, so the
ranges returned by `SemanticIndex::search` may be out of bounds.
 2023-09-06 09:41:51 +02:00
Antonio Scandurra
95b72a73ad Re-index project when a worktree is registered 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-05 17:17:58 +02:00
Antonio Scandurra
3c70b127bd Simplify SemanticIndex::index_project 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-05 16:54:48 +02:00
Antonio Scandurra
6b1dc63fc0 Retrieve embeddings based on pending files 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-05 16:16:12 +02:00
Antonio Scandurra
7b5a41dda2 Move retrieval of embeddings from the db into reindex_changed_files 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-05 16:09:24 +02:00
Antonio Scandurra
d4cff68475 🎨 2023-09-05 15:52:36 +02:00
KCaverly
8dbc0fe033 update pragma settings for improved database performance 2023-09-01 17:07:20 -04:00
KCaverly
54235f4fb1 updated embeddings background delay to 5 minutes 
Co-authored-by: Max <max@zed.dev>
 2023-09-01 13:04:09 -04:00
KCaverly
e86964eb5d optimize insert file in vector database 
Co-authored-by: Max <max@zed.dev>
 2023-09-01 13:01:37 -04:00
KCaverly
524533cfb2 flush embeddings queue when no files are parsed for 250 milliseconds 
Co-authored-by: Antonio <antonio@zed.dev>
 2023-09-01 11:24:08 -04:00
KCaverly
c4db914f0a move embeddings queue to use single hashmap for all changed paths 
Co-authored-by: Antonio <me@as-cii.com>
 2023-09-01 08:59:25 -04:00
KCaverly
afa59abbcd WIP: work towards wiring up a embeddings_for_digest hashmap that is stored for all indexed files 2023-08-31 16:42:39 -04:00
KCaverly
50cfb067e7 fill embeddings with database values and skip during embeddings queue 2023-08-31 13:19:17 -04:00
Antonio Scandurra
220533ff1a WIP 2023-08-31 18:00:57 +02:00
Antonio Scandurra
2503d54d19 Rename Sha1 to DocumentDigest 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-08-31 18:00:36 +02:00
Antonio Scandurra
3001a46f69 Reify Embedding/Sha1 structs that can be (de)serialized from SQL 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-08-31 17:55:43 +02:00
Antonio Scandurra
c763e728d1 Write to and read from the database in a transactional way 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-08-31 16:59:54 +02:00
Antonio Scandurra
35440be98e Abstract away how database transactions are executed 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-08-31 16:54:11 +02:00
KCaverly
7d4d6c871b fix bug for truncation ensuring no valid inputs are sent to openai 2023-08-30 17:42:16 -04:00
KCaverly
5abad58b0d moved semantic index to use embeddings queue to batch and managed for atomic database writes 
Co-authored-by: Max <max@zed.dev>
 2023-08-30 16:58:45 -04:00
KCaverly
76ce52df4e move queuing to embedding_queue functionality and update embedding provider to include trait items for max tokens per batch" 
Co-authored-by: Max <max@zed.dev>
 2023-08-30 16:01:28 -04:00
KCaverly
9781047156 move truncation to parsing step leveraging the EmbeddingProvider trait 2023-08-30 12:13:26 -04:00
KCaverly
76caea80f7 add should_truncate to embedding providers 2023-08-30 11:58:45 -04:00
KCaverly
e377ada1a9 added token count to documents during parsing 2023-08-30 11:05:46 -04:00
KCaverly
a7e6a65deb reindex files in the background after they have not been edited for 10 minutes 
Co-authored-by: Max <max@zed.dev>
 2023-08-29 17:14:44 -04:00
KCaverly
4f8b95cf0d add proper handling for open ai rate limit delays 2023-08-29 15:44:51 -04:00
Max Brunsfeld
404f76739c Format let-else statements 2023-08-25 10:11:32 -07:00
KCaverly
ee97bc54cf cleaned up warnings 2023-08-25 10:38:01 +02:00
KCaverly
131950f670 add handling for Added file events to semantic index 2023-08-24 18:40:08 +02:00
KCaverly
a892a51ec3 update initialize project call to accomodate for test scenarios 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 13:46:43 +02:00
KCaverly
a1519e4c38 move semantic search project intialization to a subscribe event for workspace created 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 13:14:19 +02:00
KCaverly
e8e7b294d8 add delete files operation for remaining files in database not included in current worktree 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 12:49:20 +02:00
KCaverly
afe0e74868 remove worktree_file_mtimes in state as it is no longer used 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 12:42:41 +02:00
KCaverly
aa07872a24 accomodate for duplicate entries in indexing queue 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-24 12:36:43 +02:00
KCaverly
3f9f742530 update rate limiting embeddings strategy to delay less 2023-08-24 11:45:52 +02:00
KCaverly
e42b9e910e fix async calls on project updated entries to ensure that all files are updating appropriately 2023-08-23 22:28:30 +02:00
KCaverly
09fd99b1e3 moved semantic_index project intialization to queue and channel method 2023-08-23 15:09:15 +02:00
KCaverly
328b7e523c reorganized to stop the race 2023-08-22 15:01:21 +02:00
KCaverly
aabdfa210f working on initialization + index breakup 2023-08-22 14:45:27 +02:00
KCaverly
ced2b2aec3 reworked ProjectState to include additional context 2023-08-22 11:58:48 +02:00
KCaverly
3d89cd10a4 added sha1 encoding for each document 2023-08-21 16:35:57 +02:00
Piotr Osiewicz
c68b518aec chore: fix compiler warning 
Co-authored-by: Kyle <kyle@zed.dev>
 2023-08-21 13:31:45 +02:00
Piotr Osiewicz
61041b0cd1 Do not attempt to reindex a file if previous attempts have failed. 
Add doc comment to JobHandle

Co-authored-by: Kyle <kyle@zed.dev>
 2023-08-21 13:23:11 +02:00
Piotr Osiewicz
1a88444f2f Increment job counter on JobClient::new 
Co-authored-by: Kyle <kyle@zed.dev>
 2023-08-21 13:00:56 +02:00
KCaverly
def215af9f update job handle to ensure file count is consistent 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-21 12:47:43 +02:00
KCaverly
1cae4758cc manage for edge case in which file documents are larger than the allowable limit 2023-08-21 11:29:45 +02:00
Piotr Osiewicz
e2b8e6ce63 chore: fix compiler warnings 2023-08-09 01:16:24 +02:00
Piotr Osiewicz
31fb503418 Rewrite permission queries (it no longer blocks) 
Co-authored-by: Kyle <kyle@zed.dev>
Co-authored-by: Max <max@zed.dev>
 2023-08-03 20:26:56 +02:00
KCaverly
8dd330a160 add semantic search prompt for the indexing permission on first search 
Co-authored-by: Piotr <piotr@zed.dev>
 2023-08-03 12:52:20 -04:00
KCaverly
300c693d55 catchup with main 2023-08-01 10:40:38 -04:00
KCaverly
e221f23018 add support for markdown files to semantic search 2023-08-01 10:30:34 -04:00
KCaverly
9a50b43eaa add templating languages html, erb, heex, svelte as entire parseable file types 2023-07-31 21:03:02 -04:00
KCaverly
599f674827 add php support for semantic search 2023-07-31 16:36:09 -04:00
KCaverly
89edb3d1b5 fix templating bug for parseable entire files 2023-07-31 11:41:18 -04:00
KCaverly
ca4e21881e add ruby support for semantic search 2023-07-31 10:54:30 -04:00
KCaverly
a5dd8dd0a9 add lua embedding query for semantic search 2023-07-31 10:02:28 -04:00
Kirill Bulatov
cf6e524c9a Make project search includes and excludes more user-friendly 
Allow search results that start with the include/exclude path part
 2023-07-28 12:56:44 +03:00
KCaverly
fbede4a5a3 removed old code 2023-07-26 17:11:30 -04:00
KCaverly
5c48729c7c managed for small batch size edge case in semantic search 2023-07-26 16:56:41 -04:00
KCaverly
98fde36834 batch search queries in the vector database 2023-07-26 16:36:39 -04:00
KCaverly
0b61c93a25 ensure semantic search is not enabled on stable 2023-07-26 10:22:33 -04:00
KCaverly
394a105639 fix warnings 2023-07-26 10:03:30 -04:00
KCaverly
ca6f7d8a80 add worktree previously indexed functionality to vector db 2023-07-26 09:17:04 -04:00
KCaverly
75999204ad update project search to only show semantic button visible with semantic_index enabled 2023-07-25 16:26:37 -04:00
KCaverly
cdceddd2cc update semantic index tests for elixir 2023-07-25 15:20:35 -04:00
KCaverly
97c3d97792 update semantic index tests for cpp 2023-07-25 13:30:38 -04:00
KCaverly
c86096a886 update semantic index tests for javascript 2023-07-25 10:38:37 -04:00
Max Brunsfeld
81b05f2a08 Optimize glob filtering of semantic search 
Co-authored-by: Kyle <kyle@zed.dev>
 2023-07-20 14:23:11 -07:00
KCaverly
e02d6bc0d4 add glob filtering functionality to semantic search 2023-07-20 13:46:27 -04:00
KCaverly
efe973ebe2 add embedding query for json with nested arrays and strings 
Co-authored-by: maxbrunsfeld <max@zed.dev>
 2023-07-19 16:52:44 -04:00
KCaverly
9809ec3d70 update treesitter parsing to accomodate for collapsed nested functions 
Co-authored-by: maxbrunsfeld <max@zed.dev>
 2023-07-19 15:47:05 -04:00
KCaverly
0e071919a0 parellelize embedding api calls 2023-07-18 16:09:44 -04:00
Max Brunsfeld
8d0614ce74 Populate project search results multi-buffer from semantic search 
Co-authored-by: Kyle <kyle@zed.dev>
 2023-07-18 11:44:58 -07:00
KCaverly
80ef92a3e1 fix db schema update process to ensure all tables are dropped 2023-07-18 11:14:13 -04:00
KCaverly
ed1b1a5ccd update logging for open ai embedding and remove redundant truncation 2023-07-18 11:00:21 -04:00
KCaverly
b9fdfd60f0 catch up with main 2023-07-18 10:26:28 -04:00
Max Brunsfeld
afc4c10ec1 Start work on exposing semantic search via project search view 
Co-authored-by: Kyle <kyle@zed.dev>
 2023-07-17 18:10:51 -07:00
KCaverly
d83c4ffb07 remove debug logging for enabled settings 2023-07-17 17:09:51 -04:00
KCaverly
8b42f5b1b3 rename vector_store crate to semantic_index 2023-07-17 17:06:10 -04:00