zed-industries/zed
1
1
Fork 0
mirror of https://github.com/zed-industries/zed.git synced 2024-09-19 18:41:56 +03:00
Code Issues Packages Projects Releases Wiki Activity
19,292 Commits 672 Branches 720 Tags 177 MiB
cd6acff635
Commit Graph

203 Commits

Author SHA1 Message Date
Joseph T. Lyons
516a8790b9 Add gpt-4-1106-preview model 2023-11-14 08:28:57 -05:00
Kirill Bulatov
eb8a0e7148 Uncomment persistence tests 2023-11-03 12:38:09 +02:00
Kirill Bulatov
d673efebd2 Add prettier workspace resolution test 2023-11-03 11:02:50 +02:00
KCaverly
a2c3971ad6 moved authentication for the semantic index into the EmbeddingProvider 2023-10-30 10:02:27 -04:00
KCaverly
6c8bb4b05e ensure OpenAIEmbeddingProvider is using the provider credentials 2023-10-27 08:33:35 +02:00
KCaverly
ca82ec8e8e fixed truncation error in fake language model 2023-10-26 14:05:55 +02:00
KCaverly
3447a9478c updated authentication for embedding provider 2023-10-26 11:18:16 +02:00
KCaverly
71bc35d241 catchup with main 2023-10-25 16:31:00 +02:00
KCaverly
8ffe5a3ec7 move keychain access into semantic index as opposed to on init 2023-10-24 13:26:37 +02:00
KCaverly
4e90e45999 move embedding truncation to base model 2023-10-23 14:07:45 +02:00
KCaverly
2b780ee7b2 add base model to EmbeddingProvider, not yet leveraged for truncation 2023-10-22 15:00:09 +02:00
KCaverly
d1dec8314a move OpenAIEmbeddings to OpenAIEmbeddingProvider in providers folder 2023-10-22 14:46:22 +02:00
KCaverly
106115676d update semantic search to use keychain as fallback 2023-10-21 10:19:50 -04:00
Kirill Bulatov
525ff6bf74 Remove zed -> ... -> semantic_index -> zed Cargo dependency cycle 2023-10-13 10:27:08 +03:00
Kirill Bulatov
4b15a2bd63 Rebase fixes 2023-10-11 12:56:29 +03:00
Max Brunsfeld
95342c8c33 Merge branch 'main' into channel-changes 2023-10-03 17:52:28 -07:00
Mikayla
6007c8705c
Upgrade SeaORM to latest version, also upgrade sqlite bindings, rustqlite, and remove SeaQuery 
co-authored-by: Max <max@zed.dev>
 2023-10-03 12:16:53 -07:00
Kyle Caverly
cf5d89d13c
Leverage embeddings query to collapse syntax nodes if not selected (#3067) 
Reverts zed-industries/zed#3049
 2023-10-03 12:02:47 +03:00
KCaverly
9f160537ef move collapsed only matches outside item parent in embedding.scm 2023-10-03 11:56:45 +03:00
KCaverly
3c12e711a4 add scheme for full parseable files in semantic index 2023-09-29 14:35:02 -04:00
KCaverly
0e6fd645fd leverage embeddings len returned in construction matrix multiplication 2023-09-27 10:33:04 -04:00
KCaverly
abefa2738b removed blas and increase batch size for vector search 2023-09-27 09:43:23 -04:00
KCaverly
e75f56a0f2 move to system blas 2023-09-26 12:39:22 -04:00
KCaverly
330a71d28b fixed bug limiting number of results returned 2023-09-26 10:29:55 -04:00
KCaverly
ea278b5b12 ensure desc sort and cleanup unused imports 2023-09-26 09:53:49 -04:00
Mikayla Maki
591ec02cea
Add support for the experimental Next LS for Elixir (#3024) 
This is a PR I built for a friend of a friend at StrangeLoop, who is
making a much better LSP for elixir that elixir folks want to experiment
with. This PR also improves the our debug log viewer to handle LSP
restarts.

TODO:
- [ ] Make sure NextLS binary loading works.

Release Notes:

- Added support for the experimental Next LS for Elxir, to enable it add
the following field to your settings to enable:

```json
"elixir": {
    "next": "on"
}
```
 2023-09-25 12:52:56 -05:00
KCaverly
86ec0b1d9f implement new search strategy 2023-09-25 13:44:19 -04:00
KCaverly
fbd6b5b434 cargo fmt 2023-09-22 09:46:06 -04:00
KCaverly
68c37ca2a4 move embedding provider to ai crate 2023-09-22 09:33:59 -04:00
Kyle Caverly
3c2b05be90
add semantic index status, for non authenticated users (#3005) 
Update project search semantic ui to accommodate for users who have not
set the OPENAI_API_KEY in their environment variables.

Release Notes:

- Expand Semantic Index status to include status for non authenticated
users
- Update Search UI to illustrate this status.
 2023-09-21 14:18:58 -04:00
KCaverly
7b63369df2 move api authentication to embedding provider 2023-09-21 14:00:00 -04:00
KCaverly
997f362cc2 add semantic index status, for non authenticated users 2023-09-21 13:40:01 -04:00
KCaverly
37a0c6615f remove conditional action registration for activate semantic search mode 2023-09-20 12:38:29 -04:00
Mikayla
0cceb3fdf1
Get nextLS running 2023-09-20 06:55:24 -07:00
Kyle Caverly
2f44055079
Semantic index eval (#2988) 
v0 of the Semantic Index evaluate test suite

Release Notes:

- Added eval.rs as an example to the semantic-index crates
- Generates test metrics for two small projects, as a starting point to
systematically evaluate retrieval quality
 2023-09-19 19:17:06 -04:00
KCaverly
11b3bfdc99 fix warnings 2023-09-19 19:05:26 -04:00
KCaverly
25cb79e475 remove git2 dependency for repository cloning in semantic_index eval 2023-09-19 18:55:15 -04:00
KCaverly
d85acceeec move git2 to workspace dependency globally 2023-09-19 16:13:47 -04:00
KCaverly
4f1a59ebf5 formatting 2023-09-19 12:27:33 -04:00
KCaverly
fc8dd8433c remove release channel flags in semantic_index 2023-09-19 12:20:59 -04:00
KCaverly
183758a7c5 fix Cargo.lock for merge 2023-09-19 11:44:51 -04:00
KCaverly
25bd357426 add recall and precision to semantic index 2023-09-18 18:25:02 -04:00
KCaverly
566bb9f71b add map to evaluation suite for semantic_index 2023-09-18 09:57:52 -04:00
KCaverly
1433160a08 enable include based filtering for search inside open and modified buffers 2023-09-15 15:16:20 -04:00
KCaverly
04bd107ada add ndcg@k to evaluate metrics 2023-09-15 10:36:21 -04:00
KCaverly
3a661c5977 catchup with main 2023-09-15 09:31:33 -04:00
Antonio Scandurra
ae85a520f2 Refactor semantic searching of modified buffers 2023-09-15 12:12:20 +02:00
KCaverly
796bdd3da7 update searching in modified buffers to accomodate for excluded paths 2023-09-14 19:42:06 -04:00
KCaverly
c19c8899fe add initial search inside modified buffers 2023-09-14 14:58:34 -04:00
Antonio Scandurra
f86e5a987f WIP 2023-09-14 17:42:30 +02:00
Antonio Scandurra
6a271617b4 Make path optional when parsing file 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-14 17:09:08 +02:00
KCaverly
137dda3ee6 wip eval framework for semantic index 2023-09-14 09:30:19 -04:00
KCaverly
0c1b2e5aa6 cleaned up warnings 2023-09-13 20:04:53 -04:00
KCaverly
eff44f9aa4 semantic index eval, indexing appropriately 2023-09-13 20:02:15 -04:00
KCaverly
6f29582fb0 progress on eval 2023-09-13 10:32:36 -04:00
KCaverly
d4fbe99052 add eval for gpt-engineer 2023-09-12 21:27:35 -04:00
KCaverly
0d14bbbf5b add eval values for tree-sitter 2023-09-12 20:36:06 -04:00
KCaverly
66c967da88 start work on eval script for semantic_index 2023-09-12 16:25:31 -04:00
KCaverly
e678c7d9ee swap 
SystemTime for Instant throughout rate_limit_expiry tracking
 2023-09-11 10:26:14 -04:00
KCaverly
7df21f86dd move cx notify observe for rate_limit_expiry into ProjectState in the semantic index 
Co-authored-by: Antonio <antonio@zed.dev>
 2023-09-11 10:11:40 -04:00
KCaverly
37915ec4f2 updated notify to accomodate for updated countdown 2023-09-08 16:53:16 -04:00
KCaverly
bf43f93197 updated semantic_index reset status to leverage target reset system time as opposed to duration 2023-09-08 15:04:50 -04:00
KCaverly
a5ee8fc805 initial outline for rate limiting status updates 2023-09-08 12:35:15 -04:00
KCaverly
cf5d1d91a4 update semantic search to go to no results if search query is blank 2023-09-07 14:43:41 -04:00
Antonio Scandurra
eda7e00645 Implement SemanticIndex::status and use it in project search 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-07 19:39:30 +02:00
Antonio Scandurra
65e17e212d Eagerly index project on workspace creation if it was indexed before 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-07 18:51:55 +02:00
Antonio Scandurra
a45c8c380f 💄 2023-09-07 15:25:23 +02:00
Antonio Scandurra
757a285852 Keep dropping the documents table if it exists 
This is because we renamed `documents` to `spans`.

Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-07 15:15:16 +02:00
Antonio Scandurra
93b889a93b Merge remote-tracking branch 'origin/main' into semantic-search-watch-worktrees 2023-09-07 15:07:46 +02:00
Antonio Scandurra
3ad1befb11 Remove unneeded logging 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-07 15:07:21 +02:00
KCaverly
265d02a583 update request timeout for open ai embeddings 2023-09-06 15:09:46 -04:00
KCaverly
17237f748c update token_count for OpenAIEmbeddings to accomodate for truncation 2023-09-06 15:09:15 -04:00
Antonio Scandurra
ce62173534 Rename Document to Span 2023-09-06 17:03:08 +02:00
Antonio Scandurra
de0f53b39f Ensure SemanticIndex::search waits for indexing to complete 2023-09-06 11:40:59 +02:00
Antonio Scandurra
c802680084 Clip ranges returned by SemanticIndex::search 
The files may have changed since the last time they were parsed, so the
ranges returned by `SemanticIndex::search` may be out of bounds.
 2023-09-06 09:41:51 +02:00
Antonio Scandurra
95b72a73ad Re-index project when a worktree is registered 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-05 17:17:58 +02:00
Antonio Scandurra
3c70b127bd Simplify SemanticIndex::index_project 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-05 16:54:48 +02:00
Antonio Scandurra
6b1dc63fc0 Retrieve embeddings based on pending files 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-05 16:16:12 +02:00
Antonio Scandurra
7b5a41dda2 Move retrieval of embeddings from the db into reindex_changed_files 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-09-05 16:09:24 +02:00
Antonio Scandurra
d4cff68475 🎨 2023-09-05 15:52:36 +02:00
KCaverly
8dbc0fe033 update pragma settings for improved database performance 2023-09-01 17:07:20 -04:00
KCaverly
54235f4fb1 updated embeddings background delay to 5 minutes 
Co-authored-by: Max <max@zed.dev>
 2023-09-01 13:04:09 -04:00
KCaverly
e86964eb5d optimize insert file in vector database 
Co-authored-by: Max <max@zed.dev>
 2023-09-01 13:01:37 -04:00
KCaverly
524533cfb2 flush embeddings queue when no files are parsed for 250 milliseconds 
Co-authored-by: Antonio <antonio@zed.dev>
 2023-09-01 11:24:08 -04:00
KCaverly
c4db914f0a move embeddings queue to use single hashmap for all changed paths 
Co-authored-by: Antonio <me@as-cii.com>
 2023-09-01 08:59:25 -04:00
KCaverly
afa59abbcd WIP: work towards wiring up a embeddings_for_digest hashmap that is stored for all indexed files 2023-08-31 16:42:39 -04:00
KCaverly
50cfb067e7 fill embeddings with database values and skip during embeddings queue 2023-08-31 13:19:17 -04:00
Antonio Scandurra
220533ff1a WIP 2023-08-31 18:00:57 +02:00
Antonio Scandurra
2503d54d19 Rename Sha1 to DocumentDigest 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-08-31 18:00:36 +02:00
Antonio Scandurra
3001a46f69 Reify Embedding/Sha1 structs that can be (de)serialized from SQL 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-08-31 17:55:43 +02:00
Antonio Scandurra
c763e728d1 Write to and read from the database in a transactional way 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-08-31 16:59:54 +02:00
Antonio Scandurra
35440be98e Abstract away how database transactions are executed 
Co-Authored-By: Kyle Caverly <kyle@zed.dev>
 2023-08-31 16:54:11 +02:00
KCaverly
7d4d6c871b fix bug for truncation ensuring no valid inputs are sent to openai 2023-08-30 17:42:16 -04:00
KCaverly
5abad58b0d moved semantic index to use embeddings queue to batch and managed for atomic database writes 
Co-authored-by: Max <max@zed.dev>
 2023-08-30 16:58:45 -04:00
KCaverly
76ce52df4e move queuing to embedding_queue functionality and update embedding provider to include trait items for max tokens per batch" 
Co-authored-by: Max <max@zed.dev>
 2023-08-30 16:01:28 -04:00
KCaverly
9781047156 move truncation to parsing step leveraging the EmbeddingProvider trait 2023-08-30 12:13:26 -04:00
KCaverly
76caea80f7 add should_truncate to embedding providers 2023-08-30 11:58:45 -04:00
KCaverly
e377ada1a9 added token count to documents during parsing 2023-08-30 11:05:46 -04:00
KCaverly
a7e6a65deb reindex files in the background after they have not been edited for 10 minutes 
Co-authored-by: Max <max@zed.dev>
 2023-08-29 17:14:44 -04:00
KCaverly
4f8b95cf0d add proper handling for open ai rate limit delays 2023-08-29 15:44:51 -04:00