Document the protocol

HowToWriteAClient.md

@@ -0,0 +1,145 @@
+This is a guide to writing a client for TabNine.
+
+# Introduction
+
+TabNine is invoked by the text editor plugin (the "client") as a subprocess.
+
+The client communicates with TabNine through standard input and output. TabNine does not write to standard error.
+
+Each request to TabNine is a JSON object followed by a newline, encoded as UTF-8. There must be no newlines in the JSON object.
+TabNine will produce exactly one response for each request. A response consists of a JSON object followed by a newline.
+Each line of input to TabNine corresponds to exactly one line of output. If a line of input is malformed, the corresponding output will be the JSON object `null`.
+
+It's helpful when debugging to view TabNine's log output. You can enable logging by passing `--log-file-path` as an argument to the TabNine binary.
+For example, this lets you see error messages explaining why the request is malformed.
+
+# Getting Started
+
+Run `dl_binaries.sh` (in this repository) to download the most recent version of TabNine.
+Find the TabNine binary in `binaries/<version>/<platform>`.
+Run TabNine in your terminal and paste the following command as input:
+
+```
+{"version": "1.0.0", "request": {"Autocomplete": {"before": "Hello H", "after": "", "region_includes_beginning": true, "region_includes_end": true, "filename": null}}}
+```
+You should see the following output:
+```
+{"old_prefix":"H","results":[{"new_prefix":"Hello","old_suffix":"","new_suffix":""}],"user_message":[]}
+```
+A few things to note:
+- The protocol is versioned. The protocol versions are the same as TabNine versions. To guarantee forward compatibility with future versions of TabNine, pass the current TabNine version (or any previous version) as the protocol version.
+- The completion position is specified by giving the strings before and after the cursor. If these strings are very long, you can truncate them. In this case you should set `region_includes_beginning` or `region_includes_end` to `false` to indicate that the strings do not extend to the beginning or end of the file, respectively.
+- The recommended threshold for truncation is 100 KB.
+- Autocomplete responses contain a field `user_message` which is a message that should be displayed to the user. For example, this is used to inform the user when a language server fails to start, or when TabNine hits the index size limit.
+
+# Setting up TabNine within an editor plugin
+
+You must preserve the directory structure created by `dl_binaries.sh`, or else TabNine's automatic updating will not work.
+
+When TabNine updates, it downloads the new version in the same location as the current binary but with a different version directory. For example, if the current binary is at `bin/1.0.5/x86_64-apple-darwin/TabNine`, and TabNine downloads version `1.0.7`, it will be installed at `bin/1.0.7/x86_64-apple-darwin/TabNine`.
+
+Once TabNine downloads an update, it terminates. You should restart TabNine when it terminates, up to some maximum number of restarts (say, 10).
+
+To start TabNine, list the `binaries` directory and choose the most recent version. Here is Python code from the Sublime Text client which does this:
+```python
+def parse_semver(s):
+    try:
+        return [int(x) for x in s.split('.')]
+    except ValueError:
+        return []
+
+def get_tabnine_path(binary_dir):
+    translation = {
+        ("linux",   "x32"): "i686-unknown-linux-gnu/TabNine",
+        ("linux",   "x64"): "x86_64-unknown-linux-gnu/TabNine",
+        ("osx",     "x32"): "i686-apple-darwin/TabNine",
+        ("osx",     "x64"): "x86_64-apple-darwin/TabNine",
+        ("windows", "x32"): "i686-pc-windows-gnu/TabNine.exe",
+        ("windows", "x64"): "x86_64-pc-windows-gnu/TabNine.exe",
+    }
+    versions = os.listdir(binary_dir)
+    versions.sort(key=parse_semver, reverse=True)
+    for version in versions:
+        key = sublime.platform(), sublime.arch()
+        path = os.path.join(binary_dir, version, translation[key])
+        if os.path.isfile(path):
+            add_execute_permission(path)
+            print("TabNine: starting version", version)
+            return path
+```
+
+# API Specification
+
+Each request to TabNine must be a JSON object followed by a newline. The JSON object must be a dictionary containing the fields `version` and `request`. `version` should be a string corresponding to a TabNine version. The field `request` must be a dictionary with a single key. The key must be one of the following:
+- `Autocomplete`
+- `Prefetch`
+- `GetIdentifierRegex`
+
+The value associated with the key must be of the corresponding type. For example, if the key is `Autocomplete`, the value must have type `AutocompleteArgs`.
+
+TabNine's response will be of the corresponding type. For example, if the key was `Autocomplete`, the response will be of type `AutocompleteResponse`.
+
+# API Types
+
+`null` fields can be omitted in requests.
+
+```
+AutocompleteArgs {
+  before: string,
+  after: string,
+  filename: string | null,
+  region_includes_beginning: bool,
+  region_includes_end: bool,
+  max_num_results: int | null
+}
+```
+
+`max_num_results` must be positive. More information about Autocomplete requests is in the "Getting Started" section.
+
+```
+PrefetchArgs {
+  filename: string
+}
+```
+You can use this API call to make TabNine add a file to its index even if the user hasn't requested completions in the file yet.
+
+```
+GetIdentifierRegexArgs {
+  filename: string | null
+}
+```
+This gives the regex used by TabNine to parse identifiers for the provided file.
+
+```
+AutocompleteResponse {
+  old_prefix: string,
+  results: ResultEntry[],
+  user_message: string[]
+}
+```
+
+```
+ResultEntry {
+  new_prefix: string,
+  old_suffix: string,
+  new_suffix: string,
+
+  kind: CompletionItemKind | null,
+  detail: string | null,
+  documentation: Documentation | null,
+  deprecated: bool | null
+}
+```
+
+`CompletionItemKind` and `Documentation` are specified by the [Language Server Protocol](https://microsoft.github.io/language-server-protocol/specification).
+The Language Server Protocol also specifies the meanings of `kind`, `detail`, `documentation`, and `deprecated`. Each of these fields will be omitted from the response if they are `null`.
+
+The behavior of autocompletion is as follows: when the user selects the result, the text before the cursor should be `old_prefix`, and it should be replaced by `new_prefix`. The text after the cursor should be `old_suffix`, and it should be replaced by `new_suffix`.
+
+```
+PrefetchResponse = null
+```
+
+```
+GetIdentifierRegexResponse = string
+```