.devcontainer | ||
data_processing | ||
dependency_repos | ||
docs | ||
evaluation | ||
training | ||
.DS_Store | ||
.gitignore | ||
.gitmodules | ||
code_clippy_logo.jpg | ||
code-clippy-demo.gif | ||
LICENSE | ||
README.md | ||
reindent.py | ||
requirements_scripts.txt | ||
requirements.txt |
GPT-Code-Clippy (GPT-CC)
Courtesy of the awesome Aimee Trevett!
Introduction
GPT-Code-Clippy (GPT-CC) is an open source version of GitHub Copilot, a language model -- based on GPT-3, called GPT-Codex -- that is fine-tuned on publicly available code from GitHub.
Datasets
The dataset used to train GPT-CC is obtained from SEART GitHub Search using the following criteria:
- >10 GitHub stars
- >2 commits
- Must have a licence
- Exclude forks
- Size < 70708 bytes
These repositories are then combined with all of the GitHub repositories contain in The Pile.
The repositories are then filtered for duplicate files. Filtering is performed by regexing each file in each repository to obtain a list of "variables" (the tokens which only contain alphanumeric characters) and then filtering out any files which contain the same sequence of "variables. The deduplication script is available here.
The final dataset is available here. The dataset without the duplicates filtered out is also available here.
The datasheet discussing in more detail the construction, usage, and limitation of the dataset can be found here. We hope to get it officially into Huggingface's datasets library soon!
Models
The GPT-CC models are fine-tuned versions of GPT-2 and GPT-Neo.
The available models can be found here
The ones that perform relatively well (None improve on the standard GPT-Neo 125M model except for APPs specific models and only for the APPs task):
- https://huggingface.co/flax-community/gpt-code-clippy-125M-apps-alldata
- https://huggingface.co/flax-community/gpt-code-clippy-1.3B-apps-alldata-2
- https://huggingface.co/flax-community/gpt-code-clippy-1.3B-apps-alldata
- https://huggingface.co/flax-community/gpt-code-clippy-1.3B-apps
- https://huggingface.co/flax-community/gpt-neo-125M-code-clippy
- https://huggingface.co/flax-community/gpt-neo-125M-code-clippy-dedup-filtered-no-resize-2048bs
- https://huggingface.co/flax-community/gpt-neo-125M-code-clippy-dedup-2048
TODO: which is the recommended model?
Training
Training is done using the training scripts available here.
TODO: which is the recommended way to train GPT-CC?
Evaluation
The models are also evaluated on the APPS and HumanEval datasets.
Human Eval Results
Model | pass@1 | pass@2 | pass@5 | pass@10 |
---|---|---|---|---|
EleutherAI/gpt-neo | 0.12% | 0.24% | 0.61% | 1.22% |
gpt-neo-125M-apps | 0.06% | 0.12% | 0.30% | 0.61% |
dedup-filtered-no-resize-2048bs | 0.00% | 0.00% | 0.00% | 0.00% |
1024-filtered | 0.00% | 0.00% | 0.00% | 0.00% |
dedup-2048 | 0.00% | 0.00% | 0.00% | 0.00% |
APPS Eval Results
Coming soon...
Demo
A Visual Studio Code which uses the HuggingFace Inference API is available and can be found here.
We also have Huggingface's Space demo where you can specify and problem in the format of a programming competition question.
TODO: more information about this when complete.
Further Reading
For more information about GPT-CC, GitHub Copilot, etc, see:
TODO: add more further reading.
Acknowledgements
Special thanks to our contributors!!