From 9e2213465bb67d915601b6be65698efed4f694fb Mon Sep 17 00:00:00 2001 From: ncoop57 Date: Sat, 9 Oct 2021 17:10:56 +0000 Subject: [PATCH] Add steps and requirements for running evaluation script --- evaluation/README.md | 21 +++++++++++++++++++++ evaluation/requirements.txt | 4 ++++ 2 files changed, 25 insertions(+) create mode 100644 evaluation/README.md create mode 100644 evaluation/requirements.txt diff --git a/evaluation/README.md b/evaluation/README.md new file mode 100644 index 0000000..e524be0 --- /dev/null +++ b/evaluation/README.md @@ -0,0 +1,21 @@ +# How to Evaluate + +## Human Eval + +The following steps are required to run the Human Eval step: +1. Ensure you are using python3.7 as required by [human-eval](https://github.com/openai/human-eval). We recommend conda: +``` +conda create -n human-eval python=3.7 +``` +2. Install the dependencies in this folder +``` +pip install -r requirements.txt +``` +3. Install human-eval by following the instructions on the [human-eval repo](https://github.com/openai/human-eval#usage) + + +With the following requirements performed you can now run the `evaluation.py` script: +``` +python evaluate.py --model_name_or_path=model_name_or_path --human_eval_path= --out_path=./model_results +``` +So for example if you want to evaluate the EleutherAI GPT Neo 125M \ No newline at end of file diff --git a/evaluation/requirements.txt b/evaluation/requirements.txt new file mode 100644 index 0000000..e4060e8 --- /dev/null +++ b/evaluation/requirements.txt @@ -0,0 +1,4 @@ +torch +fastcore +transformers +tqdm \ No newline at end of file