firebase2graphql: improve readme structure, update git and add more tests (#666)

community/tools/firebase2graphql/README.md

@@ -8,6 +8,25 @@ A CLI tool to help you try realtime GraphQL on your firebase data. It takes data
 
 ![GIF](https://graphql-engine-cdn.hasura.io/assets/firebase2graphql/demo.gif)
 
+## Contents
+
+- [Getting started](#quick-start)
+- [Installation](#installation)
+- [Usage](#usage)
+- [Command](#command)
+- [Usage comparison](#usage-comparison---firebase-sdk-vs-graphql)
+- [Authentication](#authentication)
+- [Next steps](#next-steps)
+- [More information](#more-information)
+   - [Working](#working)
+   - [Normalization](#normalization)
+      - [Automatic](#automatic)
+      - [Manual](#manual)
+   - [Duplicates](#duplicates)
+   - [Overwrite](#overwrite)
+- [Feedback](#feedback)
+
+
 ## Quick start
 
 1. Quickly get the GraphQL Engine running by clicking this button:
@@ -177,8 +196,6 @@ Check out [next steps](#next-steps).
 
 ## Installation
 
-### CLI
-
 ```bash
 npm install -g firebase2graphql
 ```
@@ -215,22 +232,7 @@ firebase2graphql URL [flags]
 - `-v --version`: show CLI version
 - `-h, --help`: show CLI help
 
-## Next steps
-
-Once you have imported your data, it is recommended that you make it production ready.
-
-1. Normalize the data by [removing duplicates](#duplicates).
-2. Explore the GraphQL Engine Console to play with things such as
-   
-   - [Relationships](https://docs.hasura.io/1.0/graphql/manual/schema/relationships/index.html)
-   - [Permissions](https://docs.hasura.io/1.0/graphql/manual/auth/index.html)
-   - Using SQL
-   - [Set up async business logic using event triggers](https://docs.hasura.io/1.0/graphql/manual/event-triggers/index.html)
-   - [Create new tables](https://docs.hasura.io/1.0/graphql/manual/schema/basics.html)
-
-3. Set appropriate permissions. GraphQL Engine comes with [fine grained control layer](https://docs.hasura.io/1.0/graphql/manual/auth/index.html) that can be integrated with any standard Auth provider.
-
-## Usage Comparison - Firebase SDK vs GraphQL
+## Usage comparison - Firebase SDK vs GraphQL
 
 A typical query to do a single read from the database using [Firebase SDK](https://firebase.google.com/docs/reference/), (javascript) would look something like:
 
@@ -275,7 +277,26 @@ mutation {
 }
 ```
 
-## Things to know about implementation
+## Authentication
+
+Hasura can be integrated with most standard Authentication mechanisms including Firebase and Auth0. [Check out the authentication docs here](https://docs.hasura.io/1.0/graphql/manual/auth/index.html).
+
+## Next steps
+
+Once you have imported your data, it is recommended that you make it production ready.
+
+1. [Normalize the data](#normalization)
+2. Explore the GraphQL Engine Console to play with things such as
+
+   - [Relationships](https://docs.hasura.io/1.0/graphql/manual/schema/relationships/index.html)
+   - [Permissions](https://docs.hasura.io/1.0/graphql/manual/auth/index.html)
+   - Using SQL
+   - [Set up async business logic using event triggers](https://docs.hasura.io/1.0/graphql/manual/event-triggers/index.html)
+   - [Create new tables](https://docs.hasura.io/1.0/graphql/manual/schema/basics.html)
+
+3. Set appropriate permissions. GraphQL Engine comes with [fine grained control layer](https://docs.hasura.io/1.0/graphql/manual/auth/index.html) that can be integrated with any standard Auth provider.
+
+## More information
 
 ### Working
 
@@ -287,6 +308,8 @@ If you use the flag `--normalize`, the CLI finds out if the children tables are
 
 ### Normalization
 
+#### Automatic
+
 The CLI provides a flag called `--normalize` if you want to normalize your denormalized database.
 
 A lot of guess-work is done by the CLI while normalizing the database. Here are some thing you need to know:
@@ -295,6 +318,143 @@ A lot of guess-work is done by the CLI while normalizing the database. Here are
 2. Children tables are deleted if they are detected to be duplicates of some other root or child table.
 3. In case of some children tables, when the data lacks a unique identifier, an extra unique field is added. In most cases, this field gets deleted while mergine a duplicate table with the original table.
 
+#### Manual
+
+In some cases, due to inconsistent field names or due to insufficient data, the CLI might not be able to normalize your database; but it makes sure that you do not lose any data.
+
+In such cases, you can normalize the data yourself. Lets look at an example.
+
+Consider this firebase database. This is the database of the official Firebase example app:
+
+```json
+{
+  "posts" : {
+    "-LMbLFOAW2q6GO1bD-5g" : {
+      "author" : "Eena",
+      "authorPic" : "https://lh4.googleusercontent.com/-vPOIBOxCUpo/AAAAAAAAAAI/AAAAAAAAAFo/SKk9hpOB7v4/photo.jpg",
+      "body" : "My first post content\nAnd body\nANd structure",
+      "starCount" : 0,
+      "title" : "My first post",
+      "uid" : "4UPmbcaqZKT2NdAAqBahXj4tHYN2"
+    },
+    "-LMbLIv6VKHYul7p_PZ-" : {
+      "author" : "Eena",
+      "authorPic" : "https://lh4.googleusercontent.com/-vPOIBOxCUpo/AAAAAAAAAAI/AAAAAAAAAFo/SKk9hpOB7v4/photo.jpg",
+      "body" : "AKsdjak\naklsdjaskldjklas\nasdklfjaklsdfjklsda\nasdklfjasklf",
+      "starCount" : 0,
+      "title" : "Whatta proaaa",
+      "uid" : "4UPmbcaqZKT2NdAAqBahXj4tHYN2"
+    }
+  },
+  "user-posts" : {
+    "4UPmbcaqZKT2NdAAqBahXj4tHYN2" : {
+      "-LMbLFOAW2q6GO1bD-5g" : {
+        "author" : "Eena",
+        "authorPic" : "https://lh4.googleusercontent.com/-vPOIBOxCUpo/AAAAAAAAAAI/AAAAAAAAAFo/SKk9hpOB7v4/photo.jpg",
+        "body" : "My first post content\nAnd body\nANd structure",
+        "starCount" : 0,
+        "title" : "My first post",
+        "uid" : "4UPmbcaqZKT2NdAAqBahXj4tHYN2"
+      },
+      "-LMbLIv6VKHYul7p_PZ-" : {
+        "author" : "Eena",
+        "authorPic" : "https://lh4.googleusercontent.com/-vPOIBOxCUpo/AAAAAAAAAAI/AAAAAAAAAFo/SKk9hpOB7v4/photo.jpg",
+        "body" : "AKsdjak\naklsdjaskldjklas\nasdklfjaklsdfjklsda\nasdklfjasklf",
+        "starCount" : 0,
+        "title" : "Whatta proaaa",
+        "uid" : "4UPmbcaqZKT2NdAAqBahXj4tHYN2"
+      }
+    }
+  },
+  "users" : {
+    "4UPmbcaqZKT2NdAAqBahXj4tHYN2" : {
+      "email" : "rishichandrawawhal@gmail.com",
+      "profile_picture" : "https://lh4.googleusercontent.com/-vPOIBOxCUpo/AAAAAAAAAAI/AAAAAAAAAFo/SKk9hpOB7v4/photo.jpg",
+      "username" : "Eena"
+    }
+  }
+}
+```
+
+In case of this JSON, the CLI will generate the following tables:
+
+```sql
+users (
+  _id text not null primary key,
+  email text,
+  profile_picture text,
+  username text
+)
+
+posts (
+  _id text not null primary key,
+  title text,
+  body text,
+  starCount int,
+  author text,
+  authorPic text,
+  uid text
+)
+
+user_posts (
+  _id text not null,
+  _id_2 text not null,
+  title text,
+  body text,
+  starCount int,
+  author text,
+  authorPic text,
+  uid text
+)
+```
+
+As we can see, this is not the most efficient of schemas.
+
+- `posts(uid)` can be a foreign key referencing `users(_id)`. `posts(author)` and `posts(authorPic)` can be deleted if `users` and `posts` are related via a foreign key.
+- `user_posts` table is obsolete if `posts` and `users` tables are deleted.
+
+To normalize it, here are the steps you must follow:
+
+1. Create the following foreign key constraints:
+
+    ```sql
+    ALTER TABLE "posts" ADD CONSTRAINT "posts_users__uid" FOREIGN KEY ("uid") REFERENCES "users"("_id");
+
+    ALTER TABLE "posts" DROP COLUMN "author", DROP COLUMN "authorPic";
+
+    DROP TABLE "user_posts";
+    ```
+
+   To create them, go to the Hasura Console and run the SQL in the `Data > SQL` section.
+
+2. Create the relationships. In the console, go the desired table and add the relationship suggested based on the Foreign key. For example for the `posts` table in the above schema, the suggested relationship would be suggested like:
+
+    ![suggested](assets/suggested-rel.png)
+
+   Click on `Add`. You can name it whatever you like. Lets call it `author` in this case.
+
+    ![added](assets/added-rel.png)
+
+
+   Similarly, add the suggested relationships for other tables as well.
+
+
+3. Once you have created relationships, you can start making fancy GraphQL queries like:
+
+    ```graphql
+    query {
+      users (order_by: username_asc){
+        username
+        profile_picture
+        email
+        posts (where: { starCount: { _gte: 100}}){
+          title
+          body
+          starCount
+        }
+      }
+    }
+    ```
 
 ### Duplicates
 
@@ -304,12 +464,12 @@ In such cases, you have three choices:
 
 1. Use the API as such if you prefer the exact API.
 2. Go to the UI Console and delete the duplicates and normalize the database as you feel fit.
-3. Use the `--normalize` flag and rerun the migration. In this case, the CLI will detect duplicates and make appropriate relationships between root nodes. (This feature is experimental and needs more test cases to attain stability. Contributions are welcome) 
- 
+3. Use the `--normalize` flag and rerun the migration. In this case, the CLI will detect duplicates and make appropriate relationships between root nodes. (This feature is experimental and needs more test cases to attain stability. Contributions are welcome)
+
 
 ### Overwrite
 
-If your database already contains tables with the same name as the root fields of your JSON database, the command will fail. If you want to overwrite the database anyway, you should provide an additional flag "--overwrite".
+If your database already contains tables with the same name as the root fields of your JSON database, the command will fail. If you want to overwrite the database anyway, you should provide an additional flag `--overwrite`.
 
 ## Feedback
 

community/tools/firebase2graphql/test/data-sets/fbUsers.js

@@ -1,10 +1,10 @@
 module.exports = {
-  scores: {
+  f2g_test_scores: {
     Rishi: 24,
     Rikin: 26,
     Tanmai: 27,
   },
-  author: {
+  f2g_test_author: {
     someone: {
       one: {
         name: 'Rishi',
@@ -83,7 +83,7 @@ module.exports = {
       },
     },
   },
-  articles: {
+  f2g_test_articles: {
     first: {
       title: 'Rishis article',
       body: "Rishi's article's body",

community/tools/firebase2graphql/test/data-sets/readme-example-1.json

@@ -1,5 +1,5 @@
 {
-  "Articles": {
+  "f2g_Articles": {
     "A1": {
       "Title": "Title1",
       "Body": "Body1",
@@ -55,7 +55,7 @@
       }
     }
   },
-  "Authors": {
+  "f2g_test_Authors": {
     "AT1": {
       "Name": "AName1",
       "Age": 11,
@@ -77,7 +77,7 @@
       }
     }
   },
-  "Comments": {
+  "f2g_test_Comments": {
     "C1": {
       "Body": "Comment1",
       "Author": {

community/tools/firebase2graphql/test/test.sh

@@ -5,4 +5,5 @@ else
   F2G_LOG=0 ../bin/run $TEST_HGE_URL --access-key=$TEST_X_HASURA_ACCESS_KEY --db=./data-sets/chinook.json --overwrite --normalize && node verifyChinook.js
   F2G_LOG=0 ../bin/run $TEST_HGE_URL --access-key=$TEST_X_HASURA_ACCESS_KEY --db=./data-sets/blog.json --overwrite --normalize && node verifyBlog.js
   F2G_LOG=0 ../bin/run $TEST_HGE_URL --access-key=$TEST_X_HASURA_ACCESS_KEY --db=./data-sets/chinook_nested.json --overwrite --normalize && node verifyChinookNested.js
+  F2G_LOG=0 ../bin/run $TEST_HGE_URL --access-key=$TEST_X_HASURA_ACCESS_KEY --db=./data-sets/readme-example-1.json --overwrite --normalize && node verifyRE1.js
 fi

community/tools/firebase2graphql/test/verifyChinookNested.js

@@ -56,17 +56,17 @@ const verifyDataImport = () => {
           }),
         }
       ).then(() => {
-        console.log(colors.green('✔︎ data-sets/chinook.json: Test passed'));
+        console.log(colors.green('✔︎ data-sets/chinook_nested.json: Test passed'));
         process.exit();
       }).catch(() => {
         process.exit();
       });
     } else {
-      console.log(colors.red('✖ data-sets/chinook.json: Test failed. Unexpected response.'));
+      console.log(colors.red('✖ data-sets/chinook_nested.json: Test failed. Unexpected response.'));
       process.exit();
     }
   }).catch(e => {
-    console.log(colors.red('✖ data-sets/chinook.json: Test failed. Unexpected response.'));
+    console.log(colors.red('✖ data-sets/chinook_nested.json: Test failed. Unexpected response.'));
     console.log(JSON.stringify(e, null, 2));
 
     process.exit();

community/tools/firebase2graphql/test/verifyRE1.js

@@ -0,0 +1,67 @@
+const {query} = require('graphqurl');
+const fetch = require('node-fetch');
+const colors = require('colors/safe');
+
+const complexQuery = `
+query {
+  f2g_test_Authors (order_by: Name_asc) {
+    _id
+    Name
+    f2g_Articles (order_by: Title_asc, where: { IsUnpublished: { _eq: true}}) {
+      Title
+      f2g_test_Comments (order_by: Date_asc) {
+        Body
+        Date
+      }
+    }
+  }
+}
+`;
+
+const verifyDataImport = () => {
+  query({
+    query: complexQuery,
+    endpoint: `${process.env.TEST_HGE_URL}/v1alpha1/graphql`,
+    headers: {'x-hasura-access-key': process.env.TEST_X_HASURA_ACCESS_KEY},
+  }).then(response => {
+    if (
+      response.data &&
+      response.data.f2g_test_Authors[0].f2g_Articles.length === 0 &&
+      response.data.f2g_test_Authors[1].f2g_Articles[0].f2g_test_Comments[0].Body === 'Comment1'
+    ) {
+      let sqlString = '';
+      ['Articles', 'Authors', 'Comments'].forEach(t => {
+        sqlString += `drop table public."f2g_test_${t}" cascade;`;
+      });
+      fetch(
+        `${process.env.TEST_HGE_URL}/v1/query`,
+        {
+          method: 'POST',
+          headers: {'x-hasura-access-key': process.env.TEST_X_HASURA_ACCESS_KEY},
+          body: JSON.stringify({
+            type: 'run_sql',
+            args: {
+              sql: sqlString,
+              cascade: true,
+            },
+          }),
+        }
+      ).then(() => {
+        console.log(colors.green('✔︎ data-sets/readme-example-1.json: Test passed'));
+        process.exit();
+      }).catch(() => {
+        process.exit();
+      });
+    } else {
+      console.log(colors.red('✖ data-sets/readme-example-1.json: Test failed. Unexpected response.'));
+      process.exit();
+    }
+  }).catch(e => {
+    console.log(colors.red('✖ data-sets/readme-example-1.json: Test failed. Unexpected response.'));
+    console.log(JSON.stringify(e, null, 2));
+
+    process.exit();
+  });
+};
+
+verifyDataImport();