mirror of
https://github.com/facebookarchive/prepack.git
synced 2024-08-17 18:41:06 +03:00
4319298f88
Summary: This PR adds the fuzzer I’ve been working on to the Prepack codebase so other people can contribute, provide feedback, and run it against their changes. The new commands added are: - `yarn fuzz`: Starts generating tests in the foreground and logs progress. If it finds an error it will try and shrink it before returning the shrunken program to you with the invalid results. - `yarn fuzz-sample`: See a selection of the programs generated by the fuzzer. - `yarn fuzz-overnight`: Spin up a worker for each CPU and try to find failing test cases. Your computer will be basically unusable while you run this, so leave it running overnight. Failed test cases will be saved in `fuzzer/overnight.sqlite` so in the morning you can use `sqlite3` to inspect the errors the fuzzer found. The fuzzer generates programs with an optimized function and executes them twice: 1. In Node.js 2. In Node.js after running Prepack Then compares the results. If the results are different then the fuzzer will attempt to shrink the program to something easier to debug and return this to you. See [this gist for a sample of generated programs](https://gist.github.com/calebmer/6a727c1f4aa8c08d51940e60d29d336a). Here’s an example of a function you might see: ```js function f3(a1, a2, a3, a4, a5, a6) { 2; var x2; if (0) { return a1 ? false : a2; } else { var x1 = a3; x2 = x1; } var x6; if (x2) { var x3; if (x2) { x3 = x2; } else { x3 = a4; } var x4 = x3; x6 = x4; } else { var x5; if (a5) { x5 = x2; } else { x5 = a6; } x6 = f2(x5); } return x6; } ``` So far I’ve reported four bugs reduced from test cases found by this version of the fuzzer. I’ve reported a couple more from old fuzzers I used, but these four from the current version. The shrinking process is not that good and it takes a while as the generated program can get large, so you’ll usually have to do some manual shrinking to get good bug reports. I only ran `yarn fuzz-overnight` for about an hour. It found 28 failures and I reduced those down to these 4. - #2354 - #2355 - #2361 - #2363 I expect I’ll find more bugs as these get fixed and I add more JavaScript features to the fuzzer. The features I currently have are: - Scalar primitives (number, string, boolean, null/undefined) - Functions - Conditional expressions - If statements - Variable declarations Not too many features, but enough to catch a handful of bugs. > **Note:** If this PR is too much to review, I’ve created [`calebmer/prepack-fuzzer`](https://github.com/calebmer/prepack-fuzzer) in which my work is broken up into commits as I made them. I then copied these files over to the Prepack repo. The fuzzer in this PR is a rewrite from the [fuzzer I started with](https://gist.github.com/calebmer/75dd75ebe556681d3a628e75eaffc403). The main lessons I learned from that one are that I should start with a general JS fuzzer instead of a React fuzzer (since adding JS features after the fact to fuzzer designed for React is difficult) and that all nested structures need to be generated with one recursive generator from the generator library I’m using. To generate programs I use [`leebyron/testcheck`](https://github.com/leebyron/testcheck-js) which is actually a JS compiled version of the official Clojure library [`clojure/test.check`](https://github.com/clojure/test.check). `testcheck` is designed for generative property testing at a much smaller scale then program fuzzing. So I am abusing the library a bit to use it as a fuzzer. My reasoning is that I wanted to write the fuzzer in JS (to have access to the Babel AST) and I didn’t want to write my own case generating framework. If we outgrow `testcheck` then we can keep the DSL, but rewrite the generation/shrinking logic. Although its been working fine for me so far. (Yet I am using a forked version which simply uses some unpublished features in the `testcheck` repo.) The generator code in `fuzzer/src/gen.js` may look odd to you. It uses immutable.js and a **state monad** implemented with a JS generator so the code looks imperative. I need state since generating various program components depends on things like “what variables are declared at a given point in time,” but because I’m limited to only using a single recursive generator (based on performance lessons I learned from my first fuzzer) I can’t pass around state at the generator level and must instead maintain state at the result level. At first I tried hacking together some imperative state, but when shrinking programs `testcheck` replays some generators to get new programs. So what do you do when you need a stateful process that needs to be replayed? You use a monad. I could try to fix the bugs I found, but I’d like to find more bugs. I need to add back support for React components and I need to add more language features. _Which JS language features are the most interesting to fuzz?_ I ranked all the kinds of AST nodes in our internal React bundle [and got this](https://gist.github.com/calebmer/be5e2bad4b12af683522096544fc9568). I’ll be starting with that, but the Prepack team has a better intuition around what’s good for fuzzing. I know there’s been a discussion around temporals recently that I haven’t really been following. What would be good ways to trigger this behavior? Pull Request resolved: https://github.com/facebook/prepack/pull/2374 Differential Revision: D9180836 Pulled By: calebmer fbshipit-source-id: 59d3fb59ecc1026a865672e2833f7482ed72139a |
||
|---|---|---|
| .. | ||
| src | ||
| .eslintrc | ||
| .gitignore | ||
| package.json | ||
| yarn.lock | ||