Summary:
* Fix referentialization of optimized functions
* Fix serialization issues for nested optimized functions
* Model aliasing effects for array loop operators
* Fix dependency issues with cyclic prototype dependencies
* Fix problem where a bind call can use a variable before its definition
* Speed up visitor by skipping redundant generator scopes
Reviewed By: trueadm
Differential Revision: D10191866
fbshipit-source-id: 669d6c7a49f10f3222c3dd56537e0f76a4e9d2c6
Summary: Now that babel 7 is stable we can use caret as we use from other tools (like Metro). `yarn.lock will make sure it does not get updated until we want
Reviewed By: pvdz
Differential Revision: D9861594
fbshipit-source-id: 9d27d9c91f2b4e28d8a3c914e9b578eb9f0dc90e
Summary:
- Fixed bugs that could cause generated code to throw
- Improved evaluation of abstract conditionals
- Added CLI options --warnaserror, --diagnosticaserror, --nodiagnostic
- Added full support for React.Children.map mock
- Support constructors that return an object but also throw
Reviewed By: yinghuitan
Differential Revision: D9660870
fbshipit-source-id: 49674fae56538809f6f962e49291be77f3617d2c
Summary:
Release notes: `test-internal-react` and `test-internal` now use `--max_old_space_size=32768`
As per the title, this increases the max memory allocation for Node to 32gb. `--max_old_space_size=32768`.
Pull Request resolved: https://github.com/facebook/prepack/pull/2524
Differential Revision: D9618037
Pulled By: trueadm
fbshipit-source-id: 6bd595410c6f93ccc5abbc116ee892e8de519eed
Summary:
Highlights:
Make simplification more robust when complexity limits are reached
reduces loop iteration counter to 6 before bailing out
We weren’t using the correct `initializerAstNodeName` in two places
Tease apart expressions vs. statements for operation descriptors
Simplify equality expressions where types are known
Refactor AbstractObjectValue to use evaluateWithAbstractConditional
Allow invalid render return values in the React reconciler
update Babel 7 to stable release
Speed up simplifier by using an implication cache per path branch
Add basic support for throws in React
Reviewed By: hermanventer
Differential Revision: D9597756
fbshipit-source-id: 6009ddb70f8b68151bf26ac8c15320213f67dcfc
Summary:
Fixes#2432. I found this while trying to compile our internal React Native bundle.
Prepack was incorrectly cleaning up lexical scopes in the case of a stack overflow.
This bug is a bit difficult to reproduce. Follow these steps to reproduce:
1. Produce a stack overflow in code Prepack compiles. With a recursive loop for instance.
2. Wrap in `__evaluatePureFunction`. This should catch the stack overflow and emit a residual function call.
3. Add empty functions for `global.require` and `global.__d`. You need to do this so that we don’t crash [in tracer code trying to resolve these variables](c18bd7ee97/src/methods/function.js (L82-L90)) but we instead crash [in `PrepareForOrdinaryCall`](c18bd7ee97/src/methods/function.js (L100)).
I used a `try`/`catch` to fix this which is the obvious solution, but perhaps not the most efficient. Happy to write a slightly uglier solution for performance if we feel like it is necessary.
Pull Request resolved: https://github.com/facebook/prepack/pull/2433
Differential Revision: D9469599
Pulled By: calebmer
fbshipit-source-id: bd120c2e4bd450da3364c722550977d6566ed89a
Summary:
Release notes: None
Remove dead _callRequireAndAccelerate function
Remove dead isModuleInitialized function
Remove dead --accelerateUnsupportedRequires option
Deleting dead code around checking uniqueness of previousDependencies
Add global.__eagerlyRequireModuleDependencies helper to effectively disable lazy module initialization in a code block (needed for InstantRender, to avoid having to deal with conditionally defined modules when prepacking nested callbacks)
Pull Request resolved: https://github.com/facebook/prepack/pull/2482
Differential Revision: D9492955
Pulled By: NTillmann
fbshipit-source-id: 91fe30168ffa848502982e042772462530b982a5
Summary: New alpha version for master.
Reviewed By: trueadm
Differential Revision: D9457284
fbshipit-source-id: 6b67e1095dc9bcd9545e88dcfa9a7d714434a9db
Summary:
What it says on the tin
Pull Request resolved: https://github.com/facebook/prepack/pull/2463
Differential Revision: D9456926
Pulled By: gaearon
fbshipit-source-id: fef96d4e0843662ff90eb4b0b06410f3b3bac83c
Summary:
Fixes#2419#2386#2439#2447#2432#2437#2442
* Fix havoced binding not in optimized function
* Allow arrays with widened numeric properties to update index properties
* Fix nested for statement bailout with nested for-in
* Don’t record modified bindings for immutable bindings when havocing
Reviewed By: trueadm, sebmarkbage
Differential Revision: D9456957
fbshipit-source-id: f266b8cc73012b9c721f0f9eebd48347bf0e37ae
Summary:
Fixes#2151#2222#2279#2393#2399#2404#2411#2414#2415
Added a fuzz testing tool
Added test cases
Turn crash in JSON.stringify into a diagnostic
Adds a `arrayNestedOptimizedFunctionsEnabled` flag to enable nested optimized functions derived from Array.prototype methods (like `map`) and Array.from
Refactor assignment on partial or possibly deleted property
Rewrote the joining logic to always do a full join at every join point
Removed last remnants of delayUnsupportedRequires
Reviewed By: cblappert
Differential Revision: D9349841
fbshipit-source-id: 74a16dbb015777d59d23fdfde77abbe2489c292a
Summary:
The previous changes on "Do full joins" didn't remove all
uses of the delayUnsupportedRequires feature. This does it.
In particular, this fixes the internal prepack-test.
Reviewed By: hermanventer
Differential Revision: D9287509
fbshipit-source-id: f810f5e08a10cbd14272d776799a2a8f1442d1c8
Summary:
Release note: Rewrote the joining logic to always do a full join at every join point
Closes#2151#2222#2279
I've spent a lot of time in the last few months trying to sort out problems that arise from effects being applied too many or too few times. Fixing these feel a bit like playing wack a mole and in the end no fix goes unpunished.
Stepping back a bit from the fray, it seems to me that the root cause of all this pain is the fact that joins of different kinds of completions get delayed.
Before we had path conditions and the simplifier this seemed like a rather good thing since exceptional paths did not contribute values to the normal paths and we thus had fewer abstract values to deal with and fewer places where Prepack would grind to a halt.
In the current state of things, however, it seems perfectly possible to join in all branches at every join point. I've had to decrease some limits, in particular the number of times we go around a loop with conditional exits. I've also had to make the test runner impose a limit on how many times the simplifier can invoke Path.implies.
Nevertheless, the tests seem to pass and hopefully this will also fix quite a lot of bugs that have been unresolved for many months already.
Pull Request resolved: https://github.com/facebook/prepack/pull/2402
Differential Revision: D9236263
Pulled By: hermanventer
fbshipit-source-id: 92a25b591591297afeba536429226c5a0291f451
Summary:
This PR adds the fuzzer I’ve been working on to the Prepack codebase so other people can contribute, provide feedback, and run it against their changes. The new commands added are:
- `yarn fuzz`: Starts generating tests in the foreground and logs progress. If it finds an error it will try and shrink it before returning the shrunken program to you with the invalid results.
- `yarn fuzz-sample`: See a selection of the programs generated by the fuzzer.
- `yarn fuzz-overnight`: Spin up a worker for each CPU and try to find failing test cases. Your computer will be basically unusable while you run this, so leave it running overnight. Failed test cases will be saved in `fuzzer/overnight.sqlite` so in the morning you can use `sqlite3` to inspect the errors the fuzzer found.
The fuzzer generates programs with an optimized function and executes them twice:
1. In Node.js
2. In Node.js after running Prepack
Then compares the results. If the results are different then the fuzzer will attempt to shrink the program to something easier to debug and return this to you. See [this gist for a sample of generated programs](https://gist.github.com/calebmer/6a727c1f4aa8c08d51940e60d29d336a). Here’s an example of a function you might see:
```js
function f3(a1, a2, a3, a4, a5, a6) {
2;
var x2;
if (0) {
return a1 ? false : a2;
} else {
var x1 = a3;
x2 = x1;
}
var x6;
if (x2) {
var x3;
if (x2) {
x3 = x2;
} else {
x3 = a4;
}
var x4 = x3;
x6 = x4;
} else {
var x5;
if (a5) {
x5 = x2;
} else {
x5 = a6;
}
x6 = f2(x5);
}
return x6;
}
```
So far I’ve reported four bugs reduced from test cases found by this version of the fuzzer. I’ve reported a couple more from old fuzzers I used, but these four from the current version. The shrinking process is not that good and it takes a while as the generated program can get large, so you’ll usually have to do some manual shrinking to get good bug reports. I only ran `yarn fuzz-overnight` for about an hour. It found 28 failures and I reduced those down to these 4.
- #2354
- #2355
- #2361
- #2363
I expect I’ll find more bugs as these get fixed and I add more JavaScript features to the fuzzer. The features I currently have are:
- Scalar primitives (number, string, boolean, null/undefined)
- Functions
- Conditional expressions
- If statements
- Variable declarations
Not too many features, but enough to catch a handful of bugs.
> **Note:** If this PR is too much to review, I’ve created [`calebmer/prepack-fuzzer`](https://github.com/calebmer/prepack-fuzzer) in which my work is broken up into commits as I made them. I then copied these files over to the Prepack repo.
The fuzzer in this PR is a rewrite from the [fuzzer I started with](https://gist.github.com/calebmer/75dd75ebe556681d3a628e75eaffc403). The main lessons I learned from that one are that I should start with a general JS fuzzer instead of a React fuzzer (since adding JS features after the fact to fuzzer designed for React is difficult) and that all nested structures need to be generated with one recursive generator from the generator library I’m using.
To generate programs I use [`leebyron/testcheck`](https://github.com/leebyron/testcheck-js) which is actually a JS compiled version of the official Clojure library [`clojure/test.check`](https://github.com/clojure/test.check). `testcheck` is designed for generative property testing at a much smaller scale then program fuzzing. So I am abusing the library a bit to use it as a fuzzer. My reasoning is that I wanted to write the fuzzer in JS (to have access to the Babel AST) and I didn’t want to write my own case generating framework. If we outgrow `testcheck` then we can keep the DSL, but rewrite the generation/shrinking logic. Although its been working fine for me so far. (Yet I am using a forked version which simply uses some unpublished features in the `testcheck` repo.)
The generator code in `fuzzer/src/gen.js` may look odd to you. It uses immutable.js and a **state monad** implemented with a JS generator so the code looks imperative. I need state since generating various program components depends on things like “what variables are declared at a given point in time,” but because I’m limited to only using a single recursive generator (based on performance lessons I learned from my first fuzzer) I can’t pass around state at the generator level and must instead maintain state at the result level. At first I tried hacking together some imperative state, but when shrinking programs `testcheck` replays some generators to get new programs. So what do you do when you need a stateful process that needs to be replayed? You use a monad.
I could try to fix the bugs I found, but I’d like to find more bugs. I need to add back support for React components and I need to add more language features.
_Which JS language features are the most interesting to fuzz?_
I ranked all the kinds of AST nodes in our internal React bundle [and got this](https://gist.github.com/calebmer/be5e2bad4b12af683522096544fc9568). I’ll be starting with that, but the Prepack team has a better intuition around what’s good for fuzzing. I know there’s been a discussion around temporals recently that I haven’t really been following. What would be good ways to trigger this behavior?
Pull Request resolved: https://github.com/facebook/prepack/pull/2374
Differential Revision: D9180836
Pulled By: calebmer
fbshipit-source-id: 59d3fb59ecc1026a865672e2833f7482ed72139a
Summary:
Weekly release v0.2.47:
- Many bug fixes
- Progress in support for nested optimized functions
- Fixing source map support
- Improved support for abstract behavior in switch statements
- Reduced memory usage of Prepack
Reviewed By: cblappert
Differential Revision: D9230689
fbshipit-source-id: 2094183fe4183089dfa584b865c9997147475e45
Summary:
- Bugfixes, refactors
- extended use of simplification to prepack more code
- improved shape modeling
- Added ability to materialize objects without havocing
- new `--reproUnconditionally` and `--reproOnFatal` options in Prepack
- creates zip file with all files needed to debug the failure
- creates `repro.sh` script that can open Nuclide debugger, pre-populating the original arguments causing the failure
Reviewed By: NTillmann
Differential Revision: D9101750
fbshipit-source-id: c7a2a7fc1c7814f02937f786df01ec3cb8effe4e
Summary:
Release Notes: None
The original issue here was that `nested` is defined inside of `fn2` which is a non-optimized function called by `fn` (an optimized function). That caused Prepack to not detect that `nested` was nested in `fn2`.
The fix is to use `CreatedObjects` to test for nesting instead of the environment lookup. The environment lookup fails because `nested` is evaluated with `fn2`'s effects applied but _not in `fn2`'s environment_.
This PR also adds a command I use frequently to test a single failing `test-runner` test as well as a way to skip lint because some tests can't pass lint.
Addresses the first test case of #2337.
Pull Request resolved: https://github.com/facebook/prepack/pull/2339
Differential Revision: D9074916
Pulled By: cblappert
fbshipit-source-id: 720003b965d9a9a6842d512ea41cd6402361342e
Summary:
Release Notes:
- Created DebugReproManager to capture all sourcefiles touched by Prepack (to include minimal subset of useful sourcefiles in the debug package)
- `--repro` splits into two: `--reproUnconditionally` which will create a debug package _regardless_ of Prepack's success/failure, and `--reproOnFatal`, which will _only_ create a debug package if Prepack outputs a `FatalError`.
- The debug package now includes all relevant sourcefiles (except for node modules), a copy of the version of Prepack (lib) that was used when the package was created, and a script to `yarn install` the relevant modules for the included version of Prepack, then start the Nuclide Prepack debugger with the proper parameters for files in the debug package (including original prepack arguments). This is in addition to the original input files.
- The impact of having the `DebugReproManager` on in `--reproOnFatal` mode all the time is negligible, as show in the table below. This flag will be always be enabled on Sandcastle builds so that failures are more easily debugged.
- The time difference between no repro flag and `--reproOnFatal` seems like it can be written off as simple variance between runs. The larger increase when actually creating the zip comes from reading and zipping the files, which takes time proportional how many files are touched.
- SourceMapManager was refactored to not use `Invariant` or `SourceFile`s. This is so that `DebugReproManager` import it without increasing the flow cycle, and allows the `DebugReproManager` to be passed from Prepack to the CLI to create the repro package.
- The repro option introduces a potential for a subtle race condition that is addressed as follows:
- The last `if (!success && reproMode === "none") process.exit(1);` must check reproMode because `generateDebugRepro` involves an async process (directory zipping). If there is an ongoing repro and the process exits, the repro may terminate prematurely, causing no repro to be generated. Instead, this only triggers if there is no repro -- if there is, the `generateDebugRepro` function will handle process exiting if it needs to.
Usage:
```node [prepack] [files to prepack] --reproOnFatal /Absolute/path/to/repro/bundle.zip --debugBuckRoot /buck/root```
or
```node [prepack] [files to prepack] --reproUnconditionally /Absolute/path/to/repro/bundle.zip --debugBuckRoot /buck/root```
Demo: https://www.dropbox.com/s/p62ves2p55fyyl7/--repro%20annotated%20demo.mp4?dl=0
Pull Request resolved: https://github.com/facebook/prepack/pull/2289
Differential Revision: D9002841
Pulled By: caiismyname
fbshipit-source-id: 623b362f963095f1cd8163684fd6e76596e7c4fc
Summary:
* Enhanced dead code elimination for optimized functions
* Much of the buildNode and inline Babel logic has been moved to a dedicated ResidualOperationSerializer class
* Provide a way to temporarily disable effects tracking
* Simplified forked completion constructors
* React components can have their props modelled via `__optimizeReactComponentTree`
Reviewed By: simonhj
Differential Revision: D9004189
fbshipit-source-id: 80936cbc66ad9dc350bbf0dcf0f1ff228a147f43
Summary: Bump version of alpha version of Prepack.
Reviewed By: gaearon
Differential Revision: D8892572
fbshipit-source-id: 129a265b5c8a725d4a09daa6811c0cc1a7ba2249
Summary:
- Support modeling of shapes of optimized function arguments
- fix test262 to fail CircleCI test if not enough tests pass
- upgrades Prepack to use Babel 7.0.0-beta.53
- Nuclide compatibility fix
Reviewed By: gaearon
Differential Revision: D8892151
fbshipit-source-id: 8f1397d25d26e822709ed53c8e6d37055fe396ec
Summary:
`flow check` runs a complete Flow check.
This is good for CI, but it means every re-check after edit is as slow as first check.
I'm changing local `yarn flow` to use a Flow server instead. This should speed up rechecks locally somewhat. Still won't save us from large recheck cycles, but at least we're not starting from a clean slate every time.
I'm adding `yarn flow-ci` with old behavior for the CI.
Finally, I replaced `--merge-timeout` with an equivalent recently added config option. This ensures it's respected both for the server and for the full check. Without it, the server dies.
Pull Request resolved: https://github.com/facebook/prepack/pull/2245
Differential Revision: D8891234
Pulled By: gaearon
fbshipit-source-id: 2c309a1718ed00ca5839cb3a586386e3e779f029
Summary:
Release note: none
The main part of this PR is to ensure that completions and the effects they complete are always 1-1.
Along the way some bugs got fixed#2241.
This also includes a little side project: limiting the console spew when doing "yarn test-serializer --fast", which is something I do all the time and I'm much happier this way.
Pull Request resolved: https://github.com/facebook/prepack/pull/2258
Differential Revision: D8864448
Pulled By: hermanventer
fbshipit-source-id: a7f257a7e07211ecd6069b84330aa3305609c5d2
Summary:
Since I'm adding a new experiment I figured I'd delete an equivalent sized one.
Last year I added an option that runs the Prepack program by invoking Node.js JS runtime which lets us prepack the whole module system and initialization. It's essentially a packager with perfect Node.js module resolution semantics. It did this by modeling Node's native environment as Prepack bindings.
This PR removes that whole option.
There's a few reasons why I don't think that worked out as a good idea.
- It's not solving a real need. It is hard to keep different module systems in tact. There is always something in the ecosystem that breaks down and using the canonical one solves that. However, in practice, if there is a need for bundling the ecosystem itself adapts to the toolchain. So it's not actually that hard to bundle up a CLI even with Webpack, even if it's strictly not 100% compatible, by tweaking a few downstream depenencies.
- Running the resulting bundle is tricky. The resulting bundle includes the JS parts of Node. This overlaps with what Node.js adds at runtime so it runs it twice. The ideal is actually to build a custom distribution of Node.js but this is generally overkill for what people want.
- Bindings change a lot. While Node.js's API notoriously doesn't change much. The internals do change a lot. By picking the API boundary in the middle of the internals of Node.js, it risks changing with any version. While technically observable changes, nobody else relies on these details. If this option was worth its weight, someone could probably maintain it but so far that has not been the case so we had to disable this option in CI to upgrade Node.
However, going forward I think there are alternative approaches we can explore.
- First class module system. This is something we really need at some point. A first class module system would be able to load Node.js module files from disk and package them up while excluding others. It doesn't have to be literally Node.js's module system. Close enough is ok. Especially as standards compliant ECMAScript modules get more popular. This lets us target compiling output that runs after Node's initialization.
- By introducing havocing and membranes in the boundaries, it becomes possible to initialize Node.js modules without actually knowing the internal of the boundaries.
- We've started optimizing residual functions which is much more interesting. However, this requires that code puts some constraints on how it works with its environment. It's not designed to be fully backwards compatible. That's probably a good thing but that also means that we can put constraints on the modules being Prepacked.
This removes the ability to prepack Prepack itself which is unfortunate but already wasn't being tested. To speed up Prepack itself, the [LLVM backend](https://github.com/facebook/prepack/pull/2264) seems much more useful if it can ever work on Prepack itself.
Pull Request resolved: https://github.com/facebook/prepack/pull/2267
Differential Revision: D8863788
Pulled By: sebmarkbage
fbshipit-source-id: d777ec9a95c8523b3386cfad553d9f691ec59074
Summary:
Release notes: upgrades Prepack to use Babel 7.0.0-beta.53
This is a big PR that updates all of Prepack to Babel 7. Babylon is now `babel/parser` and pretty much all of the the previous Babel packages are now located in scoped packages. I had to make a bunch of changes around Jest/Flow/Webpack to get this all working. The build times of building Prepack itself seem considerably faster (easily twice as fast locally). I followed most of the Babel 6 -> 7 upgrade guide from the Babel site in terms of changing nodes and type definitions to match the new ones.
Pull Request resolved: https://github.com/facebook/prepack/pull/2256
Differential Revision: D8850583
Pulled By: trueadm
fbshipit-source-id: 2d2aaec25c6a1ccd1ec0c08c5e7e2a71f78ac2d8
Summary:
stop abstract loops from getting stuck in infinite loops when body contains a return, throw or break completion
Object.assign should no longer lose values when snapshotting in certain cases
Allow __optimize to work in conditional contexts
Move abstract Object.assign temporals into a helper function
new color scheme for the website
Add ability to rewrite generated code via global.__output
adds an optimization to Object.assign that attempts to merge calls together where possible
Added --expectedCounts parameter to test262-runner so that success can depend on the value of the time-out and the version of the test suite that is used.
Updated the test262 submodule to latest version
Reviewed By: cblappert
Differential Revision: D8837068
fbshipit-source-id: 218879046dbb0e307cbf3a13ba9a66d11e99a3f0
Summary:
This will fail CI if we forgot to run `yarn prettier` before committing.
We do the same in React repo. It prevents committing stale files that later cause unexpected changes.
Pull Request resolved: https://github.com/facebook/prepack/pull/2212
Differential Revision: D8784406
Pulled By: gaearon
fbshipit-source-id: ca948b8e088be8886c8ba865f280ba8d72750f69
Summary:
Improved source location information
Only havoc abstract value args when really necessary
Debugger recognizes sourcemaps and opens the correct file/line when debugging in Nuclide.
Make React tests fast
Special case an expression simplification for Instant Render
Fix temporal assignments to intrinsics that happen inside non deterministic loops
Enhanced handling of Array.map applied to abstract arrays
Support dynamic invariants via __assume
We got a logo!
Add React Native mocks to Prepack
Reviewed By: simonhj
Differential Revision: D8739156
fbshipit-source-id: d535dfe0c5bcbee4aaed28f75d45cbedcc51344b
Summary:
Release notes: none
Make it easier to do local development testing of our internal bundle without having to hack the settings each time.
Closes https://github.com/facebook/prepack/pull/2202
Differential Revision: D8731922
Pulled By: trueadm
fbshipit-source-id: 71d92c3b90f448667c4a9ada1f1b0848ece5003b
Summary:
Release notes: adds React Native mocks to Prepack
This adds React Native mocks to Prepack and a few basic tests to demonstrate inlining of `View` and `Text`.
Closes https://github.com/facebook/prepack/pull/2096
Differential Revision: D8723932
Pulled By: NTillmann
fbshipit-source-id: 38bd265cd8935ebdf30266ec337378b4ea5b09d6
Summary:
Currently we have a single giant file with all tests, and a giant snapshot. This is both slow, and hard to work with and iterate on.
In this PR I will refactor our test setup.
- [x] Split it up into multiple files (gets the test running from 45s to 27s)
- [x] Run Prettier on test files
- [x] Split tests further for better performance
- [x] Make it possible to run one test file
- [x] Fix the issue with double test re-runs in watch mode on changes in the test file
- [x] Refactor error handling
- [x] Run Prettier on fixtures
- [x] Add a fast mode with `yarn test-react-fast <Filename>`
- [x] Fix double reruns on failure
Potential followups:
- [x] Figure out why test interruption broke (need https://github.com/facebook/jest/issues/6599 and https://github.com/facebook/jest/issues/6598 fixed)
- [x] Revisit weird things like `this['React']` assignment with a funny comment in every test
Closes https://github.com/facebook/prepack/pull/2187
Differential Revision: D8713639
Pulled By: gaearon
fbshipit-source-id: 5edbfa4e61610ecafff17c0e5e7f84d44cd51168
