Update shader tools to new version. Notably, this release contains spirv-cross with fixed issue https://github.com/KhronosGroup/SPIRV-Cross/issues/2129.
# Important Notes
Spirv-cross has no versioning that we could use to specify requirements for using system-wide installed versions. Instead, we have to download the prebuilt distribution by default, so we can rely on known good versions. The usage of binaries in PATH can still be enabled with a build flag, but it is discouraged due to severity of the bug and no easy way of detecting it. If the project is built with buggy shader tools version, the application will run, but it will be visually slightly broken in unexpected ways.
One of the tests for the build script had a hardcoded, windows-specific path. Now it is fixed, and a portable, temporary directory is used. Additionally, some missing asserts were added.
Some small improvements relating to rendering:
- Add a debug option: `-debug.pixel-read-period`. This can be used to measure the performance impact of checking the pointer location on different hardware. [On my development box, it makes no difference to performance.] (Closes#5490).
- Unbind pixel pack buffers after each use. This is recommended practice. It has no performance impact on my machine, and allows SpectorJS to run (`-debug.enable-spector`). (Closes#5941).
Also, simplify the profiling CLI: the `profile.load-profile` and `profile.save-profile` options have been renamed to `profile.load`/`profile.save`; `profile.save` now has a default filename, so you can capture a profile at any time in Electron with Ctrl+Alt+P and it will be written to `profile.json`.
Somebody forgot to apply `./run fmt` before committing to develop, so now we have a lot of whitespace changes in files.
# Important Notes
See https://github.com/enso-org/enso/issues/5166
Docker output is not really stable across releases. This PR adds usage of the `--quiet` flag, so it ends up in something that is effectively stable.
Unfortunately, we cannot use the `--format,` as it does not work on Windows.
Implement new Enso documentation parser; remove old Scala Enso parser.
Performance: Total time parsing documentation is now ~2ms.
# Important Notes
- Doc parsing is now done only in the frontend.
- Some engine tests had never been switched to the new parser. We should investigate tests that don't pass after the switch: #5894.
- The option to run the old searcher has been removed, as it is obsolete and was already broken before this (see #5909).
- Some interfaces used only by the old searcher have been removed.
2nd PR for IDE/Cloud authorization with cognito. This PR introduces boilerplate react app + some amplify code to fetch the access token + username of the currently logged in user, if they are already authenticated.
Registration + Login + Set Username + Forgot Password flows are to be added in next PRs to keep the changes reviewable.
Precompute MSDFs for all ASCII glyphs; after this, we no longer spend any time on MSDF computations when loading or interacting with the example projects.
Also shader precompilation (during build) is now parallel; if you have many cores and an SSD, it's now practically instant.
Closes#5722.
# Important Notes
- The *dynamic-assets* mechanism now used for MSDF data and shaders is versatile, and could be used to pre-seed any other computation-intensive runtime caches.
This PR changes build script's `ide watch` and `ide start` commands, so they don't use `electron-builder` to package. Instead, they invoke `electron` directly, significantly reducing time overhead.
`ide watch` will now start Electron process, while continuously rebuilding gui and the client in the background. Changes can be puilled by reloading within the electron, or closing the electron and letting it start once again. To stop, the script should be interrupted with `Ctrl+C`.
In cloud we want to allow users to create new project from the template. List of templates is a bit outdated and doesn't contain all from the https://github.com/enso-org/project-templates. This PR simply adds missing ones
This PR aims to fix current issues with cloud IDE.
1. Backend image:
* bumping the system version to avoid glibc version mismatch on parser;
* explicitly installing required GraalVM components;
2. Frontend upload:
* update to follow the new file naming;
* uploading the whole shaders subtree.
Rename the CLI option to set the WASM log level; applied some suggested simplifications, to `LogLevel` and to the type it was based on.
# Important Notes
This addresses review of #4017.
Logging: Replace tracing with an efficient logging implementation, with 0-runtime cost for disabled log levels. (https://www.pivotaltracker.com/story/show/183755412)
Profiling: Support submitting `profiler` events to the User Timing Web API, so that measurements can be viewed directly in the browser. (https://www.pivotaltracker.com/story/show/184003550)
# Important Notes
Logging interface:
- The macros (`warn!`, etc.) now take standard `format_args!` arguments (the tracing implementations accepted a broader syntax).
- Compile-time log levels can now be set through the CLI, like so:
`./run ide start --log-level=trace --uncollapsed-log-level=info`
Profiling:
- The hotkey Ctrl+Alt+Shift+P submits all `profiler` events logged since the application was loaded to the Web API, so that they can then be viewed with the browser's developer tools. Note that standard tools are not able to represent async task lifetimes or metadata; this is a convenient interface to a subset of `profiler` data.
- As an alternative interface, a runtime flag enables continuous measurement submission. In the browser it can be set through a URL parameter, like http://localhost:8080/?emit_user_timing_measurements=true. Note that this mode significantly impacts performance.
Upgrading to GraalVM 22.3.0.
# Important Notes
- Removed all deprecated `FrameSlot`, and replaced them with frame indexes - integers.
- Add more information to `AliasAnalysis` so that it also gathers these indexes.
- Add quick build mode option to `native-image` as default for non-release builds
- `graaljs` and `native-image` should now be downloaded via `gu` automatically, as dependencies.
- Remove `engine-runner-native` project - native image is now build straight from `engine-runner`.
- We used to have `engine-runner-native` without `sqldf` in classpath as a workaround for an internal native image bug.
- Fixed chrome inspector integration, such that it shows values of local variables both for current stack frame and caller stack frames.
- There are still many issues with the debugging in general, for example, when there is a polyglot value among local variables, a `NullPointerException` is thrown and no values are displayed.
- Removed some deprecated `native-image` options
- Remove some deprecated Truffle API method calls.
This PR updates the build script:
* fixed issue where program version check was not properly triggering;
* improved `git-clean` command to correctly clear Scala artifacts;
* added `run.ps1` wrapper to the build script that works better with PowerShell than `run.cmd`;
* increased timeouts to work around failures on macOS nightly builds;
* replaced depracated GitHub Actions APIs (set-output) with their new equivalents;
* workaround for issue with electron builder (python2 lookup) on newer macOS runner images;
* GUI and backend dispatches to cloud were completed;
* release workflow allows creating RC releases.
[ci no changelog needed]
[Task link](https://www.pivotaltracker.com/story/show/181445628).
This PR implements a Breadcrumbs panel for the new component browser.
The Breadcrumbs is a horizontal list of text labels separated by a special icon and has an optional ellipsis icon at the end.
It is implemented using the new GridView component.
Video:
Demo of adding new breadcrumbs, scrolling behavior, and selecting breadcrumbs with the mouse.
https://user-images.githubusercontent.com/6566674/189199432-77807cef-00dc-4abe-b95c-b17a536f59f6.mp4
Demo of selecting breadcrumbs with keyboard shortcuts:
https://user-images.githubusercontent.com/6566674/189199603-53e55335-73ba-4ed7-8291-4455144c06aa.mp4
# Important Notes
- This PR implements an old interaction of the design of the component browser. The new design of the breadcrumbs can not be easily integrated into the current look of the component browser, so we would need to update styles later. It should be a relatively simple task. *The implementation uses color from the new design though. (but not fonts and sizes)*
- I found a bug in the grid view implementation that causes panics at runtime in some conditions. The reason is triggering FRP endpoints while constructing new entries. This issue is fixed in the PR.
[ci no changelog needed]
[Task link](https://www.pivotaltracker.com/story/show/182955595)
This PR implements variable column widths in the new Grid View component. We need this feature to quickly implement various parts of the UI, including the breadcrumbs panel of the component browser.
There are two ways to change the width of the specific column:
1. "From the outside", using the `set_column_width` endpoint of the Grid View
2. "From the inside", using the `override_column_width` endpoint of the EntryFrp.
Both ways work similarly, but the latter is helpful for our breadcrumbs implementation, as it allows for entry to decide on the width of the column by its content.
See the screencast with three grid views. The top-left one has every even column shrunk by GridView API. Every grid view has a second column extended by EntryFrp API.
https://user-images.githubusercontent.com/6566674/185060985-7b7df076-c659-41fa-977a-22875493f8d4.mp4
This PR replaces webpack with esbuild, as our bundler.
The change leads to out-of-the-box ~5x improvement in bundling times, reducing the latency in watch-based workflows.
Along with this a new development server (with live reload capacity) has been introduced to support watch command.
[ci no changelog needed]
### Important Notes
* workflow for checking docs has been removed because it was using outdated prettier version and caused troubles; while the same check is performed in a better way by the GUI/Lint job.
* introduced little more typescript in the scripts in place of js, usually with minimal changes.
### Pull Request Description
Using the new tooling (#3491), I investigated the **performance / compile-time tradeoff** of different codegen options for release mode builds. By scripting the testing procedure, I was able to explore many possible combinations of options, which is important because their interactions (on both application performance and build time) are complex. I found **two candidate profiles** that offer specific advantages over the current `release` settings (`baseline`):
- `thin16`: Supports incremental compiles in 1/3 the time of `baseline` in common cases. Application runs about 2% slower than `baseline`.
- `fat1-O4`: Application performs 13% better than `baseline`. Compile time is almost 3x `baseline`, and non-incremental.
(See key in first chart for the settings defining these profiles.)
We can build faster or run faster, though not in the same build. Because the effect sizes are large enough to be impactful to developer and user experience, respectively, I think we should consider having it both ways. We could **split the `release` profile** into two profiles to serve different purposes:
- `release`: A profile that supports fast developer iteration, while offering realistic performance.
- `production`: A maximally-optimized profile, for nightly builds and actual releases.
Since `wasm-pack` doesn't currently support custom profiles (rustwasm/wasm-pack#1111), we can't use a Cargo profile for `production`; however, we can implement our own profile by overriding rustc flags.
### Performance details
As you can see, `thin16` is slightly slower than `baseline`; `fat1-O4` is dramatically faster.
<details>
<summary>Methodology (click to show)</summary>
I developed a procedure for benchmarking "whole application" performance, using the new "open project" workflow (which opens the IDE and loads a complex project), and some statistical analysis to account for variance. To gather this data:
Build the application with profiling:
`./run.sh ide build --profiling-level=debug`
Run the `open_project` workflow repeatedly:
`for i in $(seq 0 9); do dist/ide/linux-unpacked/enso --entry-point profile --workflow open_project --save-profile open_project_thin16_${i}.json; done`
For each profile recorded, take the new `total_self_time` output of the `intervals` tool; gather into CSV:
`echo $(for i in $(seq 0 9); do target/rust/debug/intervals < open_project_thin16_${i}.json | tail -n1 | awk '{print $2}'; do`
(Note that the output of intervals should not be considered stable; this command may need modification in the future. Eventually it would be nice to support formatted outputs...)
The data is ready to graph. I used the `boxplot` method of the [seaborn](https://seaborn.pydata.org/index.html) package, in order to show the distribution of data.
</details>
#### Build times
In the case of changing a file in `enso-prelude`, with the current `baseline` settings rebuilding takes over 3 minutes. With the `thin16` settings, the same rebuild completes in 40 seconds.
(To gather this data on different hardware or in the future, just run the new `bench-build.sh` script for each case to be measured.)
* The bash entry point was renamed `run.sh` -> `run`. Thanks to that `./run` works both on Linux and Windows with PowerShell (sadly not on CMD).
* Everyone's favorite checks for WASM size and program versions are back. These can be disabled through `--wasm-size-limit=0` and `--skip-version-check` respectively. WASM size limit is stored in `build-config.yaml`.
* Improved diagnostics for case when downloaded CI run artifact archive cannot be extracted.
* Added GH API authentication to the build script calls on CI. This should fix the macOS build failures that were occurring from time to time. (Actually they were due to runner being GitHub-hosted, not really an OS-specific issue by itself.)
* If the GH API Personal Access Token is provided, it will be validated. Later on it is difficult to say, whether fail was caused by wrong PAT or other issue.
* Renamed `clean` to `git-clean` as per suggestion to reduce risk of user accidently deleting unstaged work.
* Whitelisting dependabot from changelog checks, so PRs created by it are mergeable.
* Fixing issue where wasm-pack-action (third party) randomly failed to recognize the latest version of wasm-pack (macOS runners), leading to failed builds.
* Build logs can be filtered using `ENSO_BUILD_LOG` environment variable. See https://docs.rs/tracing-subscriber/0.3.11/tracing_subscriber/struct.EnvFilter.html#directives for the supported syntax.
* Improve help for ci-run source, to make clear that PAT token is required and what scope is expected there.
Also, JS parts were updated with some cleanups and fixes following the changes made when introducing the build script.
This PR extends the Component Group Entry with icon and option to highlight the text. Here the convert has highlighted "con".
https://user-images.githubusercontent.com/3919101/169046537-4f8b823c-322e-40dc-8abb-24d1d7092341.mp4
### Important Notes
Although this PR includes effort for adjusting Component Group style to better reflect the design, it is not entirely finished: the selection still works badly and will be fixed in another PR.
* Extends the instrumentation of the code base and upgrades some FRPs to the newer API macro.
* Extends the run-graph demo scene to specify a profile via URL without recompilation.
* Fixes labels in the flame graph demo scene.
* Fixes an issue with loading profiles that contains escaped characters.
# Important Notes
* no longer contains the upgrade of the `text::View` to `define_endpoints_2`. This should be fixed as part of the text rendering rewrite.
[ci no changelog needed]
* Extends the instrumentation of the code base and upgrades some FRPs to the newer API macro.
* Extends the run-graph demo scene to specify a profile via URL without recompilation.
* Fixes labels in the flame graph demo scene.
* Fixes an issue with loading profiles that contains escaped characters.
# Important Notes
[ci no changelog needed]
Implement a command that launches the application, runs a series of steps (a "workflow"), writes a profile to a file, and exits.
See: [#181775808](https://www.pivotaltracker.com/story/show/181775808)
# Important Notes
- The command to capture run and profile is used like: `./run profile --workflow=new_project --save-profile=out.json`. Defining some more workflows (collapse nodes, create node and edit value) comes next; they are implemented with the same infrastructure as the integration-tests.
- The `--save-profile` option can also be used when profiling interactively; when the option is provided, capturing a profile with the hotkey will write a file instead of dumping the data to the devtools console.
- If the IDE panics, the error message is now printed to the console that invoked the process, as well as the devtools console. (If a batch workflow fails, this allows us to see why.)
- New functionality (writing profile files, quitting on command, logging to console) relies on Electron APIs. These APIs are implemented in `index.js`, bridged to the render process in `preload.js`, and wrapped for use in Rust in a `debug_api` crate.
Implements a visualization that is integrated with our GUI profiling visualization for the multiprocess data implemented in #3395https://user-images.githubusercontent.com/1428930/165915395-c850c7b2-1cc5-4eb0-8f21-37565d113b1e.mp4
The visualization shows a horizontal line for Engine, Language Server and GUI and renders arrows for each message passed between them. Information about the message is revealed on hover.
# Important Notes
* this PR refactors the tooltip mechanism. Note that this has not been in active use anywhere else, as tooltips for node received a custom implementation and the tooltip that was previously implemented was used nowhere else yet.
[ci no changelog needed]
* The List View component was refactored: it allows for hiding the internal selection widget, and exposes information where the widget should be placed. This allows us to create selection widget in component list panel, so it can be animated between component groups and sections.
* Fixed some warnings when checking WASM code.
* Adjusted the style of Component Group View a little, so it better reflects the design doc. Still not ideal, because the list_view has some weird design regarding padding, but I don't want to stuck in some bigger refactoring.
I will add a video in a few minutes.
# Important Notes
https://user-images.githubusercontent.com/3919101/165507826-60329f9e-7de3-4eb2-9271-292e45568cb2.mov
See: [#181837344](https://www.pivotaltracker.com/story/show/181837344).
I've separated this PR from some deeper changes I'm making to the profile format, because the changeset was getting too complex. The new APIs and tools in this PR are fully-implemented, except the profile format is too simplistic--it doesn't currently support headers that are needed to determine the relative timings of events from different processes.
- Adds basic support for profile files containing data collected by multiple processes.
- Implements `api_events_to_profile`, a tool for converting backend message logs (#3392) to the `profiler` format so they can be merged with frontend profiles (currently they can be merged with `cat`, but the next PR will introduce a merge tool).
- Introduces `message_beanpoles`, a simple tool that diagrams timing relationships between frontend and backend messages.
### Important Notes
- All TODOs introduced here will be addressed in the next PR that defines the new format.
- Introduced a new crate, `enso_profiler_enso_data`, to be used by profile consumers that need to refer to Enso application datatypes to interpret metadata.
- Introduced a `ProfileBuilder` abstraction for writing the JSON profile format; partially decouples the runtime event log structures from the format definition.
- Introducing the conversion performed for `ProfilerBuilder` uncovered that the `.._with_same_start!` low-level `profiler` APIs don't currently work; they return `Started<_>` profilers, but that is inconsistent with the stricter data model that I introduced when I implemented `profiler_data`; they need to return profilers in a created, unstarted state. Low-level async profilers have not been a priority, but once #3382 merges we'll have a way to render their data, which will be really useful because async profilers capture *why* we're doing things. I'll bring up scheduling this in the next performance meeting.
Add logging of EnsoGL performance stats to the profiling framework. Also extends the visualization in the debug scene to show an overview of the performance stats. We now render a timeline of blocks that indicate by their colour the rough FPS range we are in:
https://user-images.githubusercontent.com/1428930/162433094-57fbb61a-b502-43bb-8815-b7fc992d3862.mp4
# Important Notes
[ci no changelog needed]
Needs to be merged after https://github.com/enso-org/enso/pull/3382 as it requires some changes about metadata logging from there. That is why this PR is currently still in draft mode and based on that branch.
* Profiling: intervals tool
* devtools profile generator
* docs
* ignore open intervals, because devtools does
* intervals tool: show interval counts
* Re-add a doc comment. Was accidentally deleted.
* Implement review.
* DURATION_FLOOR_MS
* Rename methods that apply a profile's contents to self
* Description of chrome format
* Link to profile.json format documentation is design doc
* Implement profiling-level switch described in design doc
* lint
Use a new algorithm for placement of new nodes in cases when:
- a) there is no selected node, and the `TAB` key is pressed while the mouse pointer is near an existing node (especially in an area below an existing node);
- b) a connection is dragged out from an existing node and dropped near the node (especially in an area below the node).
In both cases mentioned above, the new node will now be placed in a location suggested by an internal algorithm, aligned to existing nodes. Specifically, the placement algorithm used is similar to when pressing `TAB` with a node selected.
For more details, see: https://www.pivotaltracker.com/story/show/181076066
# Important Notes
- Visible visualizations enabled with the "eye icon" button are treated as part of a node. (In case of nodes with errors, visualizations are not visible, and are not treated as part of a node.)
* Profiling application details
Add enough profiling to account for every missed frame during startup.
See https://www.pivotaltracker.com/story/show/181499507
* Build ActiveInterval hierarchy in profiler_data
* update doctests / await_!
* docs/formatting/naming
* more graph modes
* increase WASM size
Due to new render-profile-flamegraph scene. We should remove these from the main release WASM blob one way or another.
* lint
* fix a test
* Organization (feedback)
* Add @wdanilo to Cargo.lock CODEOWNERS
As discussed after my previous PR got stuck waiting for Cargo.lock review.
* fix doctests
* Update docs. Removed a limitation.
* Creating a new node with the (+) button (#3278)
[The Task](https://www.pivotaltracker.com/story/show/180887253)
A new (+) button on the left-bottom corner appeared. It may be clicked to open searcher in the middle of the scene, as an alternative to tab key.
https://user-images.githubusercontent.com/3919101/154514279-7972ed6a-0203-47cb-9a09-82dba948cf2f.mp4
* The window_control_buttons::common was extracted to separate crate `ensogl-component-button` almost without change.
* This includes a severe refactoring of adding nodes in general in the Graph Editor. The whole responsibility of adding new nodes (and starting their editing) was moved to Graph Editor - the Project View only reacts for GE events to show searcher properly.
* The status bar was moved from the bottom-left corner to the middle-top of the scene. It does not collide with (+) button, and plays "notification" role anyway.
* The `interface` debug scene was buggy. The problem was with one expression's span-tree. When I replaced it, the scene works.
* I've removed "new searcher" API, as it is completely outdated.
* I've changed code owners of integration tests to GUI team, as it is the team writing mostly the integration tests (int rust)
* Fix regression #181528359
* Add docs & remove unused function
* Fix & enable native Rust tests
* Fix formatting
Co-authored-by: Adam Obuchowicz <adam.obuchowicz@enso.org>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
[ci no changelog needed]
This PR reverts commit [0836ce741d](0836ce741d) because of the spotted regression:
To reproduce:
1. Open a default project.
2. Without doing anything else, cmd + click on any node to edit it.
3. Abort editing by pressing escape.
4. Top-most node disappears (it is actually removed from scene)
If you start editing the bottom node - you will also see a visible regression in node searcher's position.
See thread https://discord.com/channels/401396655599124480/950730235719065620/950731247909478410 for details.
Add an API to create a flame graph from profiling data. Also adds a demo scene showcasing the functionality that generates some profiling data by measuring dummy function calls and rendering a flame graph for the dummy data (see video for the result).
Not that the functionality is not yet exposed user-facing in the GUI itself, but only as API and demo scene, therefore [ci no changelog needed]
https://user-images.githubusercontent.com/1428930/155118977-ecac0628-777c-48bd-9aa7-30ee6aef1976.mp4
# Important Notes
* Change from the initial design: labels are shown on the flame graph instead of as a tooltip. This is because tooltips are currently only implemented in the graph editor and would require some additional refactoring (probably taking the better part of a day).
* re-instated the behaviour that logs are shown in the JS console if development mode is active.
[The Task](https://www.pivotaltracker.com/story/show/180887253)
A new (+) button on the left-bottom corner appeared. It may be clicked to open searcher in the middle of the scene, as an alternative to tab key.
https://user-images.githubusercontent.com/3919101/154514279-7972ed6a-0203-47cb-9a09-82dba948cf2f.mp4
# Important Notes
* The window_control_buttons::common was extracted to separate crate `ensogl-component-button` almost without change.
* This includes a severe refactoring of adding nodes in general in the Graph Editor. The whole responsibility of adding new nodes (and starting their editing) was moved to Graph Editor - the Project View only reacts for GE events to show searcher properly.
* The status bar was moved from the bottom-left corner to the middle-top of the scene. It does not collide with (+) button, and plays "notification" role anyway.
* The `interface` debug scene was buggy. The problem was with one expression's span-tree. When I replaced it, the scene works.
* I've removed "new searcher" API, as it is completely outdated.
* I've changed code owners of integration tests to GUI team, as it is the team writing mostly the integration tests (int rust)
