This PR implements HTML generation from documentation IR for all suggestion database entries and replaces the old documentation panel with a newer one.
Additional adjustments to the looks of the documentation would be applied separately in a future PR. This PR focuses on the fastest possible delivery of a usable documentation panel. We want to test it in real-world use cases and gather feedback for future improvements.
Documentation demo scene with mocked data:
https://user-images.githubusercontent.com/6566674/213436313-88753ed8-346f-423e-956e-7db39f5dc266.mp4
Component browser with actual engine-provided data:
https://user-images.githubusercontent.com/6566674/213436375-d0ec074b-f7a6-4deb-a7de-3adee999cc86.mp4
# Important Notes
- Fixed language protocol data structures.
- Scrolling to the selected method is also implemented here.
- Also, the selected item is highlighted with yellow.
- Only some pieces of information we have are displayed. For example, we don't display return types for methods or types of arguments.
- A bunch of code related to previous implementation is removed, but probably not all of it.
The fix consists of two parts:
1. All the "review-apply" and "store temporary md" actions in the searcher controller are now guarded by an ignored transaction.
2. Because some of the temporary state may reach the UR frames assigned to other actions, added a bunch of code for removing all temporary expressions from the code and use it after restoring a frame. We may consider using it after project load as well.
### Important Notes
Added a useful method "log_err" to ResultOps (so every Result will have those).
Logging: Replace tracing with an efficient logging implementation, with 0-runtime cost for disabled log levels. (https://www.pivotaltracker.com/story/show/183755412)
Profiling: Support submitting `profiler` events to the User Timing Web API, so that measurements can be viewed directly in the browser. (https://www.pivotaltracker.com/story/show/184003550)
# Important Notes
Logging interface:
- The macros (`warn!`, etc.) now take standard `format_args!` arguments (the tracing implementations accepted a broader syntax).
- Compile-time log levels can now be set through the CLI, like so:
`./run ide start --log-level=trace --uncollapsed-log-level=info`
Profiling:
- The hotkey Ctrl+Alt+Shift+P submits all `profiler` events logged since the application was loaded to the Web API, so that they can then be viewed with the browser's developer tools. Note that standard tools are not able to represent async task lifetimes or metadata; this is a convenient interface to a subset of `profiler` data.
- As an alternative interface, a runtime flag enables continuous measurement submission. In the browser it can be set through a URL parameter, like http://localhost:8080/?emit_user_timing_measurements=true. Note that this mode significantly impacts performance.
Fixes https://www.pivotaltracker.com/story/show/184216698
Reduced impact of node dropdown widgets on load times by deferring creation of grid views until each widget is opened. This also improves node editing time, as the dropdowns are not recreated immediately.
This approach of lazy initialization now caused a significant lag when opening the dropdown. Two major causes of the lag spike is glyph generation (msdfgen, `new_glyph`) and shader compilation (happened every time, because each dropdown has unique layer stack). To reduce the impact of that, the shader compiler now caches the shaders based on generated shader source. Glyph creation hasn't been changed and is still slow. The startup performance is now roughly where it was before introducing widgets.
This PR is a draft PR while I learn EnsoGL. The eventual goal is to implement the projects list portion of the cloud dashboard in this PR. This PR will implement part of https://www.pivotaltracker.com/n/projects/2539513/stories/183557950
### Important Notes
This PR is still really rough and contains a lot of hacks & hard-coded values. The FRP usage is also likely to be suboptimal and need fixing.
Fix issues noted here: https://github.com/enso-org/enso/pull/3678#issuecomment-1273623924
- Time complexity of an operation during line-redrawing scaled quadratically with number of lines in a change; now linear.
- Time complexity of adding `n` selections to a group was `O(n^2)`. Now it is `O(n log n)`, even if the selections are added one by one.
Also fix a subtle bug I found in `Group::newest_mut`: It returned a mutable reference that allowed breaking the *sorted* invariant of the selection group. The new implementation moves the element to invalidated space before returning a reference (internally to `LazyInvariantVec`), so that if it is mutated it will be moved to its correct location.
### Important Notes
New APIs:
- `NonEmptyVec::extend_at` supports inserting a sequence of elements at a location, with asymptotically-better performance than a series of `insert`s. (This is a subset of the functionality of `Vec::splice`, a function which we can't safely offer for `NonEmptyVec`).
- `LazyInvariantVec` supports lazily-restoring an invariant on a vector. For an invariant such as *sorted* (or in this case, *sorted and merged*), this allows asymptotically-better performance than maintaining the invariant with each mutation.
This PR fixes the `code_to_insert` method of entry to insert valid code according to the newest language version. Also created a separate method for getting imports required by given entry.
Now, method entries do not add imports (except when they are extensions), and are insterted with place for this type `_.method`. Static methods and constructors are inserted with the type name, and proper import for type is added.
There are some additional work done:
* The ReferentName and NormalizedName were removed, as we are now case-sensitive.
* All QualifiedName structures were replaced with new one in `name` module, as there is no longer functional difference between type qualified name and module qualified name.
* The QualifiedName structure removes "Main" module segment where it is not necessary, thus simplifying our code base and avoiding potential issues.
* Added macro `mock_suggestion_database` which should make creating consistent mocks of SuggestionDatabase much simpler.
* Fixed bug where the visualization preview show no value for some time.
https://user-images.githubusercontent.com/3919101/202750275-0d378d5f-1482-4637-bdcd-c428a9eac0d4.mp4
# Important Notes
The tests in controller/searcher.rs file are not of the best quality, but those will be overhauled anyway when implementing my next task.
[ci no changelog needed]
[Task link](https://www.pivotaltracker.com/story/show/182675703)
This PR implements the actual integration of the breadcrumbs panel with the component list panel. A special breadcrumbs controller (`controller::searcher::breadcrumbs`) is tracking the currently opened module with a list of its parents. The searcher presenter uses the API of the controller to sync the displayed list of breadcrumbs with the controller state.
https://user-images.githubusercontent.com/6566674/193064122-7d3fc4d6-9148-4ded-a73e-767ac9ac83f8.mp4
# Important Notes
- There is an `All` breadcrumb displayed at all times at the beginning of the list. It will be replaced with a section name as part of [Section Title on Component Browser's Breadcrumbs Panel](https://www.pivotaltracker.com/story/show/182610561) task.
- I changed the implementation of `project::main_module_id`, `project::QualifiedName::main_module`, and `API::main_module` so that they are logically connected which each other.
- I adjusted the Breadcrumbs View to avoid "appearance" animation glitches when opening new modules. `set_entries` was replaced with the `set_entries_from` endpoint.
Show custom icons in Component Browser for entries that have a non-empty `Icon` section in their docs with the section's body containing a name of a predefined icon.
https://www.pivotaltracker.com/story/show/182584336
#### Visuals
A screenshot of a couple custom icons in the Component Browser:
<img width="346" alt="Screenshot 2022-07-27 at 15 55 33" src="https://user-images.githubusercontent.com/273837/181265249-d57f861f-8095-4933-9ef6-e62644e11da3.png">
# Important Notes
- The PR assigns icon names to four items in the standard library, but only three of them are shown in the Component Browser because of [a parsing bug in the Engine](https://www.pivotaltracker.com/story/show/182781673).
- Icon names are assigned only to four items in the standard library because only two currently predefined icons match entries in the currently defined Virtual Component Groups. Adjusting the definitions of icons and Virtual Component Groups is covered by [a different task](https://www.pivotaltracker.com/story/show/182584311).
- A bug in the documentation of the Enso protocol message `DocSection` is fixed. A `text` field in the `Tag` interface is renamed to `body` (this is the field name used in Engine).
Implement generation of Java AST types from the Rust AST type definitions, with support for deserializing in Java syntax trees created in Rust.
### New Libraries
#### `enso-reflect`
Implements a `#[derive(Reflect)]` macro to enable runtime analysis of datatypes. Macro interface includes helper attributes; **the Rust types and the `reflect` attributes applied to them fully determine the Java types** ultimately produced (by `enso-metamodel`). This is the most important API, as it is used in the subject crates (`enso-parser`, and dependencies with types used in the AST). [Module docs](https://github.com/enso-org/enso/blob/wip/kw/parser/ast-transpiler/lib/rust/reflect/macros/src/lib.rs).
#### `enso-metamodel`
Provides data models for data models in Rust/Java/Meta (a highly-abstracted language-independent model--I have referred to it before as the "generic representation", but that was an overloaded term).
The high-level interface consists of operations on data models, and between them. For example, the only operations needed by [the binary that drives datatype transpilation](https://github.com/enso-org/enso/blob/wip/kw/parser/ast-transpiler/lib/rust/parser/generate-java/src/main.rs) are: `rust::to_meta`, `java::from_meta`, `java::transform::optional_to_null`, `java::to_syntax`.
The low-level interface consists of direct usage of the datatypes; this is used by [the module that implements some serialization overrides](https://github.com/enso-org/enso/blob/wip/kw/parser/ast-transpiler/lib/rust/parser/generate-java/src/serialization.rs) (so that the Java interface to `Code` references can produce `String`s on demand based on serialized offset/length pairs). The serialization override mechanism is based on customizing, not replacing, the generated deserialization methods, so as to be as robust as possible to changes in the Rust source or in the transpilation process.
### Important Notes
- Rust/Java serialization is exhaustively tested for structural compatibility. A function [`metamodel::meta::serialization::testcases`](https://github.com/enso-org/enso/blob/wip/kw/parser/ast-transpiler/lib/rust/metamodel/src/meta/serialization.rs) uses `reflect`-derived data to generate serialized representations of ASTs to use as test cases. Its should-accept cases cover every type a tree can contain; it also produces a representative set of should-reject cases. A Rust `#[test]` confirms that these cases are accepted/rejected as expected, and generated Java tests (see Binaries below) check the generated Java deserialization code against the same test cases.
- Deserializing `Code` is untested. The mechanism is in place (in Rust, we serialize only the offset/length of the `Cow`; in Java, during deserialization we obtain a context object holding a buffer for all string data; the accessor generated in Java uses the buffer and the offset/length to return `String`s), but it will be easier to test once we have implemented actually parsing something and instantiating the `Cow`s with source code.
- `#[tagged_enum]` [now supports](https://github.com/enso-org/enso/blob/wip/kw/parser/ast-transpiler/lib/rust/shapely/macros/src/tagged_enum.rs#L36-L51) control over what is done with container-level attributes; they can be applied to the container and variants (default), only to the container, or only to variants.
- Generation of `sealed` classes is supported, but currently disabled by `TARGET_VERSION` in `metamodel::java::syntax` so that tests don't require Java 15 to run. (The same logic is run either way; there is a shallow difference in output.)
### Binaries
The `enso-parser-generate-java` crate defines several binaries:
- `enso-parser-generate-java`: Performs the transpilation; after integration, this will be invoked by the build script.
- `java-tests`: Generates the Java code that tests format deserialization; after integration this command will be invoked by the build script, and its Java output compiled and run during testing.
- `graph-rust`/`graph-meta`/`graph-java`: Produce GraphViz representations of data models in different typesystems; these are for developing and understanding model transformations.
Until integration, a **script regenerates the Java and runs the format tests: `./tools/parser_generate_java.sh`**. The generated code can be browsed in `target/generated_java`.
This PR introduces a new structure delivered by Searcher Controller: The Component List.
The Component list is hierarchical, and its structure reflects how the components are displayed in Component Browser - only "submodule" section at this point, other sections will be covered in next tasks.
This does not introduce anything visual; the structures are tested in unit tests.
Add a method in `SuggestionDatabase` allowing to find a suggestion entry by a fully qualified path, working faster than a linear search through all the entries in the `SuggestionDatabase`.
https://www.pivotaltracker.com/story/show/181852566
# Important Notes
- **Testing:** when testing the PR with the newest currently available nightly version of the Engine, you may observe the following warnings in the Chrome JS Developer Console (the numerical values may differ):
index.ts?ab16:289 WARN app/gui/src/model/suggestion_database.rs:61 An existing suggestion entry id at Standard.Base.Nothing.Nothing.is_nothing was overwritten with 768.
index.ts?ab16:289 WARN app/gui/src/model/suggestion_database.rs:61 An existing suggestion entry id at Standard.Base.Data.Numbers.Integer.up_to was overwritten with 936.
index.ts?ab16:289 WARN app/gui/src/model/suggestion_database.rs:61 An existing suggestion entry id at Standard.Base.Data.Numbers.Integer.down_to was overwritten with 937.
index.ts?ab16:289 WARN app/gui/src/model/suggestion_database.rs:61 An existing suggestion entry id at Standard.Base.Data.Text.Text.== was overwritten with 971.
This is a result of bugs in the Standard Library. Those are planned to be addressed by:
- #3480 short-term (to fix the specific bugs currently present in the Standard Library),
- https://www.pivotaltracker.com/story/show/182283983 long-term (to improve the Engine such that it disallows introducing this category of bugs in the Standard Library in the future).
(For more details, see also: https://discord.com/channels/401396655599124480/978929754138877962.)
As a result of the bugs mentioned above, the Engine is responding with some invalid replies. In case of such invalid replies, warnings are emitted in the JS Dev Console. Other than the warnings, the code is expected to work correctly with the Engine for the cases when the Engine returns correct data.
- A `HashMapTree` was used for storing the map. A quick back-of-the-envelope estimation of memory usage of a simpler alternative (a one-level `HashMap`) would put the alternative at ~1MB (~10k entities × averaged ~100 bytes per entity path), which was considered too much in a discussion with @farmaazon.
- The `HashMapTree::remove` method deletes a whole subtree of a `HashMapTree`. An alternative removal method was implemented for use in `suggestion_database::QualifiedNameToIdMap` to better match `HashMap` entry removal semantics.
- In case of path collisions, a warning is emitted.
- Paths are treated case-sensitively.
- The new method of the `SuggestionDatabase` type is currently only used in unit tests. This matches the explicit requirement in the Task's description.
- JS Developer Console logging was enabled for the `tracing` package. The `WARN` level was picked to match the default level enabled in the "old" logging infrastructure.
[ci no changelog needed]
Implement a command that launches the application, runs a series of steps (a "workflow"), writes a profile to a file, and exits.
See: [#181775808](https://www.pivotaltracker.com/story/show/181775808)
# Important Notes
- The command to capture run and profile is used like: `./run profile --workflow=new_project --save-profile=out.json`. Defining some more workflows (collapse nodes, create node and edit value) comes next; they are implemented with the same infrastructure as the integration-tests.
- The `--save-profile` option can also be used when profiling interactively; when the option is provided, capturing a profile with the hotkey will write a file instead of dumping the data to the devtools console.
- If the IDE panics, the error message is now printed to the console that invoked the process, as well as the devtools console. (If a batch workflow fails, this allows us to see why.)
- New functionality (writing profile files, quitting on command, logging to console) relies on Electron APIs. These APIs are implemented in `index.js`, bridged to the render process in `preload.js`, and wrapped for use in Rust in a `debug_api` crate.
[ci no changelog needed]
This is fixed copy of already reviewed #3384
[Task link](https://www.pivotaltracker.com/story/show/181413200)
This PR implements content clipping for the ScrollArea component.
List of changes:
- Implemented `InstanceWithAttachedLayer` abstraction that allows creating additional sublayers for our components. In the future, this abstraction can be used for text rendering as well (right now text rendering requires additional hardcoded layers).
- Fixed `complex-shape-system` demo scene by removing `node_searcher_mask` layer.
- Fixed `SublayersModel::remove` - it was not clearing the `layer_placement` hashmap.
- Implemented disabling the wheel scrolling in `Navigator`, and refactored it to reduce the number of functions arguments by introducing a `NavigatorSettings` struct.
Video (`scroll_area` demo):
https://user-images.githubusercontent.com/6566674/164506455-e177a7a7-9f1c-4f50-888f-112423cebbe4.mp4
# Important Notes
- `InstanceWithAttachedLayer` is implemented in such a way that it allows an extension in the future - namely to use it to simplify text rendering. The implementation might be simplified though.
[ci no changelog needed]
[Task link](https://www.pivotaltracker.com/story/show/181413200)
This PR implements content clipping for the ScrollArea component.
List of changes:
- Implemented `InstanceWithAttachedLayer` abstraction that allows creating additional sublayers for our components. In the future, this abstraction can be used for text rendering as well (right now text rendering requires additional hardcoded layers).
- Fixed `complex-shape-system` demo scene by fixing `node_searcher_mask` layer.
- Fixed `SublayersModel::remove` - it was not clearing the `layer_placement` hashmap.
- Implemented disabling the wheel scrolling in `Navigator`, and refactored it to reduce the number of functions arguments by introducing a `NavigatorSettings` struct.
Video (`scroll_area` demo):
https://user-images.githubusercontent.com/6566674/164506455-e177a7a7-9f1c-4f50-888f-112423cebbe4.mp4
# Important Notes
- `InstanceWithAttachedLayer` is implemented in such a way that it allows an extension in the future - namely to use it to simplify text rendering. The implementation might be simplified though.
When a new node is created with the <kbd>TAB</kbd> key or by clicking the `(+)` on-screen button while multiple nodes are selected, place the new node below all the selected nodes. (Previously, the new node was placed below the node that was selected earliest.)
Additionally, when placing a new node below an existing non-error node with a visualization enabled, place the new node below the visualization. (Previously, the new node was placed to the left of the visualization.)
https://www.pivotaltracker.com/story/show/180887079
#### Visuals
The following screencast demonstrates the feature on various arrangements of selected nodes, with visualization enabled and disabled.
https://user-images.githubusercontent.com/273837/159971452-148aa4d7-c0f3-4b48-871a-a2783989f403.mov
The following screencast demonstrates that new nodes created by double-clicking an output port of a node with visualization enabled are now placed below the visualization:
https://user-images.githubusercontent.com/273837/160107733-e3f7d0f9-0161-49d1-8cbd-06e18c843a20.mov
# Important Notes
- Some refactorings that were needed for this PR were ported from the #3301 PR:
- the code responsible for calculating the positions of new nodes was moved to a separate module (`new_node_position`);
- the `free_place_finder` module was made a submodule of the `new_node_position` module, due to the latter being its only user.
@akavel spotted a compilation error, when building test for graph_editor crate. The cause was that:
* prelude without serde still added serde derivatives in im_string_newtype
* and the graph_editor needs serde from prelude anyway (because it wants to have serializable ImStrings).
