This PR fixes several issues that were appearing when running CI jobs on PRs created against the repository forks:
* electron-builder on Windows and macOS will properly recognize that the secrets are missing and will not attempt to sign the artifacts;
* similarly, fixed the S3 library tests;
* test reporter step will be now skipped, as it does not support forks.
This PR removes unused, commented-out or otherwise spurious code from build script. Also, dependencies were reviewed and cleaned.
No functional changes intended.
This PR updates the build script to retry uploading release assets if the upload fails.
Network issues can cause the upload to fail, failing in turn the whole release process. This PR adds a retry mechanism to the upload process, making the release process more robust.
Now that #9815 has landed, we can finally bump electron-builder to the latest release. As this brings in python3 support out-of-the-box, workaround of the runtime-bump on macOS runners can be removed.
This PR introduces a new installer and uninstaller for the Windows platform.
Both are written in Rust and compiled to a single executable. The executable has no dependencies (other than what is included in the Windows), links the C++ runtime statically if needed.
The change is motivated by numerous issues with with the `electron-builder`-generated installers. The new installer should behave better, not have issues with long paths and unblock the `electron-builder` upgrade (which will significantly simplify the workflow definitions).
To build an installer, one needs to provide the unpacked application (generated by `electron-builder`) and the `electron-builder` configuration (with a few minor extensions). Code signing is also supported.
GitHub made arm64 runners generally available and changed macos-latest label to point to them.
The runner architecture is coupled with GH-hosted runners OS version: macos-13 is the last one to run on x64.
This PR essentially brings back the previous behavior, by explicitly requesting that all our x64 macOS jobs are run on macos-12 (as was before).
We should eventually migrate to macos-13 for x64 macOS and macos-14/macos-latest for arm64 macOS. However, this leads to issues with `npm install` getting stuck, so it should be probably reattempted after the CI rework.
This PR bumps the FlatBuffers version used by the backend to `24.3.25` (the latest version as of now).
Since the newer FlatBuffers releases come with prebuilt binaries for all platforms we target, we can simplify the build process by simply downloading the required `flatc` binary from the official FlatBuffers GitHub release page. This allows us to remove the dependency on `conda`, which was the only reliable way to get the outdated `flatc`.
The `conda` setup has been removed from the CI steps and the relevant code has been removed from the build script.
The FlatBuffers version is no longer hard-coded in the Rust build script, it is inferred from the `build.sbt` definition (similar to GraalVM).
# Important Notes
This does not affect the GUI binary protocol implementation.
While I initially wanted to update it, it turned out farly non-trivial.
As there are multiple issues with the generated TS code, it was significantly refactored by hand and it is impossible to automatically update it. Work to address this problem is left as [a future task](https://github.com/enso-org/enso/issues/9658).
As the Flatbuffers binary protocol is guaranteed to be compatible between versions (unlike the generated sources), there should be no adverse effects from bumping `flatc` only on the backend side.
This PR adds a new file to the release: `assets.json`, that offers information about the assets available in the release.
The purpose is to have one persistent link `https://github.com/enso-org/enso/releases/latest/download/assets.json` that has the current download links that can be consumed by the website.
Also, the release template has been updated to use the same assets information source, rather than duplicate the information about artifact names.
Additionally, additional step for release validation was added, so the CI can alert if one of the expected assets is missing.
Removes a bulk of rust crates that we no longer need, but that added significant install, build and testing time to the Rust parser.
Most significantly, removed `enso-web` and `enso-shapely`, and got rid of many no longer necessary `#![feature]`s. Moved two still used proc-macros from shapely to prelude. The last remaining usage of `web-sys` is within the logger (`console.log`), but we may actually want to keep that one.
This PR updates the Rust toolchain to recent nightly.
Most of the changes are related to fixing newly added warnings and adjusting the feature flags. Also the formatter changed its behavior slightly, causing some whitespace changes.
Other points:
* Changed debug level of the `buildscript` profile to `lint-tables-only` — this should improve the build times and space usage somewhat.
* Moved lint configuration to the worksppace `Cargo.toml` definition. Adjusted the formatter appropriately.
* Removed auto-generated IntelliJ run configurations, as they are not useful anymore.
* Added a few trivial stdlib nightly functions that were removed to our codebase.
* Bumped many dependencies but still not all:
* `clap` bump encountered https://github.com/clap-rs/clap/issues/5407 — for now the warnings were silenced by the lint config.
* `octocrab` — our forked diverged to far with the original, needs more refactoring.
* `derivative` — is unmaintained and has no updated version, despite introducing warnings in the generated code. There is no direct replacement.
Adds `Oracle GraalVM` configuration for some backend jobs. `Oracle GraalVM` jobs run only on Linux so far. The old jobs use `GraalVM CE`.
### Important Notes
- The JDK to download and use is deduced from the `JAVA_VENDOR` environment variable. By default, `GraalVM CE` is used.
- sbt can be started with both GraalVM CE and Oracle GraalVM without any warnings.
- If you try to start sbt with JDK from a different vendor, but with the same Java version, a warning is printed.
Current list of jobs in the `Engine CI` workflow (these jobs are visible on this PR, because they are scheduled to run on every PR):
- Engine (GraalVM CE) (linux, x86_64)
- Engine (GraalVM CE) (macos, x86_64)
- Engine (GraalVM CE) (windows, x86_64)
- **Engine (Oracle GraalVM) (linux, x86_64)**
- Scala Tests (GraalVM CE) (linux, x86_64)
- Scala Tests (GraalVM CE) (macos, x86_64)
- Scala Tests (GraalVM CE) (windows, x86_64)
- **Scala Tests (Oracle GraalVM) (linux, x86_64)**
- Standard Library Tests (GraalVM CE) (linux, x86_64)
- Standard Library Tests (GraalVM CE) (macos, x86_64)
- Standard Library Tests (GraalVM CE) (windows, x86_64)
- **Standard Library Tests (Oracle GraalVM) (linux x86_64)**
- Verify License Packages (linux, x86_64)
Benchmark Engine workflow (not visible on this PR, cannot schedule manually yet):
- Benchmark Engine (GraalVM CE)
- **Benchmark Engine (Oracle GraalVM)**
Benchmark Standard Libraries workflow (not visible on this PR, cannot schedule manually yet):
- Benchmark Standard Libraries (GraalVM CE)
- **Benchmark Standard Libraries (Oracle GraalVM)**
- Close https://github.com/enso-org/cloud-v2/issues/866
- Remove *all* references to client keys and API base URLs from the codebase.
- The app can still be built by external contributors. *However*, the cloud backend (among some other things) will be completely disabled, as the required keys and base URLs will be missing.
- Add entry to `.gitignore` to allow `*.env` files in `app/ide-desktop/lib/dashboard/`
# Important Notes
- Tested (no `.env`; `.env` with prod backend; `.pbuchu.env`) on:
- `npm run dev` in `app/ide-desktop/lib/dashboard/`
- `./run ide build`
- `./run ide2 build`
- `./run gui watch`
Removes the old GUI1 code base and reduces the Rust code footprint by removing unused code.
# Important Notes
Updates build scripts and reformats part of the codebase with the autoformatter.
After investigating some errors, I found another two missing awaits in our tests. Because those are so easy to overlook, I added a lint rule which makes failure on unhandled promise (for e2e tests only).
Also, enabled HTML reports again, with traces this time, to enable closer investigation of any failure in the future. @mwu-tow added code for uploading them in GH.
* Use glob pattern to discover stdlib tests (rather than a hardcoded list).
* Don't fail CI check immediately after failing Scala test.
* Remove meta test suite tests.
# Important Notes
The meta test suite tests are removed following the discussion with @radeusgd. In short, these were failing anyway and were supposed to be rewritten (probably using a different technology, like JUnit). The current code will be a useful reference but it doesn't have to be kept on a repository head. The relevant information and references shall be added to the task.
This has been observed to be the most random error-prone part of the Rust build scripts.
This adds several retries to the patching of the artifact size (which finalizes the upload).
Additional diagnostics was added, so we observe if the retries are actually helping, so we can better understand the issue if this is not enough to fix it.
This PR adds a native aarch64 target to our release process.
It also includes refactoring of workflow generation and minor tweaks:
* removing some workarounds in the generated action code that are not needed anymore;
* some version bumps that are harmless;
* release builds have cleaning enabled unconditionally.
Now the `clean` CI steps are run always for benchmarking jobs. We run the full `./run git-clean` before and after benchmarks. Benchmarks take long enough to make any savings by not cleaning negligible.
### Important Notes
This PR brings partial refactoring in the workflow generating code which was very dirty. I'll build on this further soon when adding proper aarch64 macOS support.
Also, some minor tweaks to the generation were made:
* not writing `always() &&` twice;
* run only the latter cleaning step for canceled jobs.
Replace our port-finding code with `portpicker` crate.
We expect that it'll greatly reduce possibility of race conditions, as the port will be picked at random, so they won't collide as easily when we use the routine more than once.
Adds these JAR modules to the `component` directory inside Engine distribution:
- `graal-language-23.1.0`
- `org.bouncycastle.*` - these need to be added for graalpy language
# Important Notes
- Remove `org.bouncycastle.*` packages from `runtime.jar` fat jar.
- Make sure that the `./run` script preinstalls GraalPy standalone distribution before starting engine tests
- Note that using `python -m venv` is only possible from standalone distribution, we cannot distribute `graalpython-launcher`.
- Make sure that installation of `numpy` and its polyglot execution example works.
- Convert `Text` to `TruffleString` before passing to GraalPy - 8ee9a2816f
Upgrade to GraalVM JDK 21.
```
> java -version
openjdk version "21" 2023-09-19
OpenJDK Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15)
OpenJDK 64-Bit Server VM GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15, mixed mode, sharing)
```
With SDKMan, download with `sdk install java 21-graalce`.
# Important Notes
- After this PR, one can theoretically run enso with any JRE with version at least 21.
- Removed `sbt bootstrap` hack and all the other build time related hacks related to the handling of GraalVM distribution.
- `project-manager` remains backward compatible - it can open older engines with runtimes. New engines now do no longer require a separate runtime to be downloaded.
- sbt does not support compilation of `module-info.java` files in mixed projects - https://github.com/sbt/sbt/issues/3368
- Which means that we can have `module-info.java` files only for Java-only projects.
- Anyway, we need just a single `module-info.class` in the resulting `runtime.jar` fat jar.
- `runtime.jar` is assembled in `runtime-with-instruments` with a custom merge strategy (`sbt-assembly` plugin). Caching is disabled for custom merge strategies, which means that re-assembly of `runtime.jar` will be more frequent.
- Engine distribution contains multiple JAR archives (modules) in `component` directory, along with `runner/runner.jar` that is hidden inside a nested directory.
- The new entry point to the engine runner is [EngineRunnerBootLoader](https://github.com/enso-org/enso/pull/7991/files#diff-9ab172d0566c18456472aeb95c4345f47e2db3965e77e29c11694d3a9333a2aa) that contains a custom ClassLoader - to make sure that everything that does not have to be loaded from a module is loaded from `runner.jar`, which is not a module.
- The new command line for launching the engine runner is in [distribution/bin/enso](https://github.com/enso-org/enso/pull/7991/files#diff-0b66983403b2c329febc7381cd23d45871d4d555ce98dd040d4d1e879c8f3725)
- [Newest version of Frgaal](https://repo1.maven.org/maven2/org/frgaal/compiler/20.0.1/) (20.0.1) does not recognize `--source 21` option, only `--source 20`.
This PR adds support for the new Vue-based GUI (aka `gui2`).
The user-facing changes are primarily:
* support for `./run gui2` and `./run ide2` commands (that build just the new GUI and the whole IDE package with new GUI embedded — respectively);
* the top-level `test` and `lint` commands will now invoke the relevant commands on the new GUI
---------
Co-authored-by: Paweł Grabarz <frizi09@gmail.com>
# Important Notes
- Binary LS endpoint is not yet handled.
- The parsing of provided source is not entirely correct, as each line (including imports) is treated as node. The usage of actual enso AST for nodes is not yet implemented.
- Modifications to the graph state are not yet synchronized back to the language server.
Use the new Enso Font; also change the anti-aliasing logic to be based on device pixel ratio, rather than platform. This will improve the clarity of font rendering on Windows/Linux machines with high pixel densities.
Design reference:
Tested on various combinations of DPR/platform:
OS X, `devicePixelRatio` = 2 (should look similar to how we were already rendering *mplus1* on OS X):
<img width="1440" alt="Screenshot 2023-08-07 at 5 46 11 PM" src="https://github.com/enso-org/enso/assets/1047859/2fdf251a-ba5e-426f-b6c4-194347a9cee4">
Windows, `devicePixelRatio` = 1.25 (should look similar to how we were already rendering *mplus1* on this platform/DPR):
Linux, `devicePixelRatio` = 1 (should look similar to how we were already rendering *mplus1* on this platform/DPR):
# Important Notes
Style changes:
- Use the Enso Font for code in Rust, replacing the DejaVu fonts.
- Use the Enso Font in HTML: code in documentation, and error visualizations.
- Change SpanWidgets from Bold to Extra Bold, to match the design.
Implementation improvements:
- The new font download is cached (and Github-authenticated); this should eliminate a "rate limit" build failure I've
encountered in the past.
- Clean up DocSection HTML rendering a bit.
- Remove a CSS file that seems to have been superseded.
This PR consists of two primary changes:
1. I've replaced `react-hot-toast` with `react-toastify` library. Both serve the same purpose — sending popup notifications (so-called "toasts"). However, the latter comes with a richer feature set that matches our requirements much better.
2. I've exposed the relevant API surface to the Rust. Now Rust code can easily send notifications.
### Important Notes
At this point, no attempt at customizing style of notifications was made (other than selecting the "light" theme).
Likely we should consider this soon after integration as a separate task.
Follow-up of recent GraalVM update #7176 that fixes downloading of GraalVM for Mac - instead of "darwin", the releases are now named "macos"
# Important Notes
Also re-enables the JDK/GraalVM version check as onLoad hook to the `sbt` process. We used to have that check a long time ago. Provides errors like this one if the `sbt` is run with a different JVM version:
```
[error] GraalVM version mismatch - you are running Oracle GraalVM 20.0.1+9.1 but GraalVM 17.0.7 is expected.
[error] GraalVM version check failed.
```
- Previous GraalVM update: https://github.com/enso-org/enso/pull/6750
Removed warnings:
- Remove deprecated `ConditionProfile.createCountingProfile()`.
- Add `@Shared` to some `@Cached` parameters (Truffle now emits warnings about potential `@Share` usage).
- Specialization method names should not start with execute
- Add limit attribute to some specialization methods
- Add `@NeverDefault` for some cached initializer expressions
- Add `@Idempotent` or `@NonIdempotent` where appropriate
BigInteger and potential Node inlining are tracked in follow-up issues.
# Important Notes
For `SDKMan` users:
```
sdk install java 17.0.7-graalce
sdk use java 17.0.7-graalce
```
For other users - download link can be found at https://github.com/graalvm/graalvm-ce-builds/releases/tag/jdk-17.0.7
Release notes: https://www.graalvm.org/release-notes/JDK_17/
R component was dropped from the release 23.0.0, only `python` is available to install via `gu install python`.
