close#11478
Changelog:
- add: `run release deploy-ydoc-polyglot` and `run release deploy-ydoc-nodejs` commands that build Ydoc Docker images and push them to ECR repository
- add: Ydoc native-image build for Linux PR checks
- update: CI release configuration to deploy (push to ECR repo) Ydoc Docker images during the Nightly and Release steps
Reduce the size of the runtime Docker image from `3.04GB` to `1.41GB`.
```
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
runtime latest ab76231c78c5 9 minutes ago 1.41GB
<none> <none> c8f70807b04f 4 hours ago 3.04GB
```
The image was bloated because of `RUN chmod` and `RUN chown` commands. When the `RUN` command modifies the file, it copies it to a new layer, resulting in a final image with multiple layers containing the same files.
# Important Notes
Note that copying directories with `COPY --chmod=5xx` sets the executable flag to both files and directories. While it is suboptimal (we only need the executable attribute for directories), having executable files in `/opt/enso` should not be an issue. Permissions can be narrowed further when the new Dockerfile `COPY` syntax is stabilized.
There is no need to generate unused variables warnings or other linting for IDE and repl users. By default linting is enabled during compilation and for those use-cases it is now disabled via runtime options.
Closes#9883
When `PROFILING_FILENAME` and `PROFILING_TIME` are set, language server will collect profiling data on startup and place it under `/opt/enso/profiling/$PROFILING_NAME` where it can be fetched from.
Needed to better analyze #9789.
- Previous GraalVM update: https://github.com/enso-org/enso/pull/6750
Removed warnings:
- Remove deprecated `ConditionProfile.createCountingProfile()`.
- Add `@Shared` to some `@Cached` parameters (Truffle now emits warnings about potential `@Share` usage).
- Specialization method names should not start with execute
- Add limit attribute to some specialization methods
- Add `@NeverDefault` for some cached initializer expressions
- Add `@Idempotent` or `@NonIdempotent` where appropriate
BigInteger and potential Node inlining are tracked in follow-up issues.
# Important Notes
For `SDKMan` users:
```
sdk install java 17.0.7-graalce
sdk use java 17.0.7-graalce
```
For other users - download link can be found at https://github.com/graalvm/graalvm-ce-builds/releases/tag/jdk-17.0.7
Release notes: https://www.graalvm.org/release-notes/JDK_17/
R component was dropped from the release 23.0.0, only `python` is available to install via `gu install python`.
This PR aims to fix current issues with cloud IDE.
1. Backend image:
* bumping the system version to avoid glibc version mismatch on parser;
* explicitly installing required GraalVM components;
2. Frontend upload:
* update to follow the new file naming;
* uploading the whole shaders subtree.
The new jgit integration tries to create config file in ~/.config/jgit/config. We are creating default ensodev with explicit `-M, --no-create-home`. As long as there wasn't any strong reasoning behind that this commit changes it to fix problems with initialising jgit.
Error that this PR fixes:
```
Jan 30 11:21:47 ip-172-31-0-83.eu-west-1.compute.internal enso_runtime[3853]: [error] [2023-01-30T11:21:47.185Z] [org.eclipse.jgit.util.FS] Cannot save config file 'FileBasedConfig[/home/ensodev/.config/jgit/config]'
Jan 30 11:21:47 ip-172-31-0-83.eu-west-1.compute.internal enso_runtime[3853]: java.io.IOException: Some(Creating directories for /home/ensodev/.config/jgit failed)
Jan 30 11:21:47 ip-172-31-0-83.eu-west-1.compute.internal enso_runtime[3853]: at org.eclipse.jgit.util.FileUtils.mkdirs(FileUtils.java:413)
Jan 30 11:21:47 ip-172-31-0-83.eu-west-1.compute.internal enso_runtime[3853]: at org.eclipse.jgit.internal.storage.file.LockFile.lock(LockFile.java:140)
Jan 30 11:21:47 ip-172-31-0-83.eu-west-1.compute.internal enso_runtime[3853]: at org.eclipse.jgit.storage.file.FileBasedConfig.save(FileBasedConfig.java:184)
Jan 30 11:21:47 ip-172-31-0-83.eu-west-1.compute.internal enso_runtime[3853]: at org.eclipse.jgit.util.FS$FileStoreAttributes.saveToConfig(FS.java:761)
Jan 30 11:21:47 ip-172-31-0-83.eu-west-1.compute.internal enso_runtime[3853]: at org.eclipse.jgit.util.FS$FileStoreAttributes.lambda$5(FS.java:443)
Jan 30 11:21:47 ip-172-31-0-83.eu-west-1.compute.internal enso_runtime[3853]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Jan 30 11:21:47 ip-172-31-0-83.eu-west-1.compute.internal enso_runtime[3853]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Jan 30 11:21:47 ip-172-31-0-83.eu-west-1.compute.internal enso_runtime[3853]: at java.base/java.lang.Thread.run(Thread.java:829)
```
Upgrading to GraalVM 22.3.0.
# Important Notes
- Removed all deprecated `FrameSlot`, and replaced them with frame indexes - integers.
- Add more information to `AliasAnalysis` so that it also gathers these indexes.
- Add quick build mode option to `native-image` as default for non-release builds
- `graaljs` and `native-image` should now be downloaded via `gu` automatically, as dependencies.
- Remove `engine-runner-native` project - native image is now build straight from `engine-runner`.
- We used to have `engine-runner-native` without `sqldf` in classpath as a workaround for an internal native image bug.
- Fixed chrome inspector integration, such that it shows values of local variables both for current stack frame and caller stack frames.
- There are still many issues with the debugging in general, for example, when there is a polyglot value among local variables, a `NullPointerException` is thrown and no values are displayed.
- Removed some deprecated `native-image` options
- Remove some deprecated Truffle API method calls.
This PR replaces hard-coded `@Builtin_Method` and `@Builtin_Type` nodes in Builtins with an automated solution
that a) collects metadata from such annotations b) generates `BuiltinTypes` c) registers builtin methods with corresponding
constructors.
The main differences are:
1) The owner of the builtin method does not necessarily have to be a builtin type
2) You can now mix regular methods and builtin ones in stdlib
3) No need to keep track of builtin methods and types in various places and register them by hand (a source of many typos or omissions as it found during the process of this PR)
Related to #181497846
Benchmarks also execute within the margin of error.
### Important Notes
The PR got a bit large over time as I was moving various builtin types and finding various corner cases.
Most of the changes however are rather simple c&p from Builtins.enso to the corresponding stdlib module.
Here is the list of the most crucial updates:
- `engine/runtime/src/main/java/org/enso/interpreter/runtime/builtin/Builtins.java` - the core of the changes. We no longer register individual builtin constructors and their methods by hand. Instead, the information about those is read from 2 metadata files generated by annotation processors. When the builtin method is encountered in stdlib, we do not ignore the method. Instead we lookup it up in the list of registered functions (see `getBuiltinFunction` and `IrToTruffle`)
- `engine/runtime/src/main/java/org/enso/interpreter/runtime/callable/atom/AtomConstructor.java` has now information whether it corresponds to the builtin type or not.
- `engine/runtime/src/main/scala/org/enso/compiler/codegen/RuntimeStubsGenerator.scala` - when runtime stubs generator encounters a builtin type, based on the @Builtin_Type annotation, it looks up an existing constructor for it and registers it in the provided scope, rather than creating a new one. The scope of the constructor is also changed to the one coming from stdlib, while ensuring that synthetic methods (for fields) also get assigned correctly
- `engine/runtime/src/main/scala/org/enso/compiler/codegen/IrToTruffle.scala` - when a builtin method is encountered in stdlib we don't generate a new function node for it, instead we look it up in the list of registered builtin methods. Note that Integer and Number present a bit of a challenge because they list a whole bunch of methods that don't have a corresponding method (instead delegating to small/big integer implementations).
During the translation new atom constructors get initialized but we don't want to do it for builtins which have gone through the process earlier, hence the exception
- `lib/scala/interpreter-dsl/src/main/java/org/enso/interpreter/dsl/MethodProcessor.java` - @Builtin_Method processor not only generates the actual code fpr nodes but also collects and writes the info about them (name, class, params) to a metadata file that is read during builtins initialization
- `lib/scala/interpreter-dsl/src/main/java/org/enso/interpreter/dsl/MethodProcessor.java` - @Builtin_Method processor no longer generates only (root) nodes but also collects and writes the info about them (name, class, params) to a metadata file that is read during builtins initialization
- `lib/scala/interpreter-dsl/src/main/java/org/enso/interpreter/dsl/TypeProcessor.java` - Similar to MethodProcessor but handles @Builtin_Type annotations. It doesn't, **yet**, generate any builtin objects. It also collects the names, as present in stdlib, if any, so that we can generate the names automatically (see generated `types/ConstantsGen.java`)
- `engine/runtime/src/main/java/org/enso/interpreter/node/expression/builtin` - various classes annotated with @BuiltinType to ensure that the atom constructor is always properly registered for the builitn. Note that in order to support types fields in those, annotation takes optional `params` parameter (comma separated).
- `engine/runtime/src/bench/scala/org/enso/interpreter/bench/fixtures/semantic/AtomFixtures.scala` - drop manual creation of test list which seemed to be a relict of the old design
