Numerous times I wasn't sure when running the IDE if I'm running the bundled engine or a development build. Usually this depends on if I launch the standalone IDE or use a development build of project-manager.
Still it's not always obvious, and making sure that your IDE is running the right engine version is very often the first step when debugging issues with e.g. engine changes not showing up properly.
Thus I thought it may be worth to add this method (currently hidden to users in component browser by marking as `PRIVATE`, one has to type it in manually):
I think it should be a helpful tool for debugging.
- Closes#11227
- Additionally, it should fix#11278 by ensuring that every scheduled message goes to the desired endpoint, by splitting each batch by endpoint.
* Add retries to HTTP Get requests
A quick solution to random network failures for GET HTTP requests.
Should reduce the number of IOExceptions that users see while fetching
data.
* Use homemade retry logic for http requests
* Add retries to whole Data.read
Previously, we added retries only to fetch HTTP_Request. That was
insufficient as intermittent errors might happen while reading body's
stream.
Enhanced our simple server's crash endpoint to allow for different kind
of failures as well as simulate random failures.
* Remove retries from Java
Increased the scope of retries in the previous commit.
* nit
* Address PR comments
* PR comments
* Remove builtin
Add tests for coalesce and fix the In-memory version
# Important Notes
The distribution/lib/Standard/Database/0.0.0-dev/src/Internal/IR/From_Spec.enso change isn't actually needed for this MR as I switched away from using literal tables for these tests as trying to mix 2 literal tables doesn't look to be supported. The change I added will let any future developers know that.
- Trying to debug #11145, I'm adding logging of when token is refreshed and when that fails, with possibly some details that are included in the response.
- Also, the expired case is now narrowed down from 400-499 to just 401 as that's what [OAuth spec](https://www.rfc-editor.org/rfc/rfc6750.html#section-3.1) says is the correct status code for expired token. Other errors are reported as unexpected error, with more details.
- Allow Interrupted Exceptions to float out of the web requests.
- Use `Type_Error` rather than Any when catching auto scoping resolving.
- Rename `Java_Exception` to `JException`
* Use moduleDependencies instead of modulePath
* Fix compilation of editions
* Fix compilation of distribution-manager
* polyglot-api needs to explicitly compile module-info
* Fix compilationOrder in library-manager and edition-updater
* engine-runner-common is module
* JPMSPlugin provides default implementation of compileModuleInfo
* Remove unused setting key from JPMSUtils.compileModuleInfo
* JPMSPlugin has internalModuleDependencies and exportedModule tasks.
* Use BuildVersion instead of buildInfo
* Manual compilation of module-info.java is reported as warning
* Define org.enso.scalalibs.wrapper meta project.
* Fix module check in JPMSPlugin.
This is a fix for projects that declare `Compile /exportJars := true`
* version-output is a module
* ydoc-server uses internalModuleDependencies
* persistance is module
* engine-common uses internalModuleDependencies
* polyglot-api does not override compileModuleInfo task
* runtime-parser uses internalModuleDependencies
* edition-updater is module
* Update moduleDependencies for distribution-manager
* editions is module
* Fix some dependencies of modules
* scala-yaml is a module
* Add scala-compiler to scala-libs-wrapper
* cli depends on scala-library module
* Add dependencies for distribution-manager module
* Add some scala-library dependencies in some modules
* engine-runner uses internalModuleDependencies
* Fix module dependencies of library-manager
* Rename org.enso.scalalibs.wrapper to org.enso.scala.wrapper
* Add jsoniter-scala-macros to org.enso.scala.wrapper fat module
* Fix dependencies of some projects
* polyglot-api does not depend on truffle-api
* Fix dependencies of some projects
* runtime does not use com.google.common
* runtime is a module
* text-buffer is a module
* refactoring-utils is a module
* runtime-compiler is a module
* runtime-instrument-common is a module
* connected-lock-manager is a module
* JPMSUtils reports project name in some error messages
* Modularize some instruments
* module-info compilation is cached
* runtime-instrument-runtime-server is module
* runtime-language-epb is module
* Remove runtime-fat-jar
* engine-runner is not a fat jar
* JPMSPlugin defines exportedModuleBin task
* Redefine componentModulesPaths task
* interpreter-dsl is module
* Redefine componentModulesPaths task
* fmt sbt
* scala-libs-wrapper is a modular fat jar
* Add some module deps to org.enso.runtime
* engine-runner is not a fat jar
* Rename package in logging-config
* Rename package in logging-service
* Rename package in logging-service-logback
* Fix dependencies of exportedModuleBin task
* Mixed projects have own compileJava task
this task does not compile only module-info.java but all the java sources. So that we can see errors more easily.
When only module-info.java is compiled, the only errors that we can see are that we did not include some modules on module-path.
* Fix definition of exportedModule task.
* Remove usages of non-existing buildInfo and replace it with BuildVersion
* Fix some dependencies of org.enso.runtime module
* module-info compilation is handled directly by FrgaalCompiler
* module-info compilation is forced for projects that has only Scala sources with single module-info.java
* Fix compilation of org.enso.runtime
* manual module-info compilation is not a warning
* Rename packages in logging-utils-akka
* Create org.enso.language.server.deps.wrapper module
* language-server is module
* Creat akka-wrapper modular fat jar
* fmt
* Define common settings for modularFatJarWrapper
* Fix compilation of json-rpc-server
* Use akka and zio wrappers
* language-server depends on org.eclipse.jgit
* Fix some dependencies - update library manifests works now!
* update library manifests invokes runner directly
* buildEngineDistribution does not copy runner.jar
* Remove EngineRunnerBootLoader
* Fix compilation of std libs
* --patch-module and --add-exports are also passed to javac
* Rename package in runtime-integration-tests.
The package name org.enso.compiler clashes with the package from the module
* Remove usage of buildInfo
* FrgaalJavaCompiler can deal with non-existing module-info.java when shouldCompileModuleInfo is true.
It just generates a warning in such case as it suggests that there is something wrong with the project configuration.
* Revert AliasAnalysisTest.scala
* Fix dependencies and java cmdline options for runtime-integration-tests
* Rename test package
* runtime-integration-test depends on logging-service-logback/Test/compile
* Rename package in logging-service-logback/Test
* Fix FrgaalJavaCompiler creation for projects
* Sanitize Test/javaOptions arguments
* Sanitize Test/javaOptions arguments
* All the JPMSPlugin settings are scoped
* Remove unused sbt tasks
* modularFatJarWrapperSettings do not override javacOptions
* Resolve issue "Cannot find TestLoggerProvider" in runtime-integration-tests
* org.enso.runtime module is open
* Test that test classes are unconditionally opened and exported
* polyglot-api-macros is a module
* JPMSPlugin handles --add-opens cmdline option
* RuntimeServerTest ensures instruments are initialized
* Add some exports to org.enso.runtime.compiler
* Add instruments on module-path to runtime-integration-tests
* Replace TestLogProviderOnClassPath with TestLogProviderOnModulePath
* Replace buildInfo with BuildVersion
* Add jpms-wrapper-scalatest
* ReportLogsOnFailure is in non-modular testkit project
* Add necessary dependencies to testkit project
* Revert "Add jpms-wrapper-scalatest"
This reverts commit 732b3427a2.
* modularize filewatcher and wrap its dependencies
* Initial fix for language-server/test
* frgaal compiler setting are scoped for Compile and Test
* Rename package in language-server/test
* Exclude com.sun.jna from wrapper jars
* Rename package in library-manager-test
* testkit is an automatic module
* process-utils is module
* akka-wrapper contains akka-http
* Some fixes for library-manager-test
* Fix dependencies for akka-wrapper
* scala-libs-wrapper exports shapeless
* lang server deps wrapper exports pureconfig
* json-rpc-server requires org.slf4j
* Add some dependencies
* lang server deps wrapper exports pureconfig.generic
* language server test requires bouncycastle provider
* language server depends on cli
* directory-watcher wrapper requires org.slf4j
* WatcherAdapter logs unsuccessful initialization errors
* Fix error reporting in WatcherAdapter
* Fix rest of the language-server tests
* language-server-deps-wrapper depends on scala-libs-wrapper
* Fix rest of the language-server tests
* Missing module-info.class in an internal project is a warning, not an error
* Rename jpms-methvin-directory-watcher-wrapper to a simpler name
* compileOrder has to be specified before libraryDependencies
* exclude module-info.java from polyglot-api-macros
* Remove temporary logging in customFrgaalCompilerSettings
* Fix compilation of logging-service-logback
* Fix compilation of runtime-benchmarks
* Fix runtime-benchmarks/run
* HostClassLoader delegates to org.graalvm.polyglot class loader if org.enso.runtime is not on boot layer
* org.enso.runtime.lnaguage.epb module must be opened to allow it to be used by annnotation processor
* fmt
* Fix afetr merge
* Add module deps after merge
* Print stack trace of the uncaught exception from the annotation processor
* Remove akka-actor-typed from akka-wrapper
* runtime-instrument-common depends on slf4j
* Fix module-path for runtime-instrument-repl-debugger
* runtime-benchmarks depends on runtime-language-arrow
* --module-path is passed directly to frgaal
* Fix some module-related cmd line options for std-benchmarks
* Revert "--module-path is passed directly to frgaal"
This reverts commit da63f66a0e.
* Avoid closing of System.err when closing Context
* Avoid processing altogether when requested annotations are empty
* Pass shouldNotLimitModules opt to frgaal
* Pass module-path and add-modules options with -J prefix to frgaal
* BenchProcessor annotation processor creates its own truffle module layer
* bench-processor and benchmarks-common are modules
* fmt
* Fix after mege
* Enable JMH annotation processor
* Fix compileOrder in some projects
* Insert TruffleBoundary to QualifiedName.
This is a revert
* Fix building of engine-runner native image
* Add more deps to the native image
* Force module-info compilation in instruments.
This fixes some weird sbt bug
* Don't run engine-runner/assembly from Rust build script
* Update docs of JPMSPlugin
* fmt
* runtime-benchmarks depends on benchmarks-common module
* Fix benchmark report writing
* std-benchmarks annot processing does not take settings from runtime-benchmarks
* Suppress interpreter only warning in annotation processor
* Runtime version manager does not expect runtime.jar fat jar
* fmt
* Fix module entry point
* Move some polyglot tests to runtime-integration-tests.
Also make their output silent
* pkg has no dependency on org.graalvm.truffle
* Fix compiler dependencies test
* Rename all runtime.jar in fake releases
* Add language-server with dependencies to component dir
* No module-info.class in target dir is warning not error
* language-server does not depend on netbeans lookup uitl
* Declare LanguageServerApi service provider in module-info
* connected-lock-manager-server is JPMS module
* task-progress-notifications is module
* Add fansi-wrapper module
* Fix compilation of connected-lock-manager-server
* Define correct Test/internalModuleDependencies for project-manager
* fmt
* Fix LauncherRunnerSpec - no runtime.jar
* Add fansi-wrapper to runtime-integration-tests and runtime-benchmarks
* Fix engine-runner native image build
* Use newer JNA version - fixes running of hyperd
* DRY
* scala-compiler DRY
* fmt
* More build.sbt refactoring
* Include runtime-instrument-id-execution in engine-runner native image
* TruffleBoundary for QualifiedName.toString
* Finding a needle in a haystack
🤦
* More scala-library DRY
* more mixed-java/scala goodies
* Fix compilation of syntax-rust-definition
* Test that engine-runner does not depend on language-server
* Append rather than assign `moduleDependencies`
`++=` is less error prone than `:=`. Also discovered some unnecessary
dependencies.
* Replace : with File.pathSeparator
* [WIP] Make logging in ProjectService more verbose
* language-server/test didn't start because of missing lookup and fansi modules
* Formatting
* org.enso.cli.task.notifications needs Akka and Circe to link
* project-manager/test depends on buildEngineDistribution
* [WIP] Even more verbose logging for creating projects
* [WIP] Even more verbose logging for creating projects
* Revert "[WIP] Even more verbose logging for creating projects"
This reverts commit a7067c8472.
* Revert "[WIP] Even more verbose logging for creating projects"
This reverts commit fc6f53d4f1.
* Revert "[WIP] Make logging in ProjectService more verbose"
This reverts commit 427428e142.
* All the project with JPMSPlugin has stripped artifact names
* Revert all placeholder fake release components to runtime.jar without version
* Eliminate a cross version hack
We shouldn't be specifying Scala dependencies with a Scala cross version
in the suffix.
* Address SBT lint warnings
* Revert "Eliminate a cross version hack"
This reverts commit 8861dab288.
* logging-service-logback is mixedJavaScalaProject
* fmt
* Stripped artifact name contains classifier.
This fixes tests as those were named like `artifact-tests.jar`.
* Don't use LocalProject unless really needed
* Add more logging when BenchProcessor fails
* logging-service-logback is not mixed project
* Work with java.io.File.getPath to avoid mixing slash and backslash on Windows
* Reapply "Eliminate a cross version hack"
This reverts commit edaa436ee8.
* Pass scalaBinaryVersion correctly
* Remove scala-compiler from the distribution
* Fix IllegalAccessErrors from serde
* typos
* License review
* fmt
* Move testLogProviderOnModulePath to TestJPMSConfiguration
* logging-service-logback is not a mixed project
---------
Co-authored-by: Jaroslav Tulach <jaroslav.tulach@enso.org>
Co-authored-by: Hubert Plociniczak <hubert.plociniczak@gmail.com>
- Added `to_display_text` for `S3_File` and `Enso_File`.
- Improved widget for `AWS_Credential` allowing use of Enso secrets.
- Adjust `S3.list_objects` to return `S3_File` objects, allowing easier drill down.
- Fix for merging inherited config with direct config in widgets.
- Add missing constant types to Date.Diff widget.
* More SQL tests
* Remove duplicate
* Fix another test
* Temp fix
* More temporay fixes
* More green
* Another green
* Another one
* Fix counts
* Temporary change
* Fix tests
* Auto-commit work in progress before clean build on 2024-09-06 10:53:46
* Another fix
* Green
* Specialise ensure_query_has_no_holes
* cleanup
* Cleanup
* Cleanup
* Clean
* Code review changes
- Change default to be from 0.
- Alter widgets so always shown.
- Tweaks to the Welcome screen following review with Product Team.
- Use 0 for default seed option in `Index_Sub_Range.Sample`.
- Add ALIASES for aggregate and running.
- Add more imports for `Widgets.enso` to fix extension methods not working.
Before 3.46, the SQLite parser had a limited stack, which could overflow for certain complex queries.
CTE optimizations make some of our queries much smaller, but also a little bit more deeply nested, causing the parser stack to overflow. 3.46 removes this stack limitation.
Closes#10910.
* add mutliseries support to d3
* fix types
* fix scales when multiseries
* fix tests
* fix all tests
* basic hardcoded legend working, on only one plot at a time
* basic legend
* legend populating correctly with wrong label names
* legend with tooltip for full length names
* remove unused variale
* key on all nodes
* tooltips on points and labels
* turn on/off d3 'brush' so tool tip is availible to the user
* use colourScale
* fix type problems
* use default colour
* rename variables and remove unneeded code
* fix label
* sort labels in legend
* is multiseries show legend and dont show y label
* address comments
* fix type problem
* Update distribution/lib/Standard/Visualization/0.0.0-dev/src/Scatter_Plot.enso
Co-authored-by: James Dunkerley <jdunkerley@users.noreply.github.com>
---------
Co-authored-by: James Dunkerley <jdunkerley@users.noreply.github.com>
- Linting updates.
- Add an `Examples.welcome` and adjust the start up project to use it.
- Merge all of Cass's work into the source code.
- Make example render in mono space font.
* Reduce akka logs in PM and LS
After #10905 we suddenly got a lot of (useless) logs from akka in
project-manager and language-server. After some investigation it turned
out that akka-typed had a special configuration to enable logging in
akka using slf4j.
As we don't need the whole library for that I only stole the
configuration.
* nit
* licensing
* missing license files
* Add today, now and time to Expression.
* Move running and compute into Column as that allows them to be used in expressions.
* Fix bug.
* Fix exports.
* Java fmt.
* Akka-related startup improvements
Realized that Akka Typed and Spray were not used **at all** yet there
were in the list of dependencies.
Additionally, the former would also show up in the list of Akka's
library-extensions, initialized on startup.
* Hardcode list of library extensions
Hardcoding a list of library extensions is not recommended but I don't
see any other way of disabling expensive Serialization extension, which
we don't seem to use anyway.
* fmt
* licensing update
This implements `DB_Column.with`, which uses `WITH ... AS` SQL clauses to remove duplicates in the generated SQL.
After a discussion with @radeusgd, we concluded that we will probably want a more complete CTE implementation, so this one is useful for now to deal with big queries (like `round`).
# Important Notes
Still to do in this PR:
- [x] Rename `with` to `let` (or something similar)
- [x] tests
- [x] documentation
- [x] remove `State` hack by moving query generation into a class and using a `Ref` field for scoping
Results on `round_float`:
| --- | SQL length in characters (unprettified) | SQL length in lines (prettified) |
| --- | --- | --- |
| Without CTEs | 13193 | 851 |
| With CTEs | 3644 | 187 |
Compare the SQL:
[without-ctes.sql.txt](https://github.com/user-attachments/files/16629356/without-ctes.sql.txt)
[with-ctes.sql.txt](https://github.com/user-attachments/files/16629357/with-ctes.sql.txt)
Update, with name shortening:
| --- | SQL length in characters (unprettified) | SQL length in lines (prettified) |
| --- | --- | --- |
| Without CTEs | 13193 | 853 |
| With CTEs | 2427 | 176 |
[without-cte.txt](https://github.com/user-attachments/files/16694328/without-cte.txt)
[with-cte.txt](https://github.com/user-attachments/files/16694327/with-cte.txt)
- Closes#10411
- Also a similar technique allowed to add partial support for SQLite as well.
- Only `include_nothing=True` case is still not supported as the technique used for Snowflake did not port well to SQLite and I did not find yet a satisfactory workaround (it is doable, but too ugly to justify it for now IMHO).
- Review and update the doc comments of public functions in the AWS library.
- Reorder the functions to make the order in component browser (and online docs better).
- Align some error handling.
- Fix bug with `list` on root S3.
- Hide `S3.get_object` as it's single read makes it bad for GUI use.
- Enables the `..` autoscoping style for creating Atoms in expressions.
- Add type checking to methods in columns.
- Auto wrap returns from method in expressions into a column as needed.
- Remove `Time_Period.Day` to remove confusion..
- Closes#10716 by un-marking the `Snowflake_Connection` as `private`.
- Does the same to `SQLServer_Connection`.
- Makes sure that `Snowflake_Connection` has correct visualization and enables links for it.
- Adds a fallback in `to_default_visualization_data` and in Table's `prepare_visualization` so that if `to_js_object` of a given type is malformed, we still get some kind of visualization + debug info about the underlying error.
Majority of warnings handling is now done via newly introduced nodes. Moreover, the underlying representation of warnings storage in `WithWarnings` was changed from `Warning[]` to `EnsoHashMap`.
# Important Notes
- Remove `ArrayRope`.
- Adds `Hyper_File` allowing reading a Tableau hyper file.
- Can read the schema and table list.
- Can read the structure of a table.
- Can read data into an Enso Table.
Working on compiler IR is a daunting task. I have therefore added a new system property `enso.compiler.dumpIr` that will help with that. It dumps the encountered IRs to `ir-dumps` directory in the [GraphViz](www.graphviz.org) format. More info in updated docs.
Note that all the functionality to dump IRs to `dot` files was already implemented. This PR just adds the command line option and updates docs.
# Important Notes
- `--dump-graphs` cmd line option is removed as per [Jaroslav's request](https://github.com/enso-org/enso/pull/10740#pullrequestreview-2216676140).
- To dump graphs, use `-Dgraal.Dump=Truffle:2` system property passed via `JAVA_OPTS` env var.
If you run `env JAVA_OPTS='-Denso.compiler.dumpIr=true' enso --run tmp.enso` where `tmp.enso` is, e.g.:
```
from Standard.Base import all
main = 42
```
You will then have something like:
```
$ ls ir-dumps
Standard.Base.Data.Filter_Condition.dot Standard.Base.Data.Time.dot Standard.Base.System.Advanced.dot Standard.Base.Warning.dot
Standard.Base.Data.Locale.dot Standard.Base.Enso_Cloud.Enso_File.dot Standard.Base.System.File.Advanced.dot tmp.dot
Standard.Base.Data.Numeric.dot Standard.Base.Errors.dot Standard.Base.System.File.dot
Standard.Base.Data.Numeric.Internal.dot Standard.Base.Network.HTTP.Internal.dot Standard.Base.System.File.Generic.dot
Standard.Base.Data.Text.Regex.Internal.dot Standard.Base.Runtime.dot Standard.Base.System.Internal.dot
```
You can then visualize any of these with `dot -Tsvg -O ir-dumps/tmp.dot`.
An example how that could look like is
Fixes#10609 by rewriting all our upload-related operations to rely on `DDL_Transaction` - an abstraction that handles 'transactionality' of `CREATE TABLE` statements dependent on if a given backend allows DDLs within transactions or not (if not it emulates transactionality by creating the tables outside of transaction and then dropping them on rollback).
- Close#10622
- Changes `project-manager` and `ensoup` launcher to run the engine/language-server with working directory set to the directory containing currently running project.
- If the working directory is _not_ "the directory containing currently running project", a warning is written to logs. This can happen if the raw `/bin/enso` engine runner is used in a different directory.
- In the Cloud, the `File.new` interprets relative paths as cloud paths relative to the Cloud directory containing the current project. Absolute paths are unaffected.
- Adjusted Filter_Condition removing keep/drop from basic filters.
- Fix Is_In to have selector.
- Fix for Date simple expressions.
- Add get_row to Table and DB_Table.
In a sequence of value-level operators, whitespace does not affect relative precedence. Functional operators still follow the space-precedence rules.
The "functional" operators are: `>> << |> |>> <| <<| : .`, application, and any operator containing `<-` or `->`. All other operators are considered value-level operators.
Asymmetric whitespace can still be used to form *operator sections* of value-level operators, e.g. `+2 * 3` is still equivalent to `x -> (x+2) * 3`.
Precedence of application is unchanged, so `f x+y` is still equivalent to `f (x + y)` and `f x+y * z` is still equivalent to `(f (x + y)) * z`.
Any attempt to use spacing to override value-level operator precedence will be caught by the new enso linter. Mixed spacing (for clarity) in value-operator expressions is allowed, as long as it is consistent with the precedences of the operators.
Closes#10366.
# Important Notes
Precedence warnings:
- The parser emits a warning if the whitespace in an expression is inconsistent with its effective precedence.
- A new enso linter can be run with `./run libraries lint`. It parses all `.enso` files in `distribution/lib` and `test`, and reports any errors or warnings. It can also be run on individual files: `cargo run --release --bin check_syntax -- file1 file2...` (the result may be easier to read than the `./run` output).
- The linter is also run as part of `./run lint`, so it is checked in CI.
Additional language change:
- The exponentiation operator (`^`) now has higher precedence than the multiplication class (`*`, `/`, `%`). This change did not affect any current enso files.
Library changes:
- The libraries have been updated. The new warnings were used to identify all affected code; the changes themselves have not been programmatically verified (in many cases their equivalence relies on the commutativity of string concatenation).
- Closes#9486
- All tests are succeeding or marked pending
- Created follow up tickets for things that still need to be addressed, including:
- Fixing upload / table update #10609
- Fixing `Count_Distinct` on Boolean columns #10611
- Running the tests on CI is not part of this PR - to be addressed separately
- Part of #9486
- Fixes `Table.union`, `merge` and `distinct` tests
- Replaces `distinct_on` in `Context` that was actually a Postgres specific addition leaking into the base with a more abstract `Context_Extension` mechanism.
- This allows us to implement the Snowflake-specific `DISTINCT` using `QUALIFY`.
- Related to #9486
- Batching of expression tests
- Fixing arithmetic by simplifying `%` and `/` operations
- Trying to share some more tables, sometimes improving performance sometimes not really
- Adding sorting and other fixes to tests to make them pass: Missing_Values_Spec, Filter_Spec, Map_Spec
- Fixing warnings related to materialization of Decimal->Integer, thus fixing Join_Spec.
