Add tests for coalesce and fix the In-memory version
# Important Notes
The distribution/lib/Standard/Database/0.0.0-dev/src/Internal/IR/From_Spec.enso change isn't actually needed for this MR as I switched away from using literal tables for these tests as trying to mix 2 literal tables doesn't look to be supported. The change I added will let any future developers know that.
- Trying to debug #11145, I'm adding logging of when token is refreshed and when that fails, with possibly some details that are included in the response.
- Also, the expired case is now narrowed down from 400-499 to just 401 as that's what [OAuth spec](https://www.rfc-editor.org/rfc/rfc6750.html#section-3.1) says is the correct status code for expired token. Other errors are reported as unexpected error, with more details.
- Allow Interrupted Exceptions to float out of the web requests.
- Use `Type_Error` rather than Any when catching auto scoping resolving.
- Rename `Java_Exception` to `JException`
* Use moduleDependencies instead of modulePath
* Fix compilation of editions
* Fix compilation of distribution-manager
* polyglot-api needs to explicitly compile module-info
* Fix compilationOrder in library-manager and edition-updater
* engine-runner-common is module
* JPMSPlugin provides default implementation of compileModuleInfo
* Remove unused setting key from JPMSUtils.compileModuleInfo
* JPMSPlugin has internalModuleDependencies and exportedModule tasks.
* Use BuildVersion instead of buildInfo
* Manual compilation of module-info.java is reported as warning
* Define org.enso.scalalibs.wrapper meta project.
* Fix module check in JPMSPlugin.
This is a fix for projects that declare `Compile /exportJars := true`
* version-output is a module
* ydoc-server uses internalModuleDependencies
* persistance is module
* engine-common uses internalModuleDependencies
* polyglot-api does not override compileModuleInfo task
* runtime-parser uses internalModuleDependencies
* edition-updater is module
* Update moduleDependencies for distribution-manager
* editions is module
* Fix some dependencies of modules
* scala-yaml is a module
* Add scala-compiler to scala-libs-wrapper
* cli depends on scala-library module
* Add dependencies for distribution-manager module
* Add some scala-library dependencies in some modules
* engine-runner uses internalModuleDependencies
* Fix module dependencies of library-manager
* Rename org.enso.scalalibs.wrapper to org.enso.scala.wrapper
* Add jsoniter-scala-macros to org.enso.scala.wrapper fat module
* Fix dependencies of some projects
* polyglot-api does not depend on truffle-api
* Fix dependencies of some projects
* runtime does not use com.google.common
* runtime is a module
* text-buffer is a module
* refactoring-utils is a module
* runtime-compiler is a module
* runtime-instrument-common is a module
* connected-lock-manager is a module
* JPMSUtils reports project name in some error messages
* Modularize some instruments
* module-info compilation is cached
* runtime-instrument-runtime-server is module
* runtime-language-epb is module
* Remove runtime-fat-jar
* engine-runner is not a fat jar
* JPMSPlugin defines exportedModuleBin task
* Redefine componentModulesPaths task
* interpreter-dsl is module
* Redefine componentModulesPaths task
* fmt sbt
* scala-libs-wrapper is a modular fat jar
* Add some module deps to org.enso.runtime
* engine-runner is not a fat jar
* Rename package in logging-config
* Rename package in logging-service
* Rename package in logging-service-logback
* Fix dependencies of exportedModuleBin task
* Mixed projects have own compileJava task
this task does not compile only module-info.java but all the java sources. So that we can see errors more easily.
When only module-info.java is compiled, the only errors that we can see are that we did not include some modules on module-path.
* Fix definition of exportedModule task.
* Remove usages of non-existing buildInfo and replace it with BuildVersion
* Fix some dependencies of org.enso.runtime module
* module-info compilation is handled directly by FrgaalCompiler
* module-info compilation is forced for projects that has only Scala sources with single module-info.java
* Fix compilation of org.enso.runtime
* manual module-info compilation is not a warning
* Rename packages in logging-utils-akka
* Create org.enso.language.server.deps.wrapper module
* language-server is module
* Creat akka-wrapper modular fat jar
* fmt
* Define common settings for modularFatJarWrapper
* Fix compilation of json-rpc-server
* Use akka and zio wrappers
* language-server depends on org.eclipse.jgit
* Fix some dependencies - update library manifests works now!
* update library manifests invokes runner directly
* buildEngineDistribution does not copy runner.jar
* Remove EngineRunnerBootLoader
* Fix compilation of std libs
* --patch-module and --add-exports are also passed to javac
* Rename package in runtime-integration-tests.
The package name org.enso.compiler clashes with the package from the module
* Remove usage of buildInfo
* FrgaalJavaCompiler can deal with non-existing module-info.java when shouldCompileModuleInfo is true.
It just generates a warning in such case as it suggests that there is something wrong with the project configuration.
* Revert AliasAnalysisTest.scala
* Fix dependencies and java cmdline options for runtime-integration-tests
* Rename test package
* runtime-integration-test depends on logging-service-logback/Test/compile
* Rename package in logging-service-logback/Test
* Fix FrgaalJavaCompiler creation for projects
* Sanitize Test/javaOptions arguments
* Sanitize Test/javaOptions arguments
* All the JPMSPlugin settings are scoped
* Remove unused sbt tasks
* modularFatJarWrapperSettings do not override javacOptions
* Resolve issue "Cannot find TestLoggerProvider" in runtime-integration-tests
* org.enso.runtime module is open
* Test that test classes are unconditionally opened and exported
* polyglot-api-macros is a module
* JPMSPlugin handles --add-opens cmdline option
* RuntimeServerTest ensures instruments are initialized
* Add some exports to org.enso.runtime.compiler
* Add instruments on module-path to runtime-integration-tests
* Replace TestLogProviderOnClassPath with TestLogProviderOnModulePath
* Replace buildInfo with BuildVersion
* Add jpms-wrapper-scalatest
* ReportLogsOnFailure is in non-modular testkit project
* Add necessary dependencies to testkit project
* Revert "Add jpms-wrapper-scalatest"
This reverts commit 732b3427a2.
* modularize filewatcher and wrap its dependencies
* Initial fix for language-server/test
* frgaal compiler setting are scoped for Compile and Test
* Rename package in language-server/test
* Exclude com.sun.jna from wrapper jars
* Rename package in library-manager-test
* testkit is an automatic module
* process-utils is module
* akka-wrapper contains akka-http
* Some fixes for library-manager-test
* Fix dependencies for akka-wrapper
* scala-libs-wrapper exports shapeless
* lang server deps wrapper exports pureconfig
* json-rpc-server requires org.slf4j
* Add some dependencies
* lang server deps wrapper exports pureconfig.generic
* language server test requires bouncycastle provider
* language server depends on cli
* directory-watcher wrapper requires org.slf4j
* WatcherAdapter logs unsuccessful initialization errors
* Fix error reporting in WatcherAdapter
* Fix rest of the language-server tests
* language-server-deps-wrapper depends on scala-libs-wrapper
* Fix rest of the language-server tests
* Missing module-info.class in an internal project is a warning, not an error
* Rename jpms-methvin-directory-watcher-wrapper to a simpler name
* compileOrder has to be specified before libraryDependencies
* exclude module-info.java from polyglot-api-macros
* Remove temporary logging in customFrgaalCompilerSettings
* Fix compilation of logging-service-logback
* Fix compilation of runtime-benchmarks
* Fix runtime-benchmarks/run
* HostClassLoader delegates to org.graalvm.polyglot class loader if org.enso.runtime is not on boot layer
* org.enso.runtime.lnaguage.epb module must be opened to allow it to be used by annnotation processor
* fmt
* Fix afetr merge
* Add module deps after merge
* Print stack trace of the uncaught exception from the annotation processor
* Remove akka-actor-typed from akka-wrapper
* runtime-instrument-common depends on slf4j
* Fix module-path for runtime-instrument-repl-debugger
* runtime-benchmarks depends on runtime-language-arrow
* --module-path is passed directly to frgaal
* Fix some module-related cmd line options for std-benchmarks
* Revert "--module-path is passed directly to frgaal"
This reverts commit da63f66a0e.
* Avoid closing of System.err when closing Context
* Avoid processing altogether when requested annotations are empty
* Pass shouldNotLimitModules opt to frgaal
* Pass module-path and add-modules options with -J prefix to frgaal
* BenchProcessor annotation processor creates its own truffle module layer
* bench-processor and benchmarks-common are modules
* fmt
* Fix after mege
* Enable JMH annotation processor
* Fix compileOrder in some projects
* Insert TruffleBoundary to QualifiedName.
This is a revert
* Fix building of engine-runner native image
* Add more deps to the native image
* Force module-info compilation in instruments.
This fixes some weird sbt bug
* Don't run engine-runner/assembly from Rust build script
* Update docs of JPMSPlugin
* fmt
* runtime-benchmarks depends on benchmarks-common module
* Fix benchmark report writing
* std-benchmarks annot processing does not take settings from runtime-benchmarks
* Suppress interpreter only warning in annotation processor
* Runtime version manager does not expect runtime.jar fat jar
* fmt
* Fix module entry point
* Move some polyglot tests to runtime-integration-tests.
Also make their output silent
* pkg has no dependency on org.graalvm.truffle
* Fix compiler dependencies test
* Rename all runtime.jar in fake releases
* Add language-server with dependencies to component dir
* No module-info.class in target dir is warning not error
* language-server does not depend on netbeans lookup uitl
* Declare LanguageServerApi service provider in module-info
* connected-lock-manager-server is JPMS module
* task-progress-notifications is module
* Add fansi-wrapper module
* Fix compilation of connected-lock-manager-server
* Define correct Test/internalModuleDependencies for project-manager
* fmt
* Fix LauncherRunnerSpec - no runtime.jar
* Add fansi-wrapper to runtime-integration-tests and runtime-benchmarks
* Fix engine-runner native image build
* Use newer JNA version - fixes running of hyperd
* DRY
* scala-compiler DRY
* fmt
* More build.sbt refactoring
* Include runtime-instrument-id-execution in engine-runner native image
* TruffleBoundary for QualifiedName.toString
* Finding a needle in a haystack
🤦
* More scala-library DRY
* more mixed-java/scala goodies
* Fix compilation of syntax-rust-definition
* Test that engine-runner does not depend on language-server
* Append rather than assign `moduleDependencies`
`++=` is less error prone than `:=`. Also discovered some unnecessary
dependencies.
* Replace : with File.pathSeparator
* [WIP] Make logging in ProjectService more verbose
* language-server/test didn't start because of missing lookup and fansi modules
* Formatting
* org.enso.cli.task.notifications needs Akka and Circe to link
* project-manager/test depends on buildEngineDistribution
* [WIP] Even more verbose logging for creating projects
* [WIP] Even more verbose logging for creating projects
* Revert "[WIP] Even more verbose logging for creating projects"
This reverts commit a7067c8472.
* Revert "[WIP] Even more verbose logging for creating projects"
This reverts commit fc6f53d4f1.
* Revert "[WIP] Make logging in ProjectService more verbose"
This reverts commit 427428e142.
* All the project with JPMSPlugin has stripped artifact names
* Revert all placeholder fake release components to runtime.jar without version
* Eliminate a cross version hack
We shouldn't be specifying Scala dependencies with a Scala cross version
in the suffix.
* Address SBT lint warnings
* Revert "Eliminate a cross version hack"
This reverts commit 8861dab288.
* logging-service-logback is mixedJavaScalaProject
* fmt
* Stripped artifact name contains classifier.
This fixes tests as those were named like `artifact-tests.jar`.
* Don't use LocalProject unless really needed
* Add more logging when BenchProcessor fails
* logging-service-logback is not mixed project
* Work with java.io.File.getPath to avoid mixing slash and backslash on Windows
* Reapply "Eliminate a cross version hack"
This reverts commit edaa436ee8.
* Pass scalaBinaryVersion correctly
* Remove scala-compiler from the distribution
* Fix IllegalAccessErrors from serde
* typos
* License review
* fmt
* Move testLogProviderOnModulePath to TestJPMSConfiguration
* logging-service-logback is not a mixed project
---------
Co-authored-by: Jaroslav Tulach <jaroslav.tulach@enso.org>
Co-authored-by: Hubert Plociniczak <hubert.plociniczak@gmail.com>
- Added `to_display_text` for `S3_File` and `Enso_File`.
- Improved widget for `AWS_Credential` allowing use of Enso secrets.
- Adjust `S3.list_objects` to return `S3_File` objects, allowing easier drill down.
- Fix for merging inherited config with direct config in widgets.
- Add missing constant types to Date.Diff widget.
* More SQL tests
* Remove duplicate
* Fix another test
* Temp fix
* More temporay fixes
* More green
* Another green
* Another one
* Fix counts
* Temporary change
* Fix tests
* Auto-commit work in progress before clean build on 2024-09-06 10:53:46
* Another fix
* Green
* Specialise ensure_query_has_no_holes
* cleanup
* Cleanup
* Cleanup
* Clean
* Code review changes
- Change default to be from 0.
- Alter widgets so always shown.
- Tweaks to the Welcome screen following review with Product Team.
- Use 0 for default seed option in `Index_Sub_Range.Sample`.
- Add ALIASES for aggregate and running.
- Add more imports for `Widgets.enso` to fix extension methods not working.
Before 3.46, the SQLite parser had a limited stack, which could overflow for certain complex queries.
CTE optimizations make some of our queries much smaller, but also a little bit more deeply nested, causing the parser stack to overflow. 3.46 removes this stack limitation.
Closes#10910.
* add mutliseries support to d3
* fix types
* fix scales when multiseries
* fix tests
* fix all tests
* basic hardcoded legend working, on only one plot at a time
* basic legend
* legend populating correctly with wrong label names
* legend with tooltip for full length names
* remove unused variale
* key on all nodes
* tooltips on points and labels
* turn on/off d3 'brush' so tool tip is availible to the user
* use colourScale
* fix type problems
* use default colour
* rename variables and remove unneeded code
* fix label
* sort labels in legend
* is multiseries show legend and dont show y label
* address comments
* fix type problem
* Update distribution/lib/Standard/Visualization/0.0.0-dev/src/Scatter_Plot.enso
Co-authored-by: James Dunkerley <jdunkerley@users.noreply.github.com>
---------
Co-authored-by: James Dunkerley <jdunkerley@users.noreply.github.com>
- Linting updates.
- Add an `Examples.welcome` and adjust the start up project to use it.
- Merge all of Cass's work into the source code.
- Make example render in mono space font.
* Reduce akka logs in PM and LS
After #10905 we suddenly got a lot of (useless) logs from akka in
project-manager and language-server. After some investigation it turned
out that akka-typed had a special configuration to enable logging in
akka using slf4j.
As we don't need the whole library for that I only stole the
configuration.
* nit
* licensing
* missing license files
* Add today, now and time to Expression.
* Move running and compute into Column as that allows them to be used in expressions.
* Fix bug.
* Fix exports.
* Java fmt.
* Akka-related startup improvements
Realized that Akka Typed and Spray were not used **at all** yet there
were in the list of dependencies.
Additionally, the former would also show up in the list of Akka's
library-extensions, initialized on startup.
* Hardcode list of library extensions
Hardcoding a list of library extensions is not recommended but I don't
see any other way of disabling expensive Serialization extension, which
we don't seem to use anyway.
* fmt
* licensing update
This implements `DB_Column.with`, which uses `WITH ... AS` SQL clauses to remove duplicates in the generated SQL.
After a discussion with @radeusgd, we concluded that we will probably want a more complete CTE implementation, so this one is useful for now to deal with big queries (like `round`).
# Important Notes
Still to do in this PR:
- [x] Rename `with` to `let` (or something similar)
- [x] tests
- [x] documentation
- [x] remove `State` hack by moving query generation into a class and using a `Ref` field for scoping
Results on `round_float`:
| --- | SQL length in characters (unprettified) | SQL length in lines (prettified) |
| --- | --- | --- |
| Without CTEs | 13193 | 851 |
| With CTEs | 3644 | 187 |
Compare the SQL:
[without-ctes.sql.txt](https://github.com/user-attachments/files/16629356/without-ctes.sql.txt)
[with-ctes.sql.txt](https://github.com/user-attachments/files/16629357/with-ctes.sql.txt)
Update, with name shortening:
| --- | SQL length in characters (unprettified) | SQL length in lines (prettified) |
| --- | --- | --- |
| Without CTEs | 13193 | 853 |
| With CTEs | 2427 | 176 |
[without-cte.txt](https://github.com/user-attachments/files/16694328/without-cte.txt)
[with-cte.txt](https://github.com/user-attachments/files/16694327/with-cte.txt)
- Closes#10411
- Also a similar technique allowed to add partial support for SQLite as well.
- Only `include_nothing=True` case is still not supported as the technique used for Snowflake did not port well to SQLite and I did not find yet a satisfactory workaround (it is doable, but too ugly to justify it for now IMHO).
- Review and update the doc comments of public functions in the AWS library.
- Reorder the functions to make the order in component browser (and online docs better).
- Align some error handling.
- Fix bug with `list` on root S3.
- Hide `S3.get_object` as it's single read makes it bad for GUI use.
- Enables the `..` autoscoping style for creating Atoms in expressions.
- Add type checking to methods in columns.
- Auto wrap returns from method in expressions into a column as needed.
- Remove `Time_Period.Day` to remove confusion..
- Closes#10716 by un-marking the `Snowflake_Connection` as `private`.
- Does the same to `SQLServer_Connection`.
- Makes sure that `Snowflake_Connection` has correct visualization and enables links for it.
- Adds a fallback in `to_default_visualization_data` and in Table's `prepare_visualization` so that if `to_js_object` of a given type is malformed, we still get some kind of visualization + debug info about the underlying error.
Majority of warnings handling is now done via newly introduced nodes. Moreover, the underlying representation of warnings storage in `WithWarnings` was changed from `Warning[]` to `EnsoHashMap`.
# Important Notes
- Remove `ArrayRope`.
- Adds `Hyper_File` allowing reading a Tableau hyper file.
- Can read the schema and table list.
- Can read the structure of a table.
- Can read data into an Enso Table.
Working on compiler IR is a daunting task. I have therefore added a new system property `enso.compiler.dumpIr` that will help with that. It dumps the encountered IRs to `ir-dumps` directory in the [GraphViz](www.graphviz.org) format. More info in updated docs.
Note that all the functionality to dump IRs to `dot` files was already implemented. This PR just adds the command line option and updates docs.
# Important Notes
- `--dump-graphs` cmd line option is removed as per [Jaroslav's request](https://github.com/enso-org/enso/pull/10740#pullrequestreview-2216676140).
- To dump graphs, use `-Dgraal.Dump=Truffle:2` system property passed via `JAVA_OPTS` env var.
If you run `env JAVA_OPTS='-Denso.compiler.dumpIr=true' enso --run tmp.enso` where `tmp.enso` is, e.g.:
```
from Standard.Base import all
main = 42
```
You will then have something like:
```
$ ls ir-dumps
Standard.Base.Data.Filter_Condition.dot Standard.Base.Data.Time.dot Standard.Base.System.Advanced.dot Standard.Base.Warning.dot
Standard.Base.Data.Locale.dot Standard.Base.Enso_Cloud.Enso_File.dot Standard.Base.System.File.Advanced.dot tmp.dot
Standard.Base.Data.Numeric.dot Standard.Base.Errors.dot Standard.Base.System.File.dot
Standard.Base.Data.Numeric.Internal.dot Standard.Base.Network.HTTP.Internal.dot Standard.Base.System.File.Generic.dot
Standard.Base.Data.Text.Regex.Internal.dot Standard.Base.Runtime.dot Standard.Base.System.Internal.dot
```
You can then visualize any of these with `dot -Tsvg -O ir-dumps/tmp.dot`.
An example how that could look like is
Fixes#10609 by rewriting all our upload-related operations to rely on `DDL_Transaction` - an abstraction that handles 'transactionality' of `CREATE TABLE` statements dependent on if a given backend allows DDLs within transactions or not (if not it emulates transactionality by creating the tables outside of transaction and then dropping them on rollback).
- Close#10622
- Changes `project-manager` and `ensoup` launcher to run the engine/language-server with working directory set to the directory containing currently running project.
- If the working directory is _not_ "the directory containing currently running project", a warning is written to logs. This can happen if the raw `/bin/enso` engine runner is used in a different directory.
- In the Cloud, the `File.new` interprets relative paths as cloud paths relative to the Cloud directory containing the current project. Absolute paths are unaffected.
- Adjusted Filter_Condition removing keep/drop from basic filters.
- Fix Is_In to have selector.
- Fix for Date simple expressions.
- Add get_row to Table and DB_Table.
In a sequence of value-level operators, whitespace does not affect relative precedence. Functional operators still follow the space-precedence rules.
The "functional" operators are: `>> << |> |>> <| <<| : .`, application, and any operator containing `<-` or `->`. All other operators are considered value-level operators.
Asymmetric whitespace can still be used to form *operator sections* of value-level operators, e.g. `+2 * 3` is still equivalent to `x -> (x+2) * 3`.
Precedence of application is unchanged, so `f x+y` is still equivalent to `f (x + y)` and `f x+y * z` is still equivalent to `(f (x + y)) * z`.
Any attempt to use spacing to override value-level operator precedence will be caught by the new enso linter. Mixed spacing (for clarity) in value-operator expressions is allowed, as long as it is consistent with the precedences of the operators.
Closes#10366.
# Important Notes
Precedence warnings:
- The parser emits a warning if the whitespace in an expression is inconsistent with its effective precedence.
- A new enso linter can be run with `./run libraries lint`. It parses all `.enso` files in `distribution/lib` and `test`, and reports any errors or warnings. It can also be run on individual files: `cargo run --release --bin check_syntax -- file1 file2...` (the result may be easier to read than the `./run` output).
- The linter is also run as part of `./run lint`, so it is checked in CI.
Additional language change:
- The exponentiation operator (`^`) now has higher precedence than the multiplication class (`*`, `/`, `%`). This change did not affect any current enso files.
Library changes:
- The libraries have been updated. The new warnings were used to identify all affected code; the changes themselves have not been programmatically verified (in many cases their equivalence relies on the commutativity of string concatenation).
- Closes#9486
- All tests are succeeding or marked pending
- Created follow up tickets for things that still need to be addressed, including:
- Fixing upload / table update #10609
- Fixing `Count_Distinct` on Boolean columns #10611
- Running the tests on CI is not part of this PR - to be addressed separately
- Part of #9486
- Fixes `Table.union`, `merge` and `distinct` tests
- Replaces `distinct_on` in `Context` that was actually a Postgres specific addition leaking into the base with a more abstract `Context_Extension` mechanism.
- This allows us to implement the Snowflake-specific `DISTINCT` using `QUALIFY`.
- Related to #9486
- Batching of expression tests
- Fixing arithmetic by simplifying `%` and `/` operations
- Trying to share some more tables, sometimes improving performance sometimes not really
- Adding sorting and other fixes to tests to make them pass: Missing_Values_Spec, Filter_Spec, Map_Spec
- Fixing warnings related to materialization of Decimal->Integer, thus fixing Join_Spec.
- Related to #9486
- Ensures that even though an integer column in Snowflake is represented by `Decimal` type, if the values are small enough, they are materialized as `Integer`.
- If the values are larger, they are still read in as `Decimal`.
- Adds tests for some other `Decimal` edge cases (various precisions and scales), and for `Float`.
Fix#10503 by creating a benchmark and then speeding it up by making sure usage of `InteropLibrary` reminds in partially evaluated code and isn't hidden behind `@TruffleBoundary`.
- Related to #9486
- Fixes types in literal tables that are used throughout the tests
- Tries to makes testing faster by disabling some edge cases, trying batching some queries, re-using the main connection and trying to re-use tables more
- Implements date/time type mapping and operations for Snowflake
- Updates type mapping to correctly reflect what Snowflake does
- Disables warnings for Integer->Decimal coercion as that's too annoying and implicitly understood in Snowflake
- Allows to select a Decimal column with `..By_Type ..Integer` (only in Snowflake backend) because the Decimal column there is its 'de-facto' Integer column replacement.
- Removes `First` and `Last` from the `Standard.Base` exports.
- Enable auto-scoping for all `Index_Sub_Range` and `Text_Sub_Range`.
- Update all use of those methods to use auto-scoping.
Addresses one of two concerns of #5298 - adds support for `--jvm` argument to allow us to switch from _native image_ built Enso binary (as developed by #10126) to regular JVM based Enso execution. This change _doesn't affect production builds_. The _native executable_ continues to be only built by `engine-runner/buildNativeImage` which is tested on CI, but not in the production jobs.
