- Closes#9599
- Implemented API for sending audit logs to the cloud on a background thread.
- If the Postgres connection is opened through a datalink, its internal JDBC connection is replaced by a wrapper that reports executed queries to the audit log.
- Also introduces `EnsoMeta` - a helper Java class that can be used in our helper libraries to access Enso types.
- I have replaced the common pattern scattered throughout the codebase with calls to this 'library' to avoid repetitive code.
- Refactored `Table.display` to share code between in-memory and DB - it was needed as the function stopped working for `DB_Table` after adding making the `Table` constructor `private`.
- Clearer error when reading a SQLite database from a remote file (tells the user to download it first).
- Follow up - correlate asset id of the data link:
#9869
- Follow up - include project name (once bug is fixed):
#9875
- Some problems/improvements of the audit log:
- The audit log system is not yet ready for high throughput of logs
#9870
- The logs may be lost if `System.exit` is used
#9871
close#9656
Changelog:
- add: `ai/completion_v2` method
- add: `Visualization.AI.print` method for converting the expression to text format
- update: The default system prompt was updated to tell AI to use the `Visualization.AI.print` method for printing.
# Important Notes
The project [New_Project_1.zip](https://github.com/enso-org/enso/files/15152993/New_Project_1.zip) contains the following main file:
```py
from Standard.Base import all
from Standard.Table import all
from Standard.Database import all
from Standard.AWS import all
import Standard.Visualization
import Standard.Visualization.Warnings
from Standard.Base.Errors.Common import Dry_Run_Operation
type Student
Value id region
main =
operator70395 = 226
operator47321 = 'east'
operator76980 = Student.Value operator70395 operator47321
operator31302 = operator47321.words True
operator91574 = 1
operator34358 = operator47321.take (Index_Sub_Range.By_Index [0, operator91574])
#### METADATA ####
[[{"index":{"value":0},"size":{"value":4}},"4cf8de7f-2014-4dfd-9ceb-0164fe26c8bf"],[{"index":{"value":0},"size":{"value":29}},"389fe7a5-e59b-440a-8801-e0bd86716094"],[{"index":{"value":0},"size":{"value":558}},"a8a81ce3-199e-479e-98dd-5f919425842d"],[{"index":{"value":5},"size":{"value":8}},"c6f0de99-3bd3-459d-a69b-5fc774dd1e3f"],[{"index":{"value":5},"size":{"value":13}},"05827411-7354-478b-99d8-8370dadd560b"],[{"index":{"value":13},"size":{"value":1}},"41de94cc-ccd2-4ccf-b9db-cf686443efa0"],[{"index":{"value":14},"size":{"value":4}},"8abd24ea-b1cb-4c10-ad16-5a190f7ed0c7"],[{"index":{"value":19},"size":{"value":6}},"5c57a62d-ed82-4195-8b78-2ed660556dd0"],[{"index":{"value":26},"size":{"value":3}},"23544334-1117-4dbc-ac7f-bb3f84575264"],[{"index":{"value":30},"size":{"value":4}},"6058ba89-dc1b-45e1-b8f3-d1a4d27975f7"],[{"index":{"value":30},"size":{"value":30}},"bdd4b4cc-9c1e-4554-b0fb-3f73dcd5a590"],[{"index":{"value":35},"size":{"value":8}},"7e7bb9b4-8e9b-4700-9dcd-83c353800da5"],[{"index":{"value":35},"size":{"value":14}},"524bafb6-d3ca-4ee5-b8de-84b5c47ca87b"],[{"index":{"value":43},"size":{"value":1}},"b68b3929-8b63-49ad-977c-b04e058f8ee4"],[{"index":{"value":44},"size":{"value":5}},"7173f374-5f53-462b-ab1b-f7b1845d729b"],[{"index":{"value":50},"size":{"value":6}},"911d381e-1bdb-4e9c-826d-a00495c8fd23"],[{"index":{"value":57},"size":{"value":3}},"2f8fa1ee-9d13-4a73-99b2-bc77d1656387"],[{"index":{"value":61},"size":{"value":4}},"11abd658-4888-476c-afcd-592edaaa53c1"],[{"index":{"value":61},"size":{"value":33}},"9f6ae949-410b-48e6-bb75-84d4ba8a1989"],[{"index":{"value":66},"size":{"value":8}},"f4726e40-dd5b-49a0-aa2a-6cea6df88d5d"],[{"index":{"value":66},"size":{"value":17}},"539b4dcf-4487-49b6-9c90-6ae58d51c2db"],[{"index":{"value":74},"size":{"value":1}},"a34a41a0-5dcf-4c47-a36c-a4f427c87c03"],[{"index":{"value":75},"size":{"value":8}},"8fdf4ee7-2e74-4cac-b9b9-4230183f91c5"],[{"index":{"value":84},"size":{"value":6}},"7ff7efdc-fed6-4cb6-bed6-392d110ff991"],[{"index":{"value":91},"size":{"value":3}},"eacff407-dc6d-4f17-a28b-329f7a0582e1"],[{"index":{"value":95},"size":{"value":4}},"f2feac9f-ea02-4070-879a-f1d493473993"],[{"index":{"value":95},"size":{"value":28}},"f9b37b67-fb54-4e47-8281-e922edb4c5d3"],[{"index":{"value":100},"size":{"value":8}},"6cf398f8-d0c9-42fa-a6e6-e740de11562f"],[{"index":{"value":100},"size":{"value":12}},"28e4d0e3-7da4-4ee2-94c7-d9a7c692f8de"],[{"index":{"value":108},"size":{"value":1}},"955e8788-816c-4eb7-b5ec-020b4863536a"],[{"index":{"value":109},"size":{"value":3}},"bdc65807-0438-4296-8b1f-1fab3a885c17"],[{"index":{"value":113},"size":{"value":6}},"fe3dd265-1855-4d66-8a2d-51153b21e699"],[{"index":{"value":120},"size":{"value":3}},"3b7a14ac-3224-4b9a-b474-f6136a658f8b"],[{"index":{"value":124},"size":{"value":6}},"7f7f18a4-06ef-46e0-bddd-b9fe60878855"],[{"index":{"value":124},"size":{"value":29}},"d49e0fe3-a638-4d7e-ba58-12db7abad20f"],[{"index":{"value":131},"size":{"value":8}},"c9411433-5e73-4927-9175-15d3b1ccb0ef"],[{"index":{"value":131},"size":{"value":22}},"b53ad817-ea4c-491e-aaf8-49967e22aff6"],[{"index":{"value":139},"size":{"value":1}},"837a4d65-8af6-4860-ae0e-ad9172af6ad2"],[{"index":{"value":140},"size":{"value":13}},"3b662906-1887-4dd6-a0e2-a222ace135ce"],[{"index":{"value":154},"size":{"value":6}},"548ca873-544f-48c9-b7ab-6aa1e7f03d72"],[{"index":{"value":154},"size":{"value":38}},"7c45872c-22b4-4f37-bcd7-b3791408f737"],[{"index":{"value":161},"size":{"value":8}},"21152f0d-4222-48e4-9376-a2a28f8f6be6"],[{"index":{"value":161},"size":{"value":22}},"1cb71d49-a06d-4dd9-bc1b-54603b416601"],[{"index":{"value":161},"size":{"value":31}},"faa46cf2-3e90-44ed-9d9e-77b7f874b338"],[{"index":{"value":169},"size":{"value":1}},"6d5fdcd5-0ddf-4c83-aafe-a446ac5ce461"],[{"index":{"value":170},"size":{"value":13}},"830d73c2-1898-41a9-8046-22d86f159ef0"],[{"index":{"value":183},"size":{"value":1}},"ab897380-c596-4182-a2c5-b7b23cf59893"],[{"index":{"value":184},"size":{"value":8}},"29f35cf2-d79d-44d0-96d0-79b11589726c"],[{"index":{"value":193},"size":{"value":4}},"45bf8220-52f2-49f4-bbcf-9566d1f1b1c4"],[{"index":{"value":193},"size":{"value":57}},"478005ac-f0ca-497a-b298-554bd64946a9"],[{"index":{"value":198},"size":{"value":8}},"c11511ce-252c-4b84-9f61-dfa5544d4916"],[{"index":{"value":198},"size":{"value":13}},"dbae65a1-bdca-44b0-a497-7934d1e42616"],[{"index":{"value":198},"size":{"value":20}},"355392ad-8c81-4304-a6d0-2bbf3690da0c"],[{"index":{"value":198},"size":{"value":27}},"0720ccb5-ebd3-4c29-8113-0aa1f251e873"],[{"index":{"value":206},"size":{"value":1}},"f5a8d310-1f58-456e-84a3-78490f05ff13"],[{"index":{"value":207},"size":{"value":4}},"10046e78-5164-407c-adef-2ab11ce77b4c"],[{"index":{"value":211},"size":{"value":1}},"ed23da0c-9193-4eea-ab9c-fee90ba8f924"],[{"index":{"value":212},"size":{"value":6}},"7a09de10-35e2-45e0-9eb8-f79fa054b733"],[{"index":{"value":218},"size":{"value":1}},"b85a0981-9d82-4204-a03a-cd3a4720c72d"],[{"index":{"value":219},"size":{"value":6}},"1b04c19d-cdf0-4661-8f4e-d980cf12c996"],[{"index":{"value":226},"size":{"value":6}},"87d57f2c-e009-48a7-98bb-0e1612905564"],[{"index":{"value":233},"size":{"value":17}},"60fd6a1c-7ada-4546-ae20-e1f8b432b4fd"],[{"index":{"value":252},"size":{"value":4}},"9558affa-b913-4576-8ee3-6af22985e0f6"],[{"index":{"value":252},"size":{"value":32}},"8479091c-9232-49d6-b375-32f89693c389"],[{"index":{"value":257},"size":{"value":7}},"3d1f13f5-e22a-4cb6-9e75-9c717e016922"],[{"index":{"value":264},"size":{"value":1}},"aa429f07-973a-4c69-8e3f-74aa4780f85a"],[{"index":{"value":269},"size":{"value":5}},"5b304cf6-5ec4-4b23-ad0d-20e38b41cdcd"],[{"index":{"value":269},"size":{"value":15}},"e3315ef2-d0ec-43b1-b8f6-5b436054aaee"],[{"index":{"value":275},"size":{"value":2}},"bc7205c4-68bf-461e-a1ba-31400b9337ff"],[{"index":{"value":278},"size":{"value":6}},"78870c05-9117-4e24-bf77-a72e7aef9c18"],[{"index":{"value":286},"size":{"value":4}},"517c743f-c775-4031-8f87-222d0f0a365f"],[{"index":{"value":286},"size":{"value":271}},"d0d107bc-4997-41d6-9c20-32295590eaac"],[{"index":{"value":291},"size":{"value":1}},"91174930-f18b-4322-84cc-deb88637d012"],[{"index":{"value":292},"size":{"value":265}},"f09a4372-3231-4f2e-99f4-84aa751f9b60"],[{"index":{"value":297},"size":{"value":13}},"613df8c9-0a40-4c94-886f-9668c2c360c2"],[{"index":{"value":297},"size":{"value":19}},"5b88a4d4-6840-4a25-a862-8b5d0b75303d"],[{"index":{"value":311},"size":{"value":1}},"6bfe1dea-09df-4358-9613-74d8326bc680"],[{"index":{"value":313},"size":{"value":3}},"e3b7fac7-0c08-4f20-8a0c-a58f1b118097"],[{"index":{"value":321},"size":{"value":13}},"37d007fe-9cb6-4f2e-a8b7-30842eac60b2"],[{"index":{"value":321},"size":{"value":22}},"4233204a-543d-437a-a9fc-9f79853a9540"],[{"index":{"value":335},"size":{"value":1}},"6c01783b-27ba-413c-84a4-6a94a37d714a"],[{"index":{"value":337},"size":{"value":1}},"34bc865a-58b5-4c49-bf85-a8cfedaf9b54"],[{"index":{"value":337},"size":{"value":6}},"bb556514-570d-4a87-8b2a-6e7f198a975f"],[{"index":{"value":338},"size":{"value":4}},"87c2cedc-898e-43c3-be3d-24e15689c373"],[{"index":{"value":342},"size":{"value":1}},"13387ff5-ec5e-453a-86f2-deae721b5549"],[{"index":{"value":348},"size":{"value":13}},"6280a86b-5469-43f3-94d5-1e1726028a54"],[{"index":{"value":348},"size":{"value":57}},"a17c5e55-5e99-4e4f-961d-9b4072ecd6e4"],[{"index":{"value":362},"size":{"value":1}},"c0720382-9ce4-451a-a04b-f248143d2528"],[{"index":{"value":364},"size":{"value":7}},"9cd51399-d9d2-4898-9bcb-be2f540999ff"],[{"index":{"value":364},"size":{"value":13}},"b4980bfc-13cc-429f-bc43-9edfc07d2406"],[{"index":{"value":364},"size":{"value":27}},"1a3f6a8c-ab27-43dd-a494-bf413334fd9a"],[{"index":{"value":364},"size":{"value":41}},"71a97e88-9b19-4ec9-b3c8-ffd2940c4cb8"],[{"index":{"value":371},"size":{"value":1}},"99f38c61-3ea7-4b02-be71-f293fbecb7d7"],[{"index":{"value":372},"size":{"value":5}},"08820818-ddb9-408a-b913-e9382e5b6dc7"],[{"index":{"value":378},"size":{"value":13}},"2cf06b4e-af9f-492e-ac13-5723a896a508"],[{"index":{"value":392},"size":{"value":13}},"6c887279-5998-45ce-81ab-c37fdeb03144"],[{"index":{"value":410},"size":{"value":13}},"6c126d22-c018-46cc-bdae-cf8b89104173"],[{"index":{"value":410},"size":{"value":40}},"a65c1fa8-9fce-4c30-bef7-d85ff1b428eb"],[{"index":{"value":424},"size":{"value":1}},"208c049e-e221-49ce-99d7-96142f2d1b1c"],[{"index":{"value":426},"size":{"value":13}},"0adf4ddd-1402-4b96-bfb0-1917cc275063"],[{"index":{"value":426},"size":{"value":19}},"f2158ac6-2dd5-482d-9f3d-0bf4812d6d6e"],[{"index":{"value":426},"size":{"value":24}},"c7bbc3e7-1377-429e-b60a-0c2d2cd4ea74"],[{"index":{"value":439},"size":{"value":1}},"1a17a7ee-31d7-4885-84ce-c902bff4ea4a"],[{"index":{"value":440},"size":{"value":5}},"47016cac-40a0-45f6-8057-2b81edb052a5"],[{"index":{"value":446},"size":{"value":4}},"ce4fa4cd-5064-4752-9dbc-5fcf4e92138a"],[{"index":{"value":455},"size":{"value":13}},"3300001d-13ea-45f4-acbe-959813bcf85b"],[{"index":{"value":455},"size":{"value":17}},"b8f5900b-2370-4b21-8a0c-06d3bbb45068"],[{"index":{"value":469},"size":{"value":1}},"19ea3187-83d6-47d1-994c-64cf0ce9af3b"],[{"index":{"value":471},"size":{"value":1}},"9457c3f8-c878-4f01-a5c7-39336f163a28"],[{"index":{"value":477},"size":{"value":13}},"a5139627-05a8-4838-b171-02666c62c348"],[{"index":{"value":477},"size":{"value":80}},"8b5c864c-055d-4d11-a470-2679328fc131"],[{"index":{"value":491},"size":{"value":1}},"eb01b0af-06cf-4d75-81ad-d0bf7664b2d7"],[{"index":{"value":493},"size":{"value":13}},"6c042eb7-e895-473d-b456-6e0eec23e958"],[{"index":{"value":493},"size":{"value":18}},"f40730ff-ac6c-45db-8970-534f3becda2a"],[{"index":{"value":493},"size":{"value":64}},"b2df8ba8-683c-45c2-9932-8e1c970c799a"],[{"index":{"value":506},"size":{"value":1}},"2289bf31-74dd-431c-97f1-3ac179715453"],[{"index":{"value":507},"size":{"value":4}},"6db4523a-0bcd-4ba5-b6c2-7720db57d93c"],[{"index":{"value":512},"size":{"value":1}},"b6faea17-664d-470a-9641-999b6a2be5bd"],[{"index":{"value":512},"size":{"value":45}},"98aea277-bc44-4ceb-85a6-24ee453b7f72"],[{"index":{"value":513},"size":{"value":15}},"c0ae1c05-026d-4712-a571-979ce260b310"],[{"index":{"value":513},"size":{"value":24}},"502e8e4d-5105-4abe-8dba-4d7293cbe80b"],[{"index":{"value":513},"size":{"value":43}},"71cee748-378b-4817-be0e-b87bc1d3d847"],[{"index":{"value":528},"size":{"value":1}},"fcaad3ac-e080-4170-9d63-b2c01295447d"],[{"index":{"value":529},"size":{"value":8}},"a053e05a-4862-4407-abe8-2746784bdfc3"],[{"index":{"value":538},"size":{"value":1}},"9adae78d-7667-4865-83e9-b64464b13696"],[{"index":{"value":538},"size":{"value":18}},"92f153e3-da05-439e-bb4f-596ec8cfa9c8"],[{"index":{"value":539},"size":{"value":1}},"dddedc5e-415d-4f5d-ab1b-19468aa079a9"],[{"index":{"value":540},"size":{"value":1}},"4adce2e5-1e60-4a63-9ff3-5781d967b704"],[{"index":{"value":542},"size":{"value":13}},"ef06ed1e-246f-4106-a4c0-1e648b8ad5e7"],[{"index":{"value":555},"size":{"value":1}},"7bd1cc5f-20b1-4dd2-8a34-ccc40f5f091a"],[{"index":{"value":556},"size":{"value":1}},"720adbd6-24b0-49ac-b123-61cc037a9636"]]
{"ide":{"node":{"e3b7fac7-0c08-4f20-8a0c-a58f1b118097":{"position":{"vector":[-224,13]}},"bb556514-570d-4a87-8b2a-6e7f198a975f":{"position":{"vector":[115,19]}},"71a97e88-9b19-4ec9-b3c8-ffd2940c4cb8":{"position":{"vector":[-224,-51]}},"c7bbc3e7-1377-429e-b60a-0c2d2cd4ea74":{"position":{"vector":[360,19]},"visualization":{"show":false,"fullscreen":false,"width":200}},"f2158ac6-2dd5-482d-9f3d-0bf4812d6d6e":{"position":{"vector":[297,-45]},"visualization":{"show":false,"fullscreen":false,"width":96}},"9457c3f8-c878-4f01-a5c7-39336f163a28":{"position":{"vector":[640,40]},"visualization":{"show":false,"fullscreen":false,"width":200}},"b2df8ba8-683c-45c2-9932-8e1c970c799a":{"position":{"vector":[219,-80]},"visualization":{"show":true,"fullscreen":false,"width":200}},"f40730ff-ac6c-45db-8970-534f3becda2a":{"position":{"vector":[219,-48]}}},"import":{}}}
```
To test the functionality, I asked AI to show me the result of the `operator70395` variable:
1. Init protocol connection
```json
{"jsonrpc":"2.0","id":0,"method":"session/initProtocolConnection","params":{"clientId":"d8e948fd-6418-43c8-9f02-54827f09e10a"}}
```
2. Create execution context
```json
{"jsonrpc":"2.0","id":0,"method":"session/initProtocolConnection","params":{"clientId":"d8e948fd-6418-43c8-9f02-54827f09e10a"}}
```
3. Push the main method
```json
{"jsonrpc":"2.0","id":0,"method":"executionContext/push","params":{"contextId":"730a66ef-4222-46f8-8a03-d766946ab2bd","stackItem":{"methodPointer":{"module":"local.New_Project_1.Main","definedOnType":"local.New_Project_1.Main","name":"main"},"positionalArgumentsExpressions":[],"type":"ExplicitCall"}}}
```
4. Ask AI for the variable contents
```json
{"jsonrpc":"2.0","id":1,"method":"ai/completion_v2","params":{"contextId":"730a66ef-4222-46f8-8a03-d766946ab2bd","expressionId":"f09a4372-3231-4f2e-99f4-84aa751f9b60","prompt":"There is 'operator70395' variable defined in the program. What is the result of the variable 'operator70395'?"}}
```
I got the following responses:
```json
{"jsonrpc":"2.0","method":"ai/completionProgress","params":{"code":"Visualization.AI.print(operator70395)","reason":"To provide the result of 'operator70395', I need to know its current value.","visualizationId":"edfb00a3-6ce5-41e1-bb8f-ab191809114e"}}
```
```json
{"jsonrpc":"2.0","id":1,"result":{"Success":{"fn":"def get_operator70395_result():\n return operator70395","fnCall":"get_operator70395_result()"}}}
```
Closes#8836.
Atom constructors can be declared as private (project-private). project-private constructors can be called only from the same project. See the encapsulation.md docs for more info.
---------
Co-authored-by: Jaroslav Tulach <jaroslav.tulach@enso.org>
Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
Co-authored-by: Hubert Plociniczak <hubert.plociniczak@gmail.com>
Co-authored-by: Kaz Wesley <kaz@lambdaverse.org>
- Closes#9534 by printing the pending groups with pending reason
- Re-introduces original ordering of tests
- Adds a progress bar to the test suite runner in 'interactive' mode (if ANSI colors are enabled, progress bar will also be)
As reported by our users, when using the AWS SSO, our code was failing with:
```
Execution finished with an error: To use Sso related properties in the 'xyz' profile, the 'sso' service module must be on the class path.
```
This PR adds the missing JARs to fix that.
Additionally it improves the license review tool UX a bit (parts of #9122):
- sorting the report by amount of problems, so that dependencies with unresolved problems appear at the top,
- semi-automatic helper button to rename package configurations after a version bump,
- button to remove stale entries from config (files or copyrights that disappeared after update),
- button to add custom copyright notice text straight from the report UI,
- button to set a file as the license for the project (creating the `custom-license` file automatically)
- ability to filter processed projects - e.g. `openLegalReviewReport AWS` will only run on the AWS subproject - saving time processing unchanged dependencies,
- updated the license search heuristic, fixing a problem with duplicates:
- if we had dependencies `netty-http` and `netty-http2`, because of a prefix-check logic, the notices for `netty-http` would also appear again for `netty-http2`, which is not valid. I have improved the heuristic to avoid these false positives and removed them from the current report.
- WIP: button to mark a license type as reviewed (not finished in this PR).
Resolves#9607 by computing `Number.hash` by converting given number to `Float` first and then computing the hash. Also the conversion from `Float.to Decimal` is exact - done via `new BigDecimal(double)`. There is `Decimal.new` that handles the user-friendly conversion. However as a result `Decimal.from 2.1 != Decimal.new 2.1` - that's the only way to ensure consistency between hash code and conversions.
This change replaces an sqllite-backed suggestions' repo with a simple, in-memory, one.
As `completion` functionality has been implemented completely in GUI, there is no need to support it in backend, which simplifies a lot of functionality.
Closes#9650 and #9471.
# Important Notes
Loading suggestions and sending them to GUI on startup is almost instantaneous. Previously it would take ~10s just for `Standard.Base`.
- Closes#9363
- Cleans up the Cloud mock as it got a bit messy. It still implements the bare minimum to be able to test basic secret and auth handling logic 'offline' (added very simple path resolution, only handling the minimum set of cases for the tests to work).
- Adds first implementation of caching Cloud replies.
- Currently only caching the `Enso_User.current`. This is a simple one to cache because we do not expect it to ever change, so it can be safely cached for a long period of time (I chose 2h to make it still refresh from time to time while not being noticeable).
- We may try using this for caching other values in future PRs.
This PR bumps the FlatBuffers version used by the backend to `24.3.25` (the latest version as of now).
Since the newer FlatBuffers releases come with prebuilt binaries for all platforms we target, we can simplify the build process by simply downloading the required `flatc` binary from the official FlatBuffers GitHub release page. This allows us to remove the dependency on `conda`, which was the only reliable way to get the outdated `flatc`.
The `conda` setup has been removed from the CI steps and the relevant code has been removed from the build script.
The FlatBuffers version is no longer hard-coded in the Rust build script, it is inferred from the `build.sbt` definition (similar to GraalVM).
# Important Notes
This does not affect the GUI binary protocol implementation.
While I initially wanted to update it, it turned out farly non-trivial.
As there are multiple issues with the generated TS code, it was significantly refactored by hand and it is impossible to automatically update it. Work to address this problem is left as [a future task](https://github.com/enso-org/enso/issues/9658).
As the Flatbuffers binary protocol is guaranteed to be compatible between versions (unlike the generated sources), there should be no adverse effects from bumping `flatc` only on the backend side.
* min and max
* changelog
* wip
* scale approach mostly works
* Revert "scale approach mostly works"
This reverts commit 88e6073f7a.
* review
* review
* return value type
- Closes#9289
- Ensures that we can refer through `Enso_File` to files that do not _yet_ exist - preparing us for implementing the Write functionalities for `Enso_File` (#9291).
- As asked for by @hubertp who was encountering flaky test failures on CI in the Http_Spec and related ones, I'm adding retry logic to make such cases much less likely.
- I've made the test server randomly fail 50% of tests and with the retry logic the tests are still passing, so I think that should be much more robust, in practice the failure rate is much much less (I imagine <1% as most of the time these tests were working and we do a ton of requests in a single CI run).
- I move the `with_retries` method to now be `Test.with_retries` which can be used anywhere in our tests for the retry logic.
- It sleeps for 0.1s between retries. Not all kinds of tests need it, this was mostly for propagation delays in the Cloud in our tests. I was thinking if the delay should be configurable, but I think the 0.1s delay is not problematic and if our tests are sometimes failing due to high machine load, the delay could also help.
- This _does not_ add retry logic to raw HTTP operations or `Data.fetch`. We may add that later, but that needs some further design. In such case we may remove some retries from tests if they become unnecessary.
This change makes sure to close the Google Analytics client after usage. This will a) ensure that resources are released properly b) potentially fix the exception that is causing problems on some platforms
# Important Notes
After this change I no longer see in **my** logs:
```
io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue
SEVERE: *~*~*~ Previous channel ManagedChannelImpl{logId=1, target=analyticsdata.googleapis.com:443} was not shutdown properly!!! ~*~*~*
Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
at io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:102)
at io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:60)
at io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:51)
at io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:668)
at io.grpc.ForwardingChannelBuilder2.build(ForwardingChannelBuilder2.java:260)
at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:436)
at com.google.api.gax.grpc.ChannelPool.<init>(ChannelPool.java:107)
at com.google.api.gax.grpc.ChannelPool.create(ChannelPool.java:85)
at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:243)
at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:237)
at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:226)
at com.google.analytics.data.v1beta.stub.GrpcBetaAnalyticsDataStub.create(GrpcBetaAnalyticsDataStub.java:217)
at com.google.analytics.data.v1beta.stub.BetaAnalyticsDataStubSettings.createStub(BetaAnalyticsDataStubSettings.java:288)
at com.google.analytics.data.v1beta.BetaAnalyticsDataClient.<init>(BetaAnalyticsDataClient.java:376)
at com.google.analytics.data.v1beta.BetaAnalyticsDataClient.create(BetaAnalyticsDataClient.java:358)
at org.graalvm.truffle/com.oracle.truffle.host.HostMethodDesc$SingleMethod$MHBase.invokeHandle(HostMethodDesc.java:371)
```
It's important because apparently that's where it would get stuck when trying to log that message.
`Jackson_Object` supported parsing but not creating JSON from text. With this change, `Jackson_Object` is on par with `JS_Object` API and replaces the latter.
The most visible differences come from more detailed parsing exception's messages. Had to add some special cases for corner cases like `NaN` or infinity.
Closes#9473.
`42 == (Error.throw "foo")` now correctly returns an `Error` rather than False
# Important Notes
The error was in the wrong usage of the `org.enso.interpreter.dsl.AcceptsError` DSL annotation.
Move the types from `Standard.Table.Data` to `Standard.Table`.
Exceptions:
- `Standard.Table.Data.Report_Unmatched` => `Standard.Table.Constants`.
- `Standard.Table.Data.Join_Kind_Cross` => `Standard.Table.Internal.Join_Kind_Cross`.
Also removed constructor as an atom type.
- `Standard.Table.Extensions.Table_Ref` => `Standard.Table.Internal.Table_Ref`.
- `Standard.Table.Data.Type.Value_Type_Helpers` => `Standard.Table.Internal.Value_Type_Helpers`.
- `Standard.Table.Data.Type.Enso_Types` => `Standard.Table.Internal.Value_Type_Helpers`.
- `Standard.Table.Data.Type.Storage` => `Standard.Table.Internal.Storage`.
Changed all `Standard.Table` imports inside project to be project.
Favoured importing from `Standard.Table.Main` in `Standard.Database`.
Also fixed some linting in Enso_File.
- Closes#9284
- Now our tests run without the default `AWS_` config, thus ensuring that the tested setups work in a clean environment.
- After all, more complicated logic was needed for buckets access - apparently the AWS SDK only allows for some operations on buckets to happen if the client is connected to the correct region. Thus detection of bucket regions had to be implemented.
- Added `AWS_Region` widget based on autoscoping.
- Fixed `AWS_Credential.profile_names` crashing if no AWS config was found. Now it returns no profiles if not found. Added a regression test.
Make expand_to_rows work for Table and Column
Make expand_column work for Column and Row
Makes solving the book club exercise easier
# Important Notes
We decided expand_column did not make sense for Table as the resulting Table of Columns would rarely be what was wanted.
Fixes#9313
[Screencast from 2024-03-22 09-09-07.webm](https://github.com/enso-org/enso/assets/3919101/6ad86145-6882-4bde-993d-b1270f1ec06c)
# Important Notes
* This is PoC, so I didn't spend time on polishing the visuals; the design will likely change.
* I modified the shortcut handler a bit, allowing making multiple actions for same binding - the action's handler will be called in unspecified order, until one of them handle the event (i.e. not return false).
* To make it working regardless of imports, I needed to export AI module in Standard.Visualization. Moreover, needed to remove build_ai_prompt for Any, because it was causing issues - expect a bug report soon.
- Fixed the `write` format dropdown.
- Added `Text_Input` widget to path in various places.
- ICONs and Widgets in various places.
- Secret name dropdown.
* Initial connection to Snowflake via an account, username and password.
* Fix databases and schemas in Snowflake.
Add warehouses.
* Add warehouse.
Update schema dropdowns.
* Add ability to set warehouse and pass at connect.
* Fix for NPE in license review
* scalafmt
* Separate Snowflake from Database.
* Scala fmt.
* Legal Review
* Avoid using ARROW for snowflake.
* Tidy up Entity_Naming_Properties.
* Fix for separating Entity_Namimg_Properties.
* Allow some tweaking of Postgres dialect to allow snowflake to use as well.
* Working on reading Date, Time and Date Times.
* Changelog.
* Java format.
* Make Snowflake Time and TimeStamp stuff work.
Move some responsibilities to Type_Mapping.
* Make Snowflake Time and TimeStamp stuff work.
Move some responsibilities to Type_Mapping.
* fix
* Update distribution/lib/Standard/Database/0.0.0-dev/src/Connection/Connection.enso
Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
* PR comments.
* Last refactor for PR.
* Fix.
---------
Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
- Closes#9300
- Now the Enso libraries are themselves capable of refreshing the access token, thus there is no more problems if the token expires during a long running workflow.
- Adds `get_optional_field` sibling to `get_required_field` for more unified parsing of JSON responses from the Cloud.
- Adds `expected_type` that checks the type of extracted fields. This way, if the response is malformed we get a nice Enso Cloud error telling us what is wrong with the payload instead of a `Type_Error` later down the line.
- Fixes `Test.expect_panic_with` to actually catch only panics. Before it used to also handle dataflow errors - but these have `.should_fail_with` instead. We should distinguish these scenarios.
Table.from_union creates a new table when passed in a vector of tables. This is especially helpful when a grouped method is run multiple times, as it can create a unified result set.
- Fix `Excel_Workbook.sheet` and add a test.
- Add icon for `Table.row_count` and `DB_Table.row_count`.
- Make `join_kind` widget `Display.Always`.
- Add expression as an option to `Aggregate_Column`.
- Add `Simple_Calculation.Copy` to create a copy.
- Add defaults to `Simple_Expression` so less errory.
- Set period to default to day for `date_diff` allowing use in expressions.
- Add `Text_Left`, `Text_Right`, `Text_Length` and `Format` to `Simple_Expression`.
This is a first naïve implementation of Table.running that only supports count. Adding it as a scaffold for the rest of the functionality and to give us a place to agree on the API. (Which I changed slightly from the design)
# Important Notes
Only supports Statistic.Count. Other functionality to follow.
* Pass additional metadata to ~/.enso/credentials
* support old and new credentials format
* fix some issues after the review
* use domain instead of iss field
* small fixes
* rename Uri -> Url
* rename cognito -> Cognito
* remove unneeded file
---------
Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
- Adds the Excel format as one of the formats supported when creating a data link.
- The data link can choose to read the file as a workbook, or read a sheet or range from it as a table, like `Excel_Format`.
- Also updated Delimited format dialog to allow customizing the quote style.
Including arrow language in the distribution by default. Added a basic example for creating an Arrow array.
Making sure that memory layout agrees with Arrow specification (padding, continuous allocation of memory chunks).
Related to #9118.
This should unblock work on allowing serialization/deserialization to/from Parquet but I'd like to delay it to a follow up ticket as it is going to be a significant amount of specialized work.
- Adjusted `AWS_Credential` to have a `Default` and removed support for `Nothing` from functions.
- Renamed `Response_Body.to_file` to `Response_Body.write`.
- Add `write` to `Response`.
- Add `Table.get_value` and `DB_Table.get_value` allowing getting a single value from a table.
- Added `Data.download` allowing downloading from a URL to a file.
- Added text widget as input widget for `Data` methods.
- Implements the core parts of #9048
- Currently the path resolution is done by resolving each segment, one by one - requiring as many API calls as there are segments in the path.
- This should be replaced in a followup PR, once https://github.com/enso-org/cloud-v2/issues/899 is implemented.
- Added `to_table` extensions on some core types.
- Added ICON to `Any.to`.
- Added ICON to `Column.info`, `Table.info`, `DB_Column.info` and `DB_Table.info`.
- Added defaults to `Table.cross_tab` and `DB_Table.cross_tab`.
- Added `name`, `get`, `at`, `inner_xml` and `outer_xml` to `XML_Document`.
- Added constants into left hand side of simple expressions.
- Added widget to `get` and `at` on `XML_Document` and `XML_Element`. (Some bug in annotation code with Dmitry)
- Altered `get` and `at` to not allow XPath and just get direct child/attribute values.
- Added `get_xpath` to `XML_Document`.
- Renamed `get_elements_by_tag_name` to `get_descendants_by_tag_name` and added new `get_children_by_tag_name`.
- Added `child_names` and `attribute_names` to `XML_Document` and `XML_Element`.
- Closes#9120
- Reorders CI steps to do the license check last (to avoid it preventing tests from running which are more important than the license check)
- Tries to reword the warnings to be clearer
- Adds some CSS to the report to more clearly indicate which elements can be clicked.
- 4 weeks ago we discussed with @jdunkerley we may want to move this module.
- `Data` is a submodule mostly for various data-structures etc., the cloud stuff did not fit that well in there.
Simplify the `Test.Suite.run_with_filter` to accept a single filter parameter that searches for all the groups and specs that matches that filter. This filter can be a simple text provided from the command line.
# Important Notes
- Pending groups are now printed at the end of the run
- `Test.Suite.run_with_filter` is simplified to accept a single filter parameter that is either `Text` or `Nothing`. See the docs.
- Passing a filter from the command line is therefore straightforward, it is treated as a regex.
- For convenience, I have left all the `main` methods in all the test sources. I have just refactored them to accept the `filter` argument from the command line.
- For example, to run only a single spec from `Vector_Spec.enso`, invoke `enso --run test/Base_Tests/src/Data/Vector_Spec.enso "should allow vector creation with a programmatic constructor"`
- **Majority of the PR is a regex replace** of `^main =` for `main filter=Nothing =` and of `suite.run_with_filter` for `suite.run_with_filter filter`.
- **Fixed some internal engine bugs:**
- `AtomWithHole` allows to specify only one hole - https://github.com/enso-org/enso/pull/9065/files#diff-0f7bb7e85cf86a965de133aa7e6b5958ceb889bd1921c01e00d3a9ceb19626ef
- NaN keys in hash maps are handled in polyglot maps as well - c5257f6c2b78f893214ff67300893b593ea05e21..db4b3c0e9828ee79208d52e02586b24bb845b0d6
`Bump` library uses parser combinators behind the scenes which are known to be good at expressing grammars but are not performance-oriented.
This change ditches the dependency in favour of an existing Java implementation. `jsemver` implements the full specification, which is probably an overkill in our case, but proved to be an almost drop-in replacement for the previous library.
Closes#8692
# Important Notes
Peformance improvements:
- roughly 50ms compared to the previous approach (from 80ms to 20-40ms)
I don't see any time spent in the new implementation during startup so it could be potentially aggressively inlined.
Further more, we could use a facade and offer our own strip down version of semver.
- Added `regex` function to `Standard.Base` as a way to easily and cleanly make regular expressions.
- Added `expr` and `Expression.Value` to distinguish expressions from text values.
- Fixed issues with `Table.join` widget so dropdown exists.
- Needed fully qualified name.
- Added default empty text values for right column to provided text input boxes.
- Deprecate `Table.filter_by_expression` and allow `Table.filter` to take an `Expression`.
- Added `Simple_Expression` and deprecated `Column_Operation`. Changes the order so takes a column then a calculation.
- Rename `column` to `value` and `new_name` to `as` on `Table.set`.
- Rename `name_column` to `names` on `Table.cross_tab`.
- Removed `Column_Ref.Expression` in favour of using `Expression.Value`.
In order to allow clever masking, slicing, filtering and arrow backing stores...
- Adding ColumnStorage interface with the base API a storage will need.
- Refactored each of the unary operations to a new `UnaryOperation` interface which makes them responsible for deciding if they can be executed.
There are two projects transitively required by `runtime`, that have akka dependencies:
- `downloader`
- `connected-lock-manager`
This PR replaces the `akka-http` dependency in `downloader` by HttpClient from JDK, and splits `connected-lock-manager` into two projects such that there are no akka classes in `runtime.jar`.
# Important Notes
- Simplify the `downloader` project - remove akka.
- Add HTTP tests to the `downloader` project that uses our `http-test-helper` that is normally used for stdlib tests.
- It required few tweaks so that we can embed that server in a unit test.
- Split `connected-lock-manager` project into two projects - remove akka from `runtime`.
- **Native image build fixes and quality of life improvements:**
- Output of `native-image` is captured 743e167aa4
- The output will no longer be intertwined with the output from other commands on the CI.
- Arguments to the `native-image` are passed via an argument file, not via command line - ba0a69de6e
- This resolves an issue on Windows with "Command line too long", for example in https://github.com/enso-org/enso/actions/runs/7934447148/job/21665456738?pr=8953#step:8:2269
Cleaning up some of the structures in Storage before working on UnaryOperations.
- Removed some legacy code: `countMask`, `Index` and `DefaultIndex`.
- Renamed `mask` to `applyFilter` on `Column` and `Storage`.
- Renamed `Table.mask` to `Table.filter`.
Updates Google_Api version for authentication and adds Google Analytics reporting api and run_google_report method.
This is an initial method for proof of concept, with further design changes to follow.
# Important Notes
Updates google-api-client to v 2.2.0 from 1.35.2
Adds google-analytics-data v 0.44.0
Follow-up of #8890
Refactor the rest of the tests to the builder API (`Test_New`):
- `Image_Tests`
- `Geo_Tests`
- `Google_Api_Test`
- `Examples_Test`
- `AWS_Tests`
- `Meta_Test_Suite_Tests`
- `Visualization_Tests`
# Important Notes
- Unrelated: Fix NPE in `File.new "/" . name`
- ✅Linting fixes and groups.
- ✅Add `File.from that:Text` and use `File` conversions instead of taking both `File` and `Text` and calling `File.new`.
- ✅Align Unix Epoc with the UTC timezone and add converting from long value to `Date_Time` using it.
- ❌Add simple first logging API allowing writing to log messages from Enso.
- ✅Fix minor style issue where a test type had a empty constructor.
- ❌Added a `long` based array builder.
- Added `File_By_Line` to read a file line by line.
- Added "fast" JSON parser based off Jackson.
- ✅Altered range `to_vector` to be a proxy Vector.
- ✅Added `at` and `get` to `Database.Column`.
- ✅Added `get` to `Table.Column`.
- ✅Added ability to expand `Vector`, `Array` `Range`, `Date_Range` to columns.
- ✅Altered so `expand_to_column` default column name will be the same as the input column (i.e. no `Value` suffix).
- ✅Added ability to expand `Map`, `JS_Object` and `Jackson_Object` to rows with two columns coming out (and extra key column).
- ✅ Fixed bug where couldn't use integer index to expand to rows.
Refactor `Base_Tests` to `Test_New` testing framework. Mostly automatic text replacements.
# Important Notes
List of changes that were not done automatically (not via automatic text replacement):
- Fix indexes in Instrumentor_Spec - f590c4a398
- If group or spec is pending, its block is not evaluated - 8d797f1a4a
- Spec_Result is not private - 5767535af2
Tests marked as *pending*:
- #8913
- #8910
- Closes#8808 - adds tests for various scenarios.
- Implements `size` using HEAD.
- Updates existing functions to changes in Cloud API.
- Adds stubs for `*_time` methods, `parent`, `path`.
- [x] TODO: resolve the `Enso_File.current_working_directory` from an environment variable.
- ~~TODO: recursive directory deletion?~~ left for later
# Important Notes
- Currently, the Cloud API does not offer an easy way to extract metadata for a file, in particular to get the parent folder from the file `id`.
- We should be able to get the parent, and stuff like creation/modified time.
- We need a way to resolve paths to asset ids, for `path` to work as well as `current_working_directory`.
- What is the environment variable that will be used to feed the `current_working_directory` property?
Refactor `test/Table_Test` to the builder API. The builder API is in a new library called `Test_New` that is alongside the old `Test` library. There will be follow-up PRs that will migrate the rest of the tests. Meanwhile, let's keep these two libraries, and merge them after the last PR.
# Important Notes
- For a brief introduction into the new API, see **Prototype 1** section in https://github.com/enso-org/enso/pull/8622#issuecomment-1889706168
- When executing all the tests, the behavior should be the same as with the old library. With the only exception that if `ENSO_TEST_ANSI_COLORS` env var is set, the output is more colorful than it used to be.
Goal of this PR is to refactor the design of OrderMask and avoid copying arrays or lists wherever possible.
We have removed a few legacy functions which were not being used.
On a poor mans benchmark seems to be quicker (13s vs 16s) and memory usage should be lower.
- Closes#8723
- Adds some missing features that were needed to make this work:
- `Enso_File.create_directory` and `Enso_File.delete`, and basic tests for it
- Changes how `Enso_Secret.list` is obtained - using a different Cloud endpoint allows us to implement the desired logic, the default endpoint was giving us _all_ secrets which was not what we wanted here.
- Implements `Enso_Secret.update` and tests for it
# Important Notes
Notes describing any problems with the current Cloud API:
https://docs.google.com/document/d/1x8RUt3KkwyhlxGux7XUGfOdtFSAZV3fI9lSSqQ3XsXk/edit
Apparently, everything that was needed to make this feature work has already been implemented, although a few features needed workarounds on Enso side to work properly.
- `up_to` should have the `step` as an optional argument.
- `Date_Range` conversion can do a clever auto rename, so if a period use the name of it.
- Add `to Table` for `Range`, `Date_Range`.
Random.Seed doesn't work in the GUI and the TEXT_ONLY tag doesn't do anything so it was incorrectly showing up in the component browser.
This MR makes Random.Seed private to hide it from the GUI and completely removes the TEXT_ONLY tag which is unused and unimplemented.
Implements `Warnings.get_all wrap_errors=True` which wraps warnings attached to values inside vectors with `Map_Error`, which includes the position of the value within the vector. See [the documentation](https://github.com/enso-org/enso/blob/develop/docs/semantics/wrapped-errors.md) for more details.
`get_all wrap_errors=True` does not change the warnings that are attached to values -- it wraps them before returning them to the caller, but does not change the original warnings attached to the values.
Wrapped warnings only appear attached to the vector itself. The values inside the vector do not have their warnings wrapped.
Warning propagation is not changed at all; `Warnings.get_all` (with default `wrap_errors=False`) behaves as before. `get_all wrap_errors=True` is meant to be used primarily by the IDE, although it can be used anywhere this wrapping is desired.
This is the follow up PR addressing the last couple of points from https://github.com/enso-org/enso/pull/8643 around what the return type from fill_nothing.
# Important Notes
The biggest change is changing what we size we need for an empty string. This change says a variable length string of length 1 and does it at a low enough level that it will effect the whole language. But I think that is correct.
Give file read its own helper widget for delimiters. Remove newline add none. The file read delimiter is similar but different to the split one and so should have its own set of options.
- Closes#8555
- Refactors the file format detection logic, compacting lots of repetitive logic for HTTP handling into helper functions.
- Some updates to CODEOWNERS.
- After [suggestion](https://github.com/enso-org/enso/pull/8497#discussion_r1429543815) from @JaroslavTulach I have tried reimplementing the URL encoding using just `URLEncode` builtin util. I will see if this does not complicate other followup improvements, but most likely all should work so we should be able to get rid of the unnecessary bloat.
- Closes#8354
- Extends `simple-httpbin` with a simple mock of the Cloud API (currently it checks the token and serves the `/users` endpoint).
- Renames `simple-httpbin` to `http-test-helper`.
- Closes#8352
- ~~Proposed fix for #8493~~
- The temporary fix is deemed not viable. I will try to figure out a workaround and leave fixing #8493 to the engine team.
- Closes#8111 by making sure that all Excel workbooks are read using a backing file (which should be more memory efficient).
- If the workbook is being opened from an input stream, that stream is materialized to a `Temporary_File`.
- Adds tests fetching Table formats from HTTP.
- Extends `simple-httpbin` with ability to serve files for our tests.
- Ensures that the `Infer` option on `Excel` format also works with streams, if content-type metadata is available (e.g. from HTTP headers).
- Implements a `Temporary_File` facility that can be used to create a temporary file that is deleted once all references to the `Temporary_File` instance are GCed.
* tests
* wip
* wip
* additional warnings
* wip
* wip
* cleanup
* nested wrapping
* multiple nestings
* wraps_error uses looks_for, test for should_fail_with
* wip
* stack trace line fix
* use catch_primitive internally
* fix warning mapping, dtf spec
* just one wrapper checker, vector spec
* missing ctor, back to non-primitive catch
* back to c_p
* put old map back
* wip
* unnest tests
* Array.map on_problems
* wip
* Revert "wip"
This reverts commit c30d171457.
* better test names
* warning logging
* wip
* wip
* move logic into ALH
* doc
* constant
* My_Error.Error
* nested
* doc
* map_primtiive in warning mapper
* composition
* ref spec
* Remove warnings prior to matching on the value
If an expression has warnings and is matched we:
1) extract the warnings
2) execute the branch of a pattern that matches the value
3) attach extracted warnings to the result
This caused warnings to reappear when doing the custom warnings
manipulation.
This is also consistent with how `CaseNode`'s `doWarning` specialization
is defined.
* fix 1
* do not auto unwrap in test error checkers
* nested error matcher
* in problems too
* dtf
* v
* statistics
* wip
* Table_Spec, map_with_index_primitive
* Column_Operations_Spec
* disable warning wrapping and Report_Warning
* unimpl test
* Warnings_Spec
* DCS
* ACG JP
* zip_primitive
* join_helpers
* Lookup_Helpers
* Table
* Data_Formatter
* Value_Type_Helpers
* revert check types changes
* table_helpers
* table tests
* remove st
* do not remove warnings from value
* vec docs, tests for zip, mwi, flat_map
* docs, fixes
* remove nested_error_matcher
* cleanup
* benchmark
* one error
* alter
* add bench to main
* review
* review
* review
* tail call
* changelog
* tail call was not a tail call
* ws
* bad import
* Added missing import
* Update distribution/lib/Standard/Base/0.0.0-dev/src/Data/Array.enso
Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
* review, ref example
* lazy benchmark data
* extra paren
* check outside of catch
* review
* vector too
* actually lazy
* disambiguate Map_Error
* finish rename
* move to extensions
* combine Additional_Warnings error
* rename to map_no_wrap
* do not catch and rethrow
* review
* wip
* remove _primitives entirely
* remove unused should_fail_with function options
* remove expected_warning as function in Problems
---------
Co-authored-by: Hubert Plociniczak <hubert.plociniczak@gmail.com>
Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
* Test illustrating problems with FQNs
Inline execution fails with `Compile error: The name `Standard` could
not be found.`.
* Ensure InlineContext carries Package Repos info
Previously, there was no requirement that inline execution should allow
for FQNs. This meant that the omission of Package Repository info went
unnoticed.
In order to be able to refer to `Standard.Visualization.Preprocessor` it
has to be exported as well.
Adds these JAR modules to the `component` directory inside Engine distribution:
- `graal-language-23.1.0`
- `org.bouncycastle.*` - these need to be added for graalpy language
# Important Notes
- Remove `org.bouncycastle.*` packages from `runtime.jar` fat jar.
- Make sure that the `./run` script preinstalls GraalPy standalone distribution before starting engine tests
- Note that using `python -m venv` is only possible from standalone distribution, we cannot distribute `graalpython-launcher`.
- Make sure that installation of `numpy` and its polyglot execution example works.
- Convert `Text` to `TruffleString` before passing to GraalPy - 8ee9a2816f
- Add a `File_For_Read` type. Used for `File_Format` to read files.
- Added `Enso_User` representing the current user in `Enso_Cloud`.
- *Will be later able to list known users.*
- Added `Enso_Secret` representing a value defined in `Enso_Cloud`.
- Value not used within Enso only accessed within polyglot Java.
- Integrated into `Username_And_Password` and can be used within JDBC connections.
- Integrated into HTTP Headers so a secret can be used as a value.
- New `URI_With_Query` with the same API as `URI`. Supporting secrets in the value.
- *Will be integrated with AWS credentials.*
- Added `Enso_File` representing a file or a folder in the cloud.
- Support the same API as `File` (like the `S3_File`).
- *Will support `enso://` URI style access.*
Upgrade to GraalVM JDK 21.
```
> java -version
openjdk version "21" 2023-09-19
OpenJDK Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15)
OpenJDK 64-Bit Server VM GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15, mixed mode, sharing)
```
With SDKMan, download with `sdk install java 21-graalce`.
# Important Notes
- After this PR, one can theoretically run enso with any JRE with version at least 21.
- Removed `sbt bootstrap` hack and all the other build time related hacks related to the handling of GraalVM distribution.
- `project-manager` remains backward compatible - it can open older engines with runtimes. New engines now do no longer require a separate runtime to be downloaded.
- sbt does not support compilation of `module-info.java` files in mixed projects - https://github.com/sbt/sbt/issues/3368
- Which means that we can have `module-info.java` files only for Java-only projects.
- Anyway, we need just a single `module-info.class` in the resulting `runtime.jar` fat jar.
- `runtime.jar` is assembled in `runtime-with-instruments` with a custom merge strategy (`sbt-assembly` plugin). Caching is disabled for custom merge strategies, which means that re-assembly of `runtime.jar` will be more frequent.
- Engine distribution contains multiple JAR archives (modules) in `component` directory, along with `runner/runner.jar` that is hidden inside a nested directory.
- The new entry point to the engine runner is [EngineRunnerBootLoader](https://github.com/enso-org/enso/pull/7991/files#diff-9ab172d0566c18456472aeb95c4345f47e2db3965e77e29c11694d3a9333a2aa) that contains a custom ClassLoader - to make sure that everything that does not have to be loaded from a module is loaded from `runner.jar`, which is not a module.
- The new command line for launching the engine runner is in [distribution/bin/enso](https://github.com/enso-org/enso/pull/7991/files#diff-0b66983403b2c329febc7381cd23d45871d4d555ce98dd040d4d1e879c8f3725)
- [Newest version of Frgaal](https://repo1.maven.org/maven2/org/frgaal/compiler/20.0.1/) (20.0.1) does not recognize `--source 21` option, only `--source 20`.
- Closes#5303
- Refactors `JoinStrategy` allowing us to 'stack' join strategies on top of each other (to some extent) - currently a `HashJoin` can be followed by another join strategy (currently `SortJoin`)
- Adds benchmarks for join
- Due to limitations of the sorting approach this will still not be as fast as possible for cases where there is more than 1 `Between` condition in a single query - trying to demonstrate that in benchmarks.
- We can replace sorting by d-dimensional [RangeTrees](https://en.wikipedia.org/wiki/Range_tree) to get `O((n + m) log^d n + k)` performance (where `n` and `m` are sizes of joined tables, `d` is the amount of `Between` conditions used in the query and `k` is the result set size).
- Follow up ticket for consideration later:
#8216
- Closes#8215
- After all, it turned out that `TreeSet` was problematic (because of not enough flexibility with duplicate key handling), so the simplest solution was to immediately implement this sub-task.
- Closes#8204
- Unrelated, but I ran into this here: adds type checks to other arguments of `set`.
- Before, putting in a Column as `new_name` (i.e. mistakenly messing up the order of arguments), lead to a hard to understand `Method `if_then_else` of type Column could not be found.`, instead now it would file with type error 'expected Text got Column`.
- Sets the default limit for `Table.read` in Database to be max 1000 rows.
- The limit for in-memory compatible API still defaults to `Nothing`.
- Adds a warning if there are more rows than limit.
- Enables a few unrelated asserts.
* doc
* one test
* date tests
* empty and nothing
* ints floats
* bools
* all columns
* regex and index
* locales
* bad formats
* all with one format
* docs
* examples, not impl db
* docs, more errors
* cleanup
* changelog
* check list
* reorder
* clue
* review
* review
* review
* review
* review
* review
* specify time zone
The change upgrades `directory-watcher` library, hoping that it will fix the problem reported in #7695 (there has been a number of bug fixes in MacOS listener since then).
Once upgraded, tests in `WatcherAdapterSpec` because the logic that attempted to ensure the proper initialization order in the test using semaphore was wrong. Now starting the watcher using `watchAsync` which only returns the future when the watcher successfully registers for paths. Ideally authors of the library would make the registration bit public
(3218d68a84/core/src/main/java/io/methvin/watcher/DirectoryWatcher.java (L229C7-L229C20)) but it is the best we can do so far.
Had to adapt to the new API in PathWatcher as well, ensuring the right order of initialization.
Should fix#7695.
Added Blank_Selector constructor and applied to remove_blank_columns, select_blank_columns, filter_blank_rows for #7931 . Changed when_any to when for readability.
Previously, constant columns were given generated names with UUIDs in them, which are long and provide no information. Instead, we now use the constant value itself to form the name.
Since these new generated names are less unique, we must explicitly make them unique, in cases where the caller did not explicilty set a name.
- Closes#7981
- Adds a `RUNTIME_ERROR` operation into the DB dialect, that may be used to 'crash' a query if a condition is met - used to validate if `lookup_and_replace` invariants are still satisfied when the query is materialized.
- Removes old `Table_Helpers.is_table` and `same_backend` checks, in favour of the new way of checking this that relies on `Table.from` conversions, and is much simpler to use and also more robust.
- Fixes#8049
- Adds tests for handling of Date_Time upload/download in Postgres.
- Adds tests for edge cases of handling of Decimal and Binary types in Postgres.
- Validate spans during existing lexer and parser unit tests, and in `enso_parser_debug`.
- Fix lost span info causing failures of updated tests.
# Important Notes
- [x] Output of `parse_all_enso_files.sh` is unchanged since before #7881 (modulo libs changes since then).
- When the parser encounters an input with the first line indented, it now creates a sub-block for lines at than indent level, and emits a syntax error (every indented block must have a parent).
- When the parser encounters a number with a base but no digits (e.g. `0x`), it now emits a `Number` with `None` in the digits field rather than a 0-length digits token.
- Fixes issue with `to_display_text` on `Date_Time`. Now a reversible format.
- Adjusted the auto width code on table so it works.
- Manually sized columns should remain fixed size.
- Enable Excel range style selection in table.
- Removed page support as not implemented yet.
# Important Notes
Will need to apply to Vue based viz at some point as well.
- Follow-up of #8055
- Adds a benchmark comparing performance of Enso Map and Java HashMap in two scenarios - _only incremental_ updates (like `Vector.distinct`) and _replacing_ updates (like keeping a counter for each key). These benchmarks can be used as a metric for #8090
Having a modest-size files in a project would lead to a timeout when the project was first initialized. This became apparent when testing delivered `.enso-project` files with some data files. After some digging there was a bug in JGit
(https://bugs.eclipse.org/bugs/show_bug.cgi?id=494323) which meant that adding such files was really slow. The implemented fix is not on by default but even with `--renormalization` turned off I did not see improvement.
In the end it didn't make sense to add `data` directory to our version control, or any other files than those in `src` or some meta files in `.enso`. Not including such files eliminates first-use initialization problems.
# Important Notes
To test, pick an existing Enso project with some data files in it (> 100MB) and remove `.enso/.vcs` directory. Previously it would timeout on first try (and work in successive runs). Now it works even on the first try.
The crash:
```
[org.enso.languageserver.requesthandler.vcs.InitVcsHandler] Initialize project request [Number(2)] for [f9a7cd0d-529c-4e1d-a4fa-9dfe2ed79008] failed with: null.
java.util.concurrent.TimeoutException: null
at org.enso.languageserver.effect.ZioExec$.<clinit>(Exec.scala:134)
at org.enso.languageserver.effect.ZioExec.$anonfun$exec$3(Exec.scala:60)
at org.enso.languageserver.effect.ZioExec.$anonfun$exec$3$adapted(Exec.scala:60)
at zio.ZIO.$anonfun$foldCause$4(ZIO.scala:683)
at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:904)
at zio.internal.FiberRuntime.evaluateEffect(FiberRuntime.scala:381)
at zio.internal.FiberRuntime.evaluateMessageWhileSuspended(FiberRuntime.scala:504)
at zio.internal.FiberRuntime.drainQueueOnCurrentThread(FiberRuntime.scala:220)
at zio.internal.FiberRuntime.run(FiberRuntime.scala:139)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
```
* Reduce extra output in compilation and tests
I couldn't stand the amount of extra output that we got when compiling
a clean project and when executing regular tests. We should strive to
keep output clean and not print anything additional to stdout/stderr.
* Getting rid of explicit setup by service loading
In order for SL4J to use service loading correctly had to upgrade to
latest slf4j. Unfortunately `TestLogProvider` which essentially
delegates to `logback` provider will lead to spurious ambiguous warnings
on multiple providers. In order to dictate which one to use and
therefore eliminate the warnings we can use the `slf4j.provider` env
var, which is only available in slf4j 2.x.
Now, there is no need to explicitly call `LoggerSetup.get().setup()` as
that is being called during service setup.
* legal review
* linter
* Ensure ConsoleHandler uses the default level
ConsoleHandler's constructor uses `Level.INFO` which is unnecessary for
tests.
* report warnings
- Adds the ability to use numbers, date/time and Boolean values as constants in `set`.
- `Table.set` can take a `Column_Operation`, allowing for deriving of a new column based on other columns.
- Added `Column_Ref` type to refer to a column in `filter`.
- Added some ALIASes.
- Added `sheet` to `Excel_Workbook` to give familiar API to read sheet.
- Added conversion from range to vector allowing easy use with Zip.
- Add `Map.from_keys_and_values.
- Fixes a bug where creating a temporary table could accidentally issue a `DROP` statement of the table name that the user provided, risking destruction of user data.
- Fortunately, the bad scenario was almost impossible, because the `DROP` statement was only issued _if_ we previously checked that the mentioned table _does not exist_ - dropping a nonexistent table does not do any harm.
- It could have been dangerous in a very unlikely scenario that a table was created just between the _existence check_ and the _drop_.
- After the fix the existence check and any modifications are done within a transaction to avoid interference from concurrent modifications, and the DROP is correctly applied to a temporary Enso table instead of the original one.
- Replaced a temporary log with proper simple logging of SQL statements into a file, if an Environment variable is set.
- Used that feature to test that no unexpected statements occur.
- Closes#7749 implementing the in-memory logic.
- Additional complications have surfaced regarding the Database logic, so it has been split off into a separate ticket: #7981
We currently scan the entire child list for each call to children, num_children, and get_child_element.
Instead, we should cache the filtered child list on first access.
This PR includes
* Reading XML from a file, stream, or string
* Reading XML via Data.fetch
* Accessing the root element, element children, and attributes
* Accessing tag text contents
* Get tags by name
* Inner / Outer XML string
- Fixes#7352 by remembering original value types in type inference mode to be able to reconstruct them for Mixed.
- Added more benchmarks for comparing performance of constructing columns.
- Fixes missing implementations that caused `Table.union` crashing on some type pairs.
- Ensures that `Loss_Of_Integer_Precision` warning is not swallowed when numeric columns are unioned to create a `Float` column.
- Adds test for all of the above cases.
- Allow to output benchmark results to a CSV by setting an environment variable - useful for quickly comparing benchmarks, e.g. in Enso.
- Shuffle a few `from`s into correct places:
- `Day_Of_Week.from` removing `Day_Of_Week_From` module.
- Adding short cut for `http` and `https` in `Data.read` so it calls onto `Data.fetch` giving a single entry point.
- Moved `URI` extensions from `Standard.Base.Data` module into `Standard.Base.Network.Extensions`.
- Added `post` extension for `URI`.
- Added `contains_key` to `JS_Object`.
- Restored `into` in `JS_Object`:
- Follows old logic populating a constructor.
- Will use conversion from `JS_Object` if present.
- Added automatic deserialization of `Date`, `Time_Of_Day` and `Date_Time` from JSON.
- Uses conversion from `JS_Object`.
- Added conversion from `Text` to a `HTTP_Method` and type checking where `HTTP_Method` used in public APIs.
- Added support for `Date`, `Time_Of_Day` and `Date_Time` in `Table.from_objects`.
- Added `expand_column` to `Table` to expand `JS_Object` to values.
- Add type checking for `Table` in `right` arguments (allowing `Column`s to be used).
- Use type checking in `Table.set` to allow for conversion to a `Column`.
- Remove some unused imports.
- Fix for bug in S3 edge case.
* Add Text.substring function and get_position helper function
For #7876 adds a Text.substring function which supports negative indexes and returns a part of a string from 0-based index 'start' and continuing for 'length'
* added substring and simplified get function
For #7876 adds a Text.substring function which supports negative indexes and returns a part of a string from 0-based index 'start' and continuing for 'length'.
Also simplified get function as it looped unnecessarily.
* Update distribution/lib/Standard/Base/0.0.0-dev/src/Data/Text/Extensions.enso
punctuation corrections
Co-authored-by: GregoryTravis <greg.m.travis@gmail.com>
* Update Text_Spec.enso
Added test for start index larger than string
* Update distribution/lib/Standard/Base/0.0.0-dev/src/Data/Text/Extensions.enso
updated Arguments: section to use consistent style
Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
* Update distribution/lib/Standard/Base/0.0.0-dev/src/Data/Text/Extensions.enso
updated Index_Out_Of_Bounds error to reference cached length
Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
* Removed Slice label and added changelog entry
* Re-added slice tag to substring
Per conversation with James, added slice back to substring
* Update CHANGELOG.md add link
---------
Co-authored-by: GregoryTravis <greg.m.travis@gmail.com>
Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
