Commit Graph

837 Commits

Author SHA1 Message Date
AdRiley
c7476c10f3
Add data cleanse component (#9879)
* Signature

* More API work

* Red

* Still red

* Green

* Red

* Green

* Red

* Green

* Refactor

* Refactor

* Refactor

* Red

* Green

* Red

* Green

* Red

* Green

* More tests

* non-ascii

* Ordering tests

* Remove tabs

* Numbers and Letters

* Changelog.md

* Add documentation

* Tests for non-text columns

* Add punctuation

* Add symbols

* Fix

* Refactor

* Refactor

* Move to base

* Fix

* Start of in-db tests. Not working

* DB versions

* Update widget

* Fix widgets

* Move tests to base

* Code review changes
2024-05-20 09:33:18 +01:00
GregoryTravis
4d49b00375
Combine builders for Vector.build and Vector.new_builder (#9922)
We have decided to keep the old new_builder, and to combine the new and old builders into one builder.
2024-05-17 16:18:47 +00:00
Radosław Waśko
38ad7b0afa
Creating datalinks from code (#9957)
- Closes #9673
- Adds the ability to save an existing Postgres connection as a datalink into the Enso Cloud, automatically promoting plain-text passwords into a Secret.
- Fixes dataflow error propagation in `JS_Object.from_pairs`.
2024-05-16 13:29:41 +00:00
Radosław Waśko
a2481036e6
Prepare for the Path resolver change to GET (#9958)
- Prepares for the Cloud API change, together with a fallback for the old API to avoid problems during migration.
- This PR should be merged before the https://github.com/enso-org/cloud-v2/pull/1236 PR is _deployed_.
2024-05-15 13:00:32 +00:00
Michał Wawrzyniec Urbańczyk
7982935c2f
[CI] Fix issues with checking PRs created against repository forks (#9926)
This PR fixes several issues that were appearing when running CI jobs on PRs created against the repository forks:
* electron-builder on Windows and macOS will properly recognize that the secrets are missing and will not attempt to sign the artifacts;
* similarly, fixed the S3 library tests;
* test reporter step will be now skipped, as it does not support forks.
2024-05-14 18:38:44 +00:00
James Dunkerley
b2aeb9fc84
Moving away from Integer | Nothing to Rows_To_Read for limiting number of rows. (#9925)
- Added a new `Rows_To_Read` type with conversions from `Nothing` and integers.
- Updated `read` on `Table`, `Column`, `DB_Table` and `DB_Column`.
- Updated `Delimited_Format.Delimited` to use `Rows_To_Read` for `row_limit`.
- Updated `Excel_Format.Sheet` and `Excel_Format.Range` to use `Rows_To_Read` for `row_limit`.
- Updated `Excel_Workbook.read` to use `Rows_To_Read`.
- Updated `Connection.read` (in all connection types) to use `Rows_To_Read`.

![image](https://github.com/enso-org/enso/assets/4699705/553c027f-f4c3-4855-9f51-2c4bcaec48a0)

![image](https://github.com/enso-org/enso/assets/4699705/a06c3912-77e0-4c10-abb8-73aed667458d)
2024-05-14 16:31:26 +00:00
James Dunkerley
2d6735edc7
Allow a Table to be used to rename_columns. (#9940)
Following an example from Steve, adjusted `rename_column` to allow a `Table` to define the mapping.

- A single Text column table gives a list of new names.
- A two Text column table gives a map from old name to new name.
- If a `DB_Table` is passed errors and tells the user to materialize the table.

![image](https://github.com/enso-org/enso/assets/4699705/3ed98330-1fce-465e-bf96-4a86e0872dd3)
2024-05-14 15:57:10 +00:00
Radosław Waśko
5f0a16c87c
Audit Logs for Postgres connections opened through a data link (#9873)
- Closes #9599
- Implemented API for sending audit logs to the cloud on a background thread.
- If the Postgres connection is opened through a datalink, its internal JDBC connection is replaced by a wrapper that reports executed queries to the audit log.
- Also introduces `EnsoMeta` - a helper Java class that can be used in our helper libraries to access Enso types.
- I have replaced the common pattern scattered throughout the codebase with calls to this 'library' to avoid repetitive code.
- Refactored `Table.display` to share code between in-memory and DB - it was needed as the function stopped working for `DB_Table` after adding making the `Table` constructor `private`.
- Clearer error when reading a SQLite database from a remote file (tells the user to download it first).
- Follow up - correlate asset id of the data link:
#9869
- Follow up - include project name (once bug is fixed):
#9875
- Some problems/improvements of the audit log:
- The audit log system is not yet ready for high throughput of logs
#9870
- The logs may be lost if `System.exit` is used
#9871
2024-05-11 08:54:33 +00:00
James Dunkerley
d97754da17
Error messages for rename_columns and Vector.duplicates (#9917)
- Improve error message for `rename_columns`.
- Add `length` to `Set` and `Map`.
- Add `duplicates` to `Vector` (and `Array`).

![image](https://github.com/enso-org/enso/assets/4699705/623df253-52e8-4bdc-a69c-ac8dc3ca594e)
2024-05-10 17:43:50 +00:00
GregoryTravis
720d32cbe3
Convert most remaining uses of Vector.new_builder to Vector.build (#9891) 2024-05-09 15:52:53 +00:00
AdRiley
55be1057fa
Add missing tests (#9897)
* Add missing tests

* Fix

* Fix tests
2024-05-09 15:47:31 +01:00
AdRiley
e25ec96aaa
Add table running variance skew sd and kurtosis (#9854)
Adds support for Variance, Skew, Standard Deviation and Kurtosis to Table.Running.
2024-05-09 08:45:29 +00:00
GregoryTravis
a2c36def26
Add Vector.build_multiple and handle dataflow error and warning propagation in build and build_multiple (#9766) 2024-05-07 19:30:21 +00:00
AdRiley
15976a8505
Make table.Running return integer typed columns for min/max (#9853)
* New Tests

* Green

* Running min for longs

* Unsupported types test

* Revert

* Add support for all the integer types

* Another test
2024-05-07 10:49:12 +01:00
James Dunkerley
c0fd6eed2d
Small tweaks from Ned, Mark, Cass and Steve's feedback. (#9859)
- Rename `.info` to `.column_info`.
- Rename `.select_by_type` to `.select_columns_by_type`. (to be merged into `select_columns` functionality)
- Rename `.remove_by_type` to `.remove_columns_by_type`. (to be merged into `remove_columns` functionality)
- Adding "field" `ALIAS`es.
- Making `Table.Value` private (and hence hiding `java_table`).
- Making `Column.Value` private (and hence hiding `java_column`, added method to `Storage` to allow tests to work).
- Making `DB_Table.Value` private (and hence hiding `connection`, `context` and `internal_columns`).
- Making `DB_Column.Value` private (and hence hiding `connection`, `sql_type_reference`, `expression` and `context`).
- Add widgets for everywhere `Filter_Condition` used (fix missing entry for lower, can't use auto-scoping).
![image](https://github.com/enso-org/enso/assets/4699705/bad7d5c8-cd01-4384-a1c8-dbcd7a5ad92b)
- Fix for widgets in `Simple_Calculation`: `date_add` and `format`.
![image](https://github.com/enso-org/enso/assets/4699705/14eeef2e-5069-4a88-8b68-eb675497addf)
![image](https://github.com/enso-org/enso/assets/4699705/75d4e097-f171-4bf6-a27e-f9d396c44a92)
- Fix for nested `Row`, `Column` and `DB_Column` rendering in Table Viz.
![image](https://github.com/enso-org/enso/assets/4699705/8c50c519-7c34-4228-bab7-be938a6ce1bf)
![image](https://github.com/enso-org/enso/assets/4699705/c83ad301-f31e-451d-a8be-b022e1456b4e)

# Important Notes
- Can't use `.` to access a private constructor field when mapping, have to use a lambda.
- We should agree convention for private backing fields which are wrapped by a public getter (went with `internal_` for this pass).
2024-05-06 13:49:12 +00:00
Pavel Marek
2af217c3e6
from conversion propagates dataflow error (#9856)
Just add a dataflow error sentinel specialization to `InvokeConversionNode`.
2024-05-03 14:48:55 +00:00
AdRiley
f647045214
Make excel writer work for all types (#9846)
* New Test

* Improve DateTime recognition

* Re-enable slow test

* If there is a time take it regardless of format

* If there is a time take it regardless of format

* Code Review Changes
2024-05-03 07:09:54 +01:00
AdRiley
142e11dddc
Fix Test (#9836) 2024-05-02 12:30:09 +01:00
James Dunkerley
d2e6ff260e
Restructure SQLite_Details. (#9832)
```
type SQLite_Details
SQLite location:File|In_Memory

type In_Memory
```
to
```
type SQLite
From_File location:File

In_Memory
```

# Important Notes
Splits the In-Memory entry for Database Connect but still works nicely.

![image](https://github.com/enso-org/enso/assets/4699705/ec798ce0-9f41-4903-a2fd-722a9e37743c)

![image](https://github.com/enso-org/enso/assets/4699705/f233b055-893e-4c56-a23d-562e982560f6)
2024-05-01 22:15:41 +00:00
Pavel Marek
6b0361c33c
Panics can be caught from TCO loops (#9705)
In certain cases, when the `action` of `Panic.catch` is tail-call-optimized (via `@Tail_Call`) annotation, the panic is not caught. Fixed by ensuring that the `action` of `Panic.catch` is executed as `NOT_TAIL` rather than `TAIL_DIRECT`.

# Important Notes
The `handler` parameter of `Panic.catch` is executed as `NOT_TAIL` as well, just to be sure.
2024-05-01 11:13:29 +00:00
James Dunkerley
4d6d6f239c
Handle URL encoding automatically in query string. (#9823)
A small fix to automatically encode the query string.
Attaches a warning if needed.

![image](https://github.com/enso-org/enso/assets/4699705/032bdb59-6896-46c0-b970-f5a542cc6adf)

![image](https://github.com/enso-org/enso/assets/4699705/6b2075b9-3c98-4de2-8a34-c860ecd65d0c)
2024-04-30 22:03:46 +00:00
Radosław Waśko
f87c75a3da
Prepare for Cloud Environment variable name change (#9821)
Changing `ENSO_PROJECT_DIRECTORY_PATH` to `ENSO_CLOUD_PROJECT_DIRECTORY_PATH`.
2024-04-30 16:02:12 +00:00
AdRiley
d1bf4cb771
Add Ignored_Nothing_Values (#9770)
Add a `IgnoredNothing` warning for Table.Running

![image](https://github.com/enso-org/enso/assets/1720119/1941d278-2c33-43fe-a175-8bcc65bae51a)

![image](https://github.com/enso-org/enso/assets/1720119/b5f6b235-d939-4868-9490-de0f226ea1a2)

![image](https://github.com/enso-org/enso/assets/1720119/a1d617a6-a684-4cc1-be13-c4907d2e6876)
2024-04-30 13:30:40 +00:00
AdRiley
32c3f5f3e8
Make Table.should_equal and Column.should_equal consider NaN equal (#9799)
* Make Column.should_equal detect colums of different types and think nan==nan

* Refactor Table.should_equal

* More Column tests

* Adjust spacing

* Tests Green

* Check same number of columns

* Refactor

* Extra test

* Code Review Changes

* Fix

* Fix

* Fix tests

* Fix Tests

* Fix Test

* Fix test

* Code review change
2024-04-29 22:21:34 +01:00
Pavel Marek
660c5e7a9d
Atom constructors can be private (#9692)
Closes #8836.

Atom constructors can be declared as private (project-private). project-private constructors can be called only from the same project. See the encapsulation.md docs for more info.

---------

Co-authored-by: Jaroslav Tulach <jaroslav.tulach@enso.org>
Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
Co-authored-by: Hubert Plociniczak <hubert.plociniczak@gmail.com>
Co-authored-by: Kaz Wesley <kaz@lambdaverse.org>
2024-04-29 14:43:18 +02:00
Radosław Waśko
16e1999040
Fix copy_to/move_to edge case when source and target are the same (#9798)
- Adds tests for an edge case where the source and target of copy/move operation are the same.
- And fixes the behaviour to be as expected.
2024-04-29 12:29:07 +00:00
Radosław Waśko
df4a7efcdd
Add AWS SSO JARs to the Standard.AWS library (#9782)
As reported by our users, when using the AWS SSO, our code was failing with:
```
Execution finished with an error: To use Sso related properties in the 'xyz' profile, the 'sso' service module must be on the class path.
```

This PR adds the missing JARs to fix that.

Additionally it improves the license review tool UX a bit (parts of #9122):
- sorting the report by amount of problems, so that dependencies with unresolved problems appear at the top,
- semi-automatic helper button to rename package configurations after a version bump,
- button to remove stale entries from config (files or copyrights that disappeared after update),
- button to add custom copyright notice text straight from the report UI,
- button to set a file as the license for the project (creating the `custom-license` file automatically)
- ability to filter processed projects - e.g. `openLegalReviewReport AWS` will only run on the AWS subproject - saving time processing unchanged dependencies,
- updated the license search heuristic, fixing a problem with duplicates:
- if we had dependencies `netty-http` and `netty-http2`, because of a prefix-check logic, the notices for `netty-http` would also appear again for `netty-http2`, which is not valid. I have improved the heuristic to avoid these false positives and removed them from the current report.
- WIP: button to mark a license type as reviewed (not finished in this PR).
2024-04-25 18:44:51 +00:00
Radosław Waśko
0c989c6938
Writing to Data Links (#9750)
- Closes #9292
- and also closes #9522
2024-04-25 17:55:49 +00:00
GregoryTravis
b150b091b0
Convert Vector.new_builder to Vector.build in Standard.Base (#9754)
* Vector.build

* new_builder deprecated

* convert one

* convert vector_spec

* limit param

* convert vector

* docs

* changelog

* wip

* rename limit to capacity

* wip

* review

* append comments

* review

* wip

* wip

* fix tests

* docs

* wip

* wip

* fix filter warning propagation

* doc weird dependencies

* fix tests

* cleanup

* check builder result
2024-04-25 10:38:29 -04:00
Jaroslav Tulach
0d495ffd97
Make conversion of double to BigDecimal exact (#9740)
Resolves #9607 by computing `Number.hash` by converting given number to `Float` first and then computing the hash. Also the conversion from `Float.to Decimal` is exact - done via `new BigDecimal(double)`. There is `Decimal.new` that handles the user-friendly conversion. However as a result `Decimal.from 2.1 != Decimal.new 2.1` - that's the only way to ensure consistency between hash code and conversions.
2024-04-25 11:22:50 +00:00
James Dunkerley
fb9cf38914
Excel_Workbook.read_many (#9759)
- Some minor linting fixes.
- Adjust `headers` parameter so a dedicated type.
![image](https://github.com/enso-org/enso/assets/4699705/989f464d-df95-410e-a03b-36661f1c4a37)
- Fix bug with `read` on an `Excel_Workbook` so error handled more gracefully and not panicking to UI.
![image](https://github.com/enso-org/enso/assets/4699705/23b4575f-daad-4719-a5cc-30d064bd7f7a)
- Fix bug when writing to a file with an `Excel_Format` with an invalid extension which was causing a panic.
![image](https://github.com/enso-org/enso/assets/4699705/dc0e055c-c1b6-482f-b129-eb69f6554d72)
- Add `read_many` to `Excel_Workbook` allowing reading more than one sheet at a time.
2024-04-24 13:16:44 +00:00
Jaroslav Tulach
ff62c1e25f
Access non-self arguments when widget annotations are computed (#9410) 2024-04-24 03:58:32 +02:00
AdRiley
6f3e649e7a
Make errors propagate out of each (#9742)
Make errors propagate out of each
2024-04-23 17:14:54 +00:00
Jaroslav Tulach
96fdb67204
Special treatment for ..True and ..False (#9767) 2024-04-23 15:26:41 +02:00
AdRiley
4a97bfa31f
Add table running functionality for Sum, Mean, Min, Max. (#9577)
* Add Table.Running

* Code Review fixes

* Code Review changes

* Change null handling
2024-04-23 09:45:43 +01:00
GregoryTravis
d665f4d9c2
Change ‘decimal’ to ‘float’ in Numbers_Spec (#9718) 2024-04-23 08:09:42 +00:00
GregoryTravis
bc61d686c1
Integer.to_decimal and Float.to_decimal (#9716) 2024-04-23 08:06:31 +00:00
Jaroslav Tulach
11dda5b9bc
Allow ..Nothing to resolve to Nothing (#9746) 2024-04-22 06:54:07 +02:00
AdRiley
ceaba7f48d
Make excel writer work for custom types (#9752) 2024-04-20 10:34:06 +01:00
GregoryTravis
681df82697
Implement Vector.build: implementation and some tests (#9725)
* Vector.build

* new_builder deprecated

* convert one

* convert vector_spec

* limit param

* convert vector

* docs

* changelog

* rename limit to capacity

* review

* append comments

* review
2024-04-18 17:52:48 -04:00
GregoryTravis
56ae80a66b
Add tests for BigDecimal internal representation methods (#9717) 2024-04-18 20:42:41 +00:00
GregoryTravis
86ecd3e027
Add Decimal.floor, .ceil, and .trunc (#9694) 2024-04-17 18:42:38 +00:00
Radosław Waśko
34f3331561
Implement recursive argument for File.delete (#9719)
- Closes #8993
2024-04-17 16:53:45 +00:00
Radosław Waśko
41a39eb019
Fix Json.parse "" edge case and add a test for it (#9722)
- Fixes #9720
2024-04-17 08:48:48 +00:00
Radosław Waśko
fda41cbfd1
Writing Cloud files (#9686)
- Closes #9291
2024-04-16 14:01:03 +00:00
Radosław Waśko
bdda1830b7
Integrate Cloud path resolver (#9662)
- Closes #9363
- Cleans up the Cloud mock as it got a bit messy. It still implements the bare minimum to be able to test basic secret and auth handling logic 'offline' (added very simple path resolution, only handling the minimum set of cases for the tests to work).
- Adds first implementation of caching Cloud replies.
- Currently only caching the `Enso_User.current`. This is a simple one to cache because we do not expect it to ever change, so it can be safely cached for a long period of time (I chose 2h to make it still refresh from time to time while not being noticeable).
- We may try using this for caching other values in future PRs.
2024-04-12 13:03:09 +00:00
Pavel Marek
4dc7992ab5
Export works without import (#9683) 2024-04-12 14:23:34 +02:00
GregoryTravis
e3afa5561d
Add Decimal.round (#9672) 2024-04-11 15:47:50 +00:00
GregoryTravis
bf90e2e8a8
Attach warning when converting from Decimal to Integer (#9660) 2024-04-11 15:44:58 +00:00
GregoryTravis
676e989f7f
Add Decimal.min and .max (#9663)
* min and max

* changelog

* wip

* scale approach mostly works

* Revert "scale approach mostly works"

This reverts commit 88e6073f7a.

* review

* review

* return value type
2024-04-11 09:58:10 -04:00