Commit Graph

797 Commits

Author SHA1 Message Date
Radosław Waśko
48c17845a7
Fixing Database tests and Snowflake Dialect - part 3 out of ... (#10458)
- Related to #9486
- Fixes types in literal tables that are used throughout the tests
- Tries to makes testing faster by disabling some edge cases, trying batching some queries, re-using the main connection and trying to re-use tables more
- Implements date/time type mapping and operations for Snowflake
- Updates type mapping to correctly reflect what Snowflake does
- Disables warnings for Integer->Decimal coercion as that's too annoying and implicitly understood in Snowflake
- Allows to select a Decimal column with `..By_Type ..Integer` (only in Snowflake backend) because the Decimal column there is its 'de-facto' Integer column replacement.
2024-07-10 13:21:30 +00:00
Jaroslav Tulach
ee0175c364
Support re-running failed tests with runEngineDistribution (#10483) 2024-07-10 07:05:29 +02:00
James Dunkerley
8da06309e9
Date Time Pickers, Temporarily Disable Encoding.default (#10493)
- Widgets for Date_Time, Time_Of_Day and Time_Zone.
- Disable Encoding.default for now as big performance impact on CSVs.

![image](https://github.com/enso-org/enso/assets/4699705/c1b936f0-3ab4-490c-8fe5-2310ef1ed079)

![image](https://github.com/enso-org/enso/assets/4699705/d5e29ec4-cc52-41e5-a532-17cd6dff34b9)

![image](https://github.com/enso-org/enso/assets/4699705/61455519-ea63-4275-9c7a-603714ff9f85)

![image](https://github.com/enso-org/enso/assets/4699705/48ccd3ad-5e15-49f9-87cd-4710ca559843)
2024-07-09 21:04:08 +00:00
GregoryTravis
71a6e2162e
Fix return type for Postgres Decimal division (#10479) 2024-07-09 15:22:14 +00:00
Radosław Waśko
a3dc50fe1e
Replace presigned S3 URL with lambda request (#10456)
- Closes #10419
2024-07-09 09:36:10 +00:00
James Dunkerley
4b3e4ae15e
Rename Map to Dictionary and Set to Hashset. (#10474)
- Rename `Map` to `Dictionary`.
- Rename `Set` to `Hashset`.
- Add a deprecated place holder for the static method of `Map`.
2024-07-09 09:12:23 +00:00
James Dunkerley
4c0647ea29
Stop publishing First/Last as constructors and use auto-scoping for take and drop. (#10467)
- Removes `First` and `Last` from the `Standard.Base` exports.
- Enable auto-scoping for all `Index_Sub_Range` and `Text_Sub_Range`.
- Update all use of those methods to use auto-scoping.
2024-07-08 10:26:30 +00:00
James Dunkerley
018d4c312f
Stop publishing Postgres constructor, update Postgres_Details.Postgres to Postgres.Server. (#10466)
![image](https://github.com/enso-org/enso/assets/4699705/6d0d4167-e97b-4765-8079-650ad091ce60)

- Rename `Postgres_Details` to `Postgres`.
- Rename `Postgres` constructor to `Server`.
- Update SPI.
- Linting issues (indent, missing doc comment)
2024-07-08 07:58:08 +00:00
James Dunkerley
d65371096b
Deserialise large integer and decimals from JSON automatically. (#10463)
- Sort large integer and decimal JSON deserialization.
- Change type to be Integer instead of BigInt for large integers.
- Add tests.
- Update Table viz.
- Preserve white space in JSON viz.
![image](https://github.com/enso-org/enso/assets/4699705/48c83616-c0ed-4cb4-862a-34cd4fff09aa)

![image](https://github.com/enso-org/enso/assets/4699705/5bae9ccd-1d0f-4b70-aea5-d4cebc3d9df8)
2024-07-05 21:56:08 +00:00
James Dunkerley
c2c4b95116
Final step removing the Problem_Behavior publishing. (#10461)
- Remove publishing the constructors.
- Fix any missed use in libs.
- Alter tests to generally use auto-scoped calls.
- `on_incomparable` to `on_problems`.
2024-07-05 18:41:36 +00:00
James Dunkerley
2124cd022a
Batch Two of Problem_Behavior (#10454)
Part 2 of removing the public constructors.
Completes review of the libraries.

Tests left to do then can remove the export.
2024-07-05 13:40:29 +00:00
AdRiley
fc93b4d121
Refactor database dialect types (#10437)
* Auto-commit work in progress before clean build on 2024-07-03 14:17:22

* Refactor

* Revert

* revert

* Code Review feedback

* Green

* 2 Red

* Green

* Renames

* Code review changes

* Code review changes
2024-07-05 13:08:25 +01:00
James Dunkerley
82678aa7cc
Starting to clean up exporting Problem_Behavior constructor refactor. (#10449)
Use type check everywhere in Standard.Base for Problem_Behavior.
2024-07-05 10:05:51 +00:00
James Dunkerley
0661f17d1c
Tune Text.trim, fix for Text.split (#10445)
- Rename `Location.Start` to `Location.Left`.
- Rename `Location.End` to `Location.Right`.
- Use auto-scoping for `Location`.
- Tune widgets for `Text.trim`.
- Correct signature of `Text.split`.
- Adjist `generateLocallyUniqueIdent` to not fail on bad signature.
2024-07-04 22:24:56 +00:00
James Dunkerley
9a2aed92f1
Separating list from read function and other small tweaks. (#10434)
- Rename `Faker.string_value` to `Faker.text_value`.
- Remove `Regex.pattern_string` as duplicate of `Regex.pattern`.
- Sort the Date picker.
- Rename `Data.list_directory` to `Data.list`.
- Remove support for reading a directory.

![image](https://github.com/enso-org/enso/assets/4699705/b42bb3c9-e63b-49f2-8cdc-4666cb6d968e)

![image](https://github.com/enso-org/enso/assets/4699705/97f49891-5ae5-4f0a-9a41-6200888fcd86)
2024-07-03 22:00:53 +00:00
marthasharkey
ee39fd7f53
Visualise excelwork book sheets, db tables/schemas and add node adding gets to query them (#10362)
changes to current table (and db table) on index click:
- node added on double click
- tooltip added
- cursor changes to pointer
- styled as a link

![7473-table-w-links](https://github.com/enso-org/enso/assets/170310417/5e60c177-3f83-4db7-be86-fa8a9d493204)


Row table types:
Added on click to show value
![7473-row-links](https://github.com/enso-org/enso/assets/170310417/82f878ea-420d-4308-99bf-c77a6340a8c3)



Excel Workbook connections
- show sheet names in column
- sheet names are clickable that add a 'read' node for the corresponding sheet
- Shown in table with header title: Sheets

![7473-excel-links-sheets](https://github.com/enso-org/enso/assets/170310417/748f524e-5cca-4c20-b458-132af9a57ec1)


SQLite Connection:
- shows available schemas in column
- schemas are clickable that add a 'query' node for the corresponding schema/table
![7473-sql-lite-links](https://github.com/enso-org/enso/assets/170310417/21a8006f-c462-4128-874d-05380d3bab00)


Postgres Connection:
- shows available tables in a column
- tables are clickable and add a 'query' node for the corresponding table
![7473-postgres-links](https://github.com/enso-org/enso/assets/170310417/66b134ff-80fd-4995-b445-505919e25cfa)

JS_Object
- style keys as links
![7473-json-links](https://github.com/enso-org/enso/assets/170310417/9ec17c85-c1f4-42ab-b40a-06d6bc29d54f)
2024-07-03 13:58:07 +00:00
James Dunkerley
4bb82fae4a
Fix issues with Date.new Widgets and Some Viz Fixes. (#10430)
- Do not include nothing values for optional parameters when serializing widgets.
- Serialize Nothing as `{"type": "Nothing"}` for the default JSON viz.
- Add JS Object support for decimal.
- Fix `to_json` for large integers

![image](https://github.com/enso-org/enso/assets/4699705/88a404a3-537d-4be7-9a47-c896d2f8b378)

![image](https://github.com/enso-org/enso/assets/4699705/ce09955e-b377-431a-aca3-28f442c772a6)

![image](https://github.com/enso-org/enso/assets/4699705/d0c99a0b-1964-4874-8d87-41979a17a359)

![image](https://github.com/enso-org/enso/assets/4699705/32ade951-0921-4837-9c35-9b60488f863f)

![image](https://github.com/enso-org/enso/assets/4699705/f0fba27a-dfcc-4f17-a883-deb23757f74e)
2024-07-03 06:41:40 +00:00
GregoryTravis
48fb999eb3
Implement Decimal support for Postgres backend (#10216)
* treat scale nothing as unspecifed

* cast to decimal

* float int biginteger

* conversion failure ints

* loss of decimal precision

* precision loss for mixed column to float

* mixed columns

* loss of precision on inexact float conversion

* cleanup, reuse

* changelog

* review

* no fits bd

* no warning on 0.1 conversion

* fmt

* big_decimal_fetcher

* default fetcher and statement setting

* round-trip d

* fix warning

* expr +10

* double builder retype to bigdecimal

* Use BD fetcher for underspecified postgres numeric column, not inferred builder, and do not use biginteger builder for integral bigdecimal values

* fix tests

* fix test

* cast_op_type

* no-ops for other dialects

* Types

* sum + avg

* avg + sum test

* fix test

* update agg type inference test

* wip

* is_int8, stddev

* more doc, overflow check

* fmt

* finish round-trip test

* wip
2024-07-02 15:01:55 -04:00
James Dunkerley
69b5f719e8
Adjust Colours. (#10417)
- Change the lightness of the component colours to match Figma.
- Remove the background image from tailwind.css.
- Alter the code editor background to be white.
- Make the selection colour much brighter in the dashboard.

![image](https://github.com/enso-org/enso/assets/4699705/4ae8f03a-bdeb-48cf-86eb-aa058a2a1b01)

![image](https://github.com/enso-org/enso/assets/4699705/eb5d4cf8-d275-4c0d-b876-4a29ce7768bf)

![image](https://github.com/enso-org/enso/assets/4699705/0df64403-75e5-4e44-a818-58cedbbcd3c9)

# Important Notes
Removed the `is-hidden-file` package from the project manager shim. It seems to be unreliable in Win11 and as just for local testing made just do dot file logic.
2024-07-01 20:01:18 +00:00
Radosław Waśko
db4f7ab3b5
Fixing Database tests and Snowflake Dialect - part 2 out of ... (#10319)
- Part of #9486
- Fixing our tests to not rely on deterministic ordering of created Tables in Database backends
- Before, SQLite and Postgres used to mostly return rows in the order they were inserted in, but Snowflake does not.
- Fixing various parts of Snowflake dialect.
2024-06-27 14:54:00 +00:00
James Dunkerley
d92078471b
Rename order_by to sort for Table and DB_Table. (#10372)
- Rename `order_by` to `sort` for `Table` and `DB_Table`.
- Added deprecated placeholder.
- Fixed a couple of minor deprecated mistakes.

![image](https://github.com/enso-org/enso/assets/4699705/96c32fa7-33e5-400a-9d3a-ebf330886911)
2024-06-26 17:46:09 +00:00
AdRiley
6bc7d4f786
Fix signatures for distinct and duplicates (#10365)
Fixes #10364
2024-06-25 12:59:27 +00:00
Jaroslav Tulach
0cde0e7f96
More detailed No_Such_Method error message for function like objects (#10328) 2024-06-25 11:36:56 +02:00
James Dunkerley
e6c8ec7ab5
Changes from session with Ned (#10349)
- Removed `second_row` and `second_column` from the `Table` and `DB_Table`.
- Added `first_value` and `last_value` to the `Table` and `DB_Table`.
- Fixed bug where negative index access wasn't allowed on `Column`.
- Added error if negative index access used on `DB_Column`. Tells user they have to materialize.
- Fix argument order for `Table.text_cleanse` and a couple of typo corrections.
- Rename `auto_value_type` to `auto_cast` on table and columns.
2024-06-24 12:47:14 +00:00
AdRiley
c324c78e23
Add duplicates component (#10323)
* Update existing behaviou to match new

* Add signatures

* Red test

* First test green

* sbt javafmtAll

* In-Memory working

* Not implemeted for In-Db

* Docs

* Disable tests for in-db

* Changelog

* Code review changes

* Fix

* Fix

* Fixc tests
2024-06-24 13:29:03 +03:00
James Dunkerley
791dba6729
Autoscoping for File_Format and minor tweaks. (#10348)
- Add `rename` ALIAS to `Table.use_first_row_as_names`.
- Add a shorthand to `Faker.string_value` to allow quick creation of fake values such as National Insurance numbers.
- Add `match` ALIAS for `Text.find` and `Text.find_all`.
- Auto scoping for `File_Format`. Demonstrates how to use it in a dynamic context.
- `SQLite_Format.For_File` renamed to `SQLite_Format.SQLite` (though kept for backwards compatibility.
- Fixed bug in `SQLite_Format` which was calling a non-existent constructor.

![image](https://github.com/enso-org/enso/assets/4699705/4506d27c-c1ff-4ad6-9276-53c2ae00de17)

![image](https://github.com/enso-org/enso/assets/4699705/9043ffb0-6740-42ba-91f8-ab0df555f20f)

![image](https://github.com/enso-org/enso/assets/4699705/03122fac-bdbb-4bcf-ac96-9491da41a8b9)

![image](https://github.com/enso-org/enso/assets/4699705/79122aac-a74a-435d-9849-ac3421f7d080)

![image](https://github.com/enso-org/enso/assets/4699705/54544de8-9aea-4dc6-bb4d-a7d8233c6814)
2024-06-24 08:28:54 +00:00
Jaroslav Tulach
fe2cf49568
Run whole test/Base_Tests in native image runner (#10296) 2024-06-21 06:03:53 +02:00
James Dunkerley
5042592d24
Some formatting issues and a few tweaks (#10298)
- Linting fixes.
- Adjust doc comments with `<` or `>` in plain text areas to use `&lt;` and `&gt;`.
- Refactor Statistics module and add auto-scoping.
- Add auto-scoping to `Text.to_case`.
- Fix type issue with `Table.get_value`.
- Private constructor for `Excel_Workbook` and move `xls_format` to public method.
- Add `fields` widget to `to_table`.
- Add `simple_expr` to make `Table.set` clearer.

![image](https://github.com/enso-org/enso/assets/4699705/3e21e800-142c-4006-a6c2-dd6196b76c9a)

![image](https://github.com/enso-org/enso/assets/4699705/d40dcbfd-a35e-4849-9e1a-f4a418d562dd)
2024-06-20 10:44:36 +00:00
Radosław Waśko
a8358512ad
Small fixes to Cloud Integration (#10303)
- Includes HTTP method in error message
- Does not do special handling for `403` status code - this was wrong and led to `Unauthorized` error when the real cause was lack of permssions in the Cloud. The errors should be more understandable now.
- Adds `projectSessionId` to audit log metadata.
- Fixes a test (`Secrets_Spec`) that did not have unique names and would fail if cleanup of previous runs failed (or if ran in parallel).
2024-06-18 09:41:33 +00:00
marthasharkey
bb16db9c79
Remove legacy matrix and legacy objects from table visualization and remove the index column (#10260)
- remove legacy matrix and object types from vue component and any further code relating to those
- remove the index and index header being sent in the json for tables
- add has_index_col flag for json hat previously sent 'indicies_header' and 'indicies' so that index/# column is still rendered where required
2024-06-17 09:17:20 +00:00
Radosław Waśko
3a4784c226
Initial separation of Snowflake_Dialect from Postgres_Dialect (#10266)
- Part of #9486
- Building on top of initial work by @jdunkerley and finishing it
- Reverted the changes to the Postgres_Dialect from last Snowflake work and split the Snowflake_Dialect into a separate module.
- Moved from `rounding_decimal_places_not_allowed_for_floats` to `supports_float_round_decimal_places` (as too confusing).
- Added Snowflake_Dialect type.
- Extracted `Snowflake_Spec` into separate `Snowflake_Tests`
- It imports the common tests from `Table_Tests`.
- Some initial adaptations to make the snowflake dialect not-crash.
- Adding `Internals_Access` proxy to allow external implementations to access our internal data structures without directly exposing them to users. Users should not use these.
- Adding profiling of SQL to check performance.
2024-06-13 16:12:20 +00:00
AdRiley
fadb81abe6
Fix Text_Cleanse tests (#10263)
I hadn't connected the Text_Cleanse tests up properly and as a result they weren't running or working. This fixes that.
2024-06-13 08:25:32 +00:00
marthasharkey
55af1b9ffd
Add icon to indicate value type in table (#10225)
closes #10018

Sends the value type within the json for table visualisation
Header uses a html template to show it's value type to the right of the title
Displays the value type also in a tooltip, this displays the types "display_text"

<img width="392" alt="image" src="https://github.com/enso-org/enso/assets/170310417/0828e6a2-b30f-4ac7-9a8f-46b4a9cfac91">


tooltip:
<img width="498" alt="image" src="https://github.com/enso-org/enso/assets/170310417/f9964f90-9337-42d3-a0ef-3c58f6d74621">
2024-06-12 09:06:55 +00:00
AdRiley
783276ca34
Make test framework list out failed tests to use in filter (#10241)
* List out failed tests

* Code review changes
2024-06-11 21:15:28 +01:00
Radosław Waśko
c4b0ca8f69
Ensure type ascriptions are correctly transformed to facilitate checking of more complex return types (#10229)
- Fixes #9980
- Adds some tests to ensure types like `|` or `&` (in addition to `!` from the ticket) correctly work in return type check.
- Fixes a weird behaviour where we used to avoid processing type related IR transformations inside of type ascriptions.
- Adds parentheses to type representations if they are more complex: `A | B & C` is unclear as it can either mean `A | (B & C)` or `(A | B) & C` which have different meanings. If we now have an operation with such nesting, the sub expressions are wrapped in parentheses to disambiguate.
2024-06-11 17:11:03 +00:00
Pavel Marek
5fa29c51b5
Fix Meta.enso_project (#10192)
Fixes `Standard.Base.Meta.Enso_Project.enso_project` to return a project descriptor for the *main* project, i.e., the one configured as a *root* for the engine.

# Important Notes
`enso_project` builtin no longer iterates the stack frames to infer the project descriptor. It derives it from the default package repository.
2024-06-11 08:31:40 +00:00
Radosław Waśko
41d02e95ef
Implement Windows-1252 fallback logic for Encoding.Default (#10190)
- Closes #10148
- [x] Tests for `Restartable_Input_Stream`, `peek_bytes` and `skip_n_bytes`.
- [x] Report `Managed_Resource` stack overflow bug: #10211
- [x] Followup possible optimization: #10220
- [x] Test use-case from blog.
2024-06-10 10:49:26 +00:00
James Dunkerley
d938c96c55
Adding type annotations and enabling auto-scoping (#10173)
- Renamed `Missing_Required_Argument` to `Missing_Argument`, and added `throw` method.
- Add default widget to `Case_Sensitivity.Insensitive locale`.
- Switch to auto scoping for `parse_type_selector`.
- Add type annotation to various simple typed arguments in `Table` and `DB_Table`.
- Altered `Filter_Condition` to have `Missing_Argument` for all non-defaulted arguments.
- Added resolution of `Column_Ref` passed as auto-scoped to `Table_Ref`.
- Altered `Simple_Calculation` to have `Missing_Argument` for all non-defaulted arguments.
- Altered `Join_Condition` to have `Missing_Argument` for all non-defaulted arguments.
- Altered `Sort_Column` to have `Missing_Argument` for all non-defaulted arguments.
- Altered `Aggregate_Column` to have `Missing_Argument` for all non-defaulted arguments.

**rename_columns:**
![image](https://github.com/enso-org/enso/assets/4699705/08aaba0f-687a-450c-9781-8eadc062bd50)

**aggregate:**
![image](https://github.com/enso-org/enso/assets/4699705/c29e7944-1a1c-4020-9fe0-528d874b8049)

**join:**
![image](https://github.com/enso-org/enso/assets/4699705/50038166-e56d-48c5-9eeb-bd46fa415e46)

**set:**
![image](https://github.com/enso-org/enso/assets/4699705/bee2462a-dafb-4bd4-b102-ec73edb4fb93)
2024-06-10 07:52:32 +00:00
GregoryTravis
4aa3d52b60
Implement conversions for Decimal column (#10206)
* treat scale nothing as unspecifed

* cast to decimal

* float int biginteger

* conversion failure ints

* loss of decimal precision

* precision loss for mixed column to float

* mixed columns

* loss of precision on inexact float conversion

* cleanup, reuse

* changelog

* review

* no fits bd

* no warning on 0.1 conversion

* fmt
2024-06-07 15:37:32 -04:00
AdRiley
fe6eafd06e
Add By_Type option to more components (#10183)
* reorder_columns

* format

* cast

* auto_value_types

* replace

* fill_nothing

* fill_empty

* text_replace

* text_cleanse

* Add By_Type

* Fix cast

* Fix more tests

* Fix the table helper

* Bug fix

* Remove check
2024-06-07 16:26:23 +01:00
Hubert Plociniczak
1bc14252df
Extract mutable builder from ModuleScope (#9914)
Refactored mutable parts of `ModuleScope` into builder to make it easier to reduce unnecessary locks.

# Important Notes
Elements of ModuleScope (types, imports etc) are used while _building_ of it may still be in progress. In order to make static typing happy, every `ModuleScope.Builder` can be exposed as (unmodifiable) `ModuleScope`.
2024-06-05 16:57:08 +00:00
Radosław Waśko
698f4b5640
Remove workaround for #8937 after it has been fixed (#10149)
- Since #8937 was fixed by #10104 the workaround is no longer needed
2024-06-05 14:23:12 +00:00
GregoryTravis
5fad3558a6
BigDecimalBuilder and arithmetic operations. (#9950)
* hack

* make a column

* add

* no scale=0 on BD type

* a test

* wip

* 3 arithmetic ops

* /

* wip

* BigDecimalPowerOp

* wip

* mod test

* NumericBinaryOpReturningBigDecimal

* with scalar

* misc arithmetic tests

* fix integralBigDecimalToInteger

* mixed columns

* bigdecimal pow via double

* cleanup

* j2e on get

* arithmetic exception

* mod 0

* cleanup

* fmt

* changelog

* check type first

* merge

* mc error message

* add BD case to Builder.java

* fmt

* changelog

* add BD case to StorageConverter.java

* fmt

* fix test
2024-06-04 13:59:31 -04:00
AdRiley
7c35781a14
Add select/remove column by type (#10159)
* stash

* Working first pass

* Tests

* remove columns

* more tests

* More tests

* DB_Table

* In DB

* Deprecate

* Update documentation

* More docs

* Remove strict

* Remove tests

* Fix widgets

* Add doc

* Spaces

* Fix In-DB
2024-06-04 16:23:46 +01:00
marthasharkey
24d209abf9
Add Columns_To_Add type to move away from Integer | Nothing (#10152)
Adds new type "Coulmns_To_Add" that replaces the Integer | Nothing type
And adds new widget to split_to_columns and tokenise_to_columns
<img width="397" alt="image" src="https://github.com/enso-org/enso/assets/170310417/1b155682-0992-4cc0-8964-62b389ee7072">

Closes #10041
2024-06-04 15:06:08 +00:00
Radosław Waśko
7cf80f3196
Handle UTF BOM when decoding text (#10130)
- Improve BOM handling: detect and skip the BOM character, Default encoding that detects encoding based on BOM if present, warnings if unexpected BOM is encountered.
- Closes #9849
- Windows-1252 fallback will be done as a separate PR as it has additional complexity. Tracked in ticket #10148.
2024-06-04 13:22:19 +00:00
GregoryTravis
1c2955655c
Add no_warning_propagation flag to Vector.build and Builder.new (#10079)
* no_warning_propagation flag

* doc

* benchmarks

* change to propagate_warnings, test

* fix java benchmark

* add benchmark to main
2024-06-04 09:06:09 -04:00
James Dunkerley
f1a1901058
Bug fix for read by line. (#10161)
Needs to pass -1 not Nothing to JSON.
2024-06-03 21:33:45 +00:00
Jaroslav Tulach
9632f04e9b
Directly allocate warnings asVectorEnsoObjects (#10143) 2024-06-01 06:23:49 +02:00
AdRiley
af4177e61f
Make table default visualisation (#10128)
Makes table the default vis type.

Before:
![image](https://github.com/enso-org/enso/assets/1720119/8a5d113a-ac06-4dd8-afdb-151cdbc149ab)

After:
![image](https://github.com/enso-org/enso/assets/1720119/d7c33cf7-355d-4517-8967-1963ff506f97)

Closes #10093
2024-05-31 15:02:02 +00:00