Commit Graph

877 Commits

Author SHA1 Message Date
Jaroslav Tulach
0cde0e7f96
More detailed No_Such_Method error message for function like objects (#10328) 2024-06-25 11:36:56 +02:00
James Dunkerley
e6c8ec7ab5
Changes from session with Ned (#10349)
- Removed `second_row` and `second_column` from the `Table` and `DB_Table`.
- Added `first_value` and `last_value` to the `Table` and `DB_Table`.
- Fixed bug where negative index access wasn't allowed on `Column`.
- Added error if negative index access used on `DB_Column`. Tells user they have to materialize.
- Fix argument order for `Table.text_cleanse` and a couple of typo corrections.
- Rename `auto_value_type` to `auto_cast` on table and columns.
2024-06-24 12:47:14 +00:00
AdRiley
c324c78e23
Add duplicates component (#10323)
* Update existing behaviou to match new

* Add signatures

* Red test

* First test green

* sbt javafmtAll

* In-Memory working

* Not implemeted for In-Db

* Docs

* Disable tests for in-db

* Changelog

* Code review changes

* Fix

* Fix

* Fixc tests
2024-06-24 13:29:03 +03:00
James Dunkerley
791dba6729
Autoscoping for File_Format and minor tweaks. (#10348)
- Add `rename` ALIAS to `Table.use_first_row_as_names`.
- Add a shorthand to `Faker.string_value` to allow quick creation of fake values such as National Insurance numbers.
- Add `match` ALIAS for `Text.find` and `Text.find_all`.
- Auto scoping for `File_Format`. Demonstrates how to use it in a dynamic context.
- `SQLite_Format.For_File` renamed to `SQLite_Format.SQLite` (though kept for backwards compatibility.
- Fixed bug in `SQLite_Format` which was calling a non-existent constructor.

![image](https://github.com/enso-org/enso/assets/4699705/4506d27c-c1ff-4ad6-9276-53c2ae00de17)

![image](https://github.com/enso-org/enso/assets/4699705/9043ffb0-6740-42ba-91f8-ab0df555f20f)

![image](https://github.com/enso-org/enso/assets/4699705/03122fac-bdbb-4bcf-ac96-9491da41a8b9)

![image](https://github.com/enso-org/enso/assets/4699705/79122aac-a74a-435d-9849-ac3421f7d080)

![image](https://github.com/enso-org/enso/assets/4699705/54544de8-9aea-4dc6-bb4d-a7d8233c6814)
2024-06-24 08:28:54 +00:00
Jaroslav Tulach
fe2cf49568
Run whole test/Base_Tests in native image runner (#10296) 2024-06-21 06:03:53 +02:00
James Dunkerley
5042592d24
Some formatting issues and a few tweaks (#10298)
- Linting fixes.
- Adjust doc comments with `<` or `>` in plain text areas to use `&lt;` and `&gt;`.
- Refactor Statistics module and add auto-scoping.
- Add auto-scoping to `Text.to_case`.
- Fix type issue with `Table.get_value`.
- Private constructor for `Excel_Workbook` and move `xls_format` to public method.
- Add `fields` widget to `to_table`.
- Add `simple_expr` to make `Table.set` clearer.

![image](https://github.com/enso-org/enso/assets/4699705/3e21e800-142c-4006-a6c2-dd6196b76c9a)

![image](https://github.com/enso-org/enso/assets/4699705/d40dcbfd-a35e-4849-9e1a-f4a418d562dd)
2024-06-20 10:44:36 +00:00
Radosław Waśko
a8358512ad
Small fixes to Cloud Integration (#10303)
- Includes HTTP method in error message
- Does not do special handling for `403` status code - this was wrong and led to `Unauthorized` error when the real cause was lack of permssions in the Cloud. The errors should be more understandable now.
- Adds `projectSessionId` to audit log metadata.
- Fixes a test (`Secrets_Spec`) that did not have unique names and would fail if cleanup of previous runs failed (or if ran in parallel).
2024-06-18 09:41:33 +00:00
marthasharkey
bb16db9c79
Remove legacy matrix and legacy objects from table visualization and remove the index column (#10260)
- remove legacy matrix and object types from vue component and any further code relating to those
- remove the index and index header being sent in the json for tables
- add has_index_col flag for json hat previously sent 'indicies_header' and 'indicies' so that index/# column is still rendered where required
2024-06-17 09:17:20 +00:00
Radosław Waśko
3b566709f3
Type Inference PoC - iteration 1 - most basic type inference (#8652)
- Closes #8590
2024-06-13 16:23:56 +00:00
Radosław Waśko
3a4784c226
Initial separation of Snowflake_Dialect from Postgres_Dialect (#10266)
- Part of #9486
- Building on top of initial work by @jdunkerley and finishing it
- Reverted the changes to the Postgres_Dialect from last Snowflake work and split the Snowflake_Dialect into a separate module.
- Moved from `rounding_decimal_places_not_allowed_for_floats` to `supports_float_round_decimal_places` (as too confusing).
- Added Snowflake_Dialect type.
- Extracted `Snowflake_Spec` into separate `Snowflake_Tests`
- It imports the common tests from `Table_Tests`.
- Some initial adaptations to make the snowflake dialect not-crash.
- Adding `Internals_Access` proxy to allow external implementations to access our internal data structures without directly exposing them to users. Users should not use these.
- Adding profiling of SQL to check performance.
2024-06-13 16:12:20 +00:00
AdRiley
fadb81abe6
Fix Text_Cleanse tests (#10263)
I hadn't connected the Text_Cleanse tests up properly and as a result they weren't running or working. This fixes that.
2024-06-13 08:25:32 +00:00
Jaroslav Tulach
5339484285
shouldCreateClosureRootNode when valueHasSomeTypeCheck (#10262)
Fixes #10236.
2024-06-12 14:04:50 +00:00
Jaroslav Tulach
87864bf564
Presence of Any disables type checks (#10248)
Fixes #10237 by avoiding creation of `ReadArgumentCheckNode` when the requested type allows `Any`.
2024-06-12 11:10:23 +00:00
marthasharkey
55af1b9ffd
Add icon to indicate value type in table (#10225)
closes #10018

Sends the value type within the json for table visualisation
Header uses a html template to show it's value type to the right of the title
Displays the value type also in a tooltip, this displays the types "display_text"

<img width="392" alt="image" src="https://github.com/enso-org/enso/assets/170310417/0828e6a2-b30f-4ac7-9a8f-46b4a9cfac91">


tooltip:
<img width="498" alt="image" src="https://github.com/enso-org/enso/assets/170310417/f9964f90-9337-42d3-a0ef-3c58f6d74621">
2024-06-12 09:06:55 +00:00
Jaroslav Tulach
aaaebcabf8
Implement and benchmark ArrowOperationPlus node (#10150)
Prototype of #10056 showing `+` operation implemented in the _Arrow language_.
2024-06-11 12:50:59 +00:00
Pavel Marek
5fa29c51b5
Fix Meta.enso_project (#10192)
Fixes `Standard.Base.Meta.Enso_Project.enso_project` to return a project descriptor for the *main* project, i.e., the one configured as a *root* for the engine.

# Important Notes
`enso_project` builtin no longer iterates the stack frames to infer the project descriptor. It derives it from the default package repository.
2024-06-11 08:31:40 +00:00
Paweł Grabarz
940ca2de5a
Reduce probability of non-determinism test randomly failing. (#10135)
The standard library vector sampling test happened to fail by random chance of two consecutive samplings being equal.

![image](https://github.com/enso-org/enso/assets/919491/3c9c73a8-51da-468c-a42d-88a99d30ecbf)

To prevent that from happening again, the sampled vector and number of samples was increased. Also, the non-determinism test for some reason was actually performed 3 times, giving 3 opportunities for samplings to accidentaly match and fail the test. Removed that outer loop, since one non-equality is plenty enough to pass the test.
2024-06-10 13:23:09 +00:00
Radosław Waśko
41d02e95ef
Implement Windows-1252 fallback logic for Encoding.Default (#10190)
- Closes #10148
- [x] Tests for `Restartable_Input_Stream`, `peek_bytes` and `skip_n_bytes`.
- [x] Report `Managed_Resource` stack overflow bug: #10211
- [x] Followup possible optimization: #10220
- [x] Test use-case from blog.
2024-06-10 10:49:26 +00:00
James Dunkerley
d938c96c55
Adding type annotations and enabling auto-scoping (#10173)
- Renamed `Missing_Required_Argument` to `Missing_Argument`, and added `throw` method.
- Add default widget to `Case_Sensitivity.Insensitive locale`.
- Switch to auto scoping for `parse_type_selector`.
- Add type annotation to various simple typed arguments in `Table` and `DB_Table`.
- Altered `Filter_Condition` to have `Missing_Argument` for all non-defaulted arguments.
- Added resolution of `Column_Ref` passed as auto-scoped to `Table_Ref`.
- Altered `Simple_Calculation` to have `Missing_Argument` for all non-defaulted arguments.
- Altered `Join_Condition` to have `Missing_Argument` for all non-defaulted arguments.
- Altered `Sort_Column` to have `Missing_Argument` for all non-defaulted arguments.
- Altered `Aggregate_Column` to have `Missing_Argument` for all non-defaulted arguments.

**rename_columns:**
![image](https://github.com/enso-org/enso/assets/4699705/08aaba0f-687a-450c-9781-8eadc062bd50)

**aggregate:**
![image](https://github.com/enso-org/enso/assets/4699705/c29e7944-1a1c-4020-9fe0-528d874b8049)

**join:**
![image](https://github.com/enso-org/enso/assets/4699705/50038166-e56d-48c5-9eeb-bd46fa415e46)

**set:**
![image](https://github.com/enso-org/enso/assets/4699705/bee2462a-dafb-4bd4-b102-ec73edb4fb93)
2024-06-10 07:52:32 +00:00
Jaroslav Tulach
b5969cf31c
Sort rational numbers and primes first benchmarks (#10142)
Investigating #6361 - can we speed callback from `java.util.TimSort.sort` to Enso by using `CallTarget`?
2024-06-09 12:46:41 +00:00
GregoryTravis
4aa3d52b60
Implement conversions for Decimal column (#10206)
* treat scale nothing as unspecifed

* cast to decimal

* float int biginteger

* conversion failure ints

* loss of decimal precision

* precision loss for mixed column to float

* mixed columns

* loss of precision on inexact float conversion

* cleanup, reuse

* changelog

* review

* no fits bd

* no warning on 0.1 conversion

* fmt
2024-06-07 15:37:32 -04:00
AdRiley
fe6eafd06e
Add By_Type option to more components (#10183)
* reorder_columns

* format

* cast

* auto_value_types

* replace

* fill_nothing

* fill_empty

* text_replace

* text_cleanse

* Add By_Type

* Fix cast

* Fix more tests

* Fix the table helper

* Bug fix

* Remove check
2024-06-07 16:26:23 +01:00
Jaroslav Tulach
396d70ddc0
Execute foreign function and check autoscoped constructor result (#10187)
Fixes #10151 and also fixes #10180 and fixes #10186.
2024-06-06 13:16:27 +00:00
GregoryTravis
5fad3558a6
BigDecimalBuilder and arithmetic operations. (#9950)
* hack

* make a column

* add

* no scale=0 on BD type

* a test

* wip

* 3 arithmetic ops

* /

* wip

* BigDecimalPowerOp

* wip

* mod test

* NumericBinaryOpReturningBigDecimal

* with scalar

* misc arithmetic tests

* fix integralBigDecimalToInteger

* mixed columns

* bigdecimal pow via double

* cleanup

* j2e on get

* arithmetic exception

* mod 0

* cleanup

* fmt

* changelog

* check type first

* merge

* mc error message

* add BD case to Builder.java

* fmt

* changelog

* add BD case to StorageConverter.java

* fmt

* fix test
2024-06-04 13:59:31 -04:00
AdRiley
7c35781a14
Add select/remove column by type (#10159)
* stash

* Working first pass

* Tests

* remove columns

* more tests

* More tests

* DB_Table

* In DB

* Deprecate

* Update documentation

* More docs

* Remove strict

* Remove tests

* Fix widgets

* Add doc

* Spaces

* Fix In-DB
2024-06-04 16:23:46 +01:00
marthasharkey
24d209abf9
Add Columns_To_Add type to move away from Integer | Nothing (#10152)
Adds new type "Coulmns_To_Add" that replaces the Integer | Nothing type
And adds new widget to split_to_columns and tokenise_to_columns
<img width="397" alt="image" src="https://github.com/enso-org/enso/assets/170310417/1b155682-0992-4cc0-8964-62b389ee7072">

Closes #10041
2024-06-04 15:06:08 +00:00
Radosław Waśko
7cf80f3196
Handle UTF BOM when decoding text (#10130)
- Improve BOM handling: detect and skip the BOM character, Default encoding that detects encoding based on BOM if present, warnings if unexpected BOM is encountered.
- Closes #9849
- Windows-1252 fallback will be done as a separate PR as it has additional complexity. Tracked in ticket #10148.
2024-06-04 13:22:19 +00:00
GregoryTravis
1c2955655c
Add no_warning_propagation flag to Vector.build and Builder.new (#10079)
* no_warning_propagation flag

* doc

* benchmarks

* change to propagate_warnings, test

* fix java benchmark

* add benchmark to main
2024-06-04 09:06:09 -04:00
AdRiley
af4177e61f
Make table default visualisation (#10128)
Makes table the default vis type.

Before:
![image](https://github.com/enso-org/enso/assets/1720119/8a5d113a-ac06-4dd8-afdb-151cdbc149ab)

After:
![image](https://github.com/enso-org/enso/assets/1720119/d7c33cf7-355d-4517-8967-1963ff506f97)

Closes #10093
2024-05-31 15:02:02 +00:00
AdRiley
06327f8fde
Add statistic product (#10122)
Add Statistic.Product

![image](https://github.com/enso-org/enso/assets/1720119/f7fc7bb5-9efe-4dbe-9150-cd9e5101c553)
2024-05-31 09:29:52 +00:00
Pavel Marek
270f708e4f
Implement private methods (#10060)
Add support for private methods. Most of the changes are in parser and compiler. The runtime checking of private functions was already present since #9692

# Important Notes
- Only top-level methods can be declared `private`.
- private method cannot be called from different project
- private method cannot be accessed from polyglot code (private method does not exist for polyglot code)
2024-05-31 08:00:20 +00:00
Jaroslav Tulach
c4bf9182aa
Give qualified name of type a location (#10138)
Tests and fixes #9084.
2024-05-30 19:15:46 +00:00
Jaroslav Tulach
b2cf2f75fb
Extract the State from thunk's scope (#10120) 2024-05-30 17:24:06 +02:00
Jaroslav Tulach
6897f0d30c
Suspended arguments need one more child scope (#10104) 2024-05-28 17:53:08 +02:00
AdRiley
28bd5c522b
Add named_pattern and make it usable in Text_Replace (#10040)
Creates a new type Named_Expression and allows it to be used in text_replace

![image](https://github.com/enso-org/enso/assets/1720119/673a62a1-1ce5-4e1f-8289-0fa10e87b9de)

![image](https://github.com/enso-org/enso/assets/1720119/2f555e9b-d8c5-41e3-8000-d959d5818666)

![image](https://github.com/enso-org/enso/assets/1720119/d2ae2b0f-5bc4-4e9e-a391-402b58ee72d5)

![image](https://github.com/enso-org/enso/assets/1720119/3bf0547f-c2c7-4987-a45e-d922be2b0bae)
2024-05-27 13:01:26 +00:00
James Dunkerley
ab4b1f0f35
Add day_of_week and day_of_year to Column and DB_Column (#10081)
- Adds support for getting the weekday as an integer (1 Monday - 7 Sunday - ISO standard).
- Add support for getting the day of year.
2024-05-27 11:29:25 +00:00
James Dunkerley
d8059fd22c
Some tweaks following Steve's testings (#10042)
- Add ranged number widget to `at` and `get`.
- Add defaults to `at` and `get` picking the first item.
- PRIVATE on various Excel_Workbook methods. It still works like a connection but not shown in CB.
2024-05-27 09:04:29 +00:00
Radosław Waśko
9d75f79ff9
Cloud Integration updates: renames in file metadata structure, remove path resolver workaround, partial fix for datalink resolver issue (#10050)
- Supersedes #9966 as I wanted to test these changes in one go.
- Fixes #10037 caused by lack of CI check and my oversight (forgot to run full tests after a minor change).
- Fixes a regression after [file metadata fields were renamed](c09d856ac8 (diff-9f59b6a0ee3155efecdc70c1ea0c90ab5cde00b5623d84363118b1793f941c46R2037)).
- Fixes handling of creating new datalinks and using them after cache was cleared (e.g. workflow restart).
- This was caused by troubles with path resolver.
- The fix addresses the most common issue and adds a test for it (test flushes the caches to ensure path resolver is used instead of the cached value).
- Some related issues were discovered on the cloud side, tracked by https://github.com/enso-org/cloud-v2/issues/1252
2024-05-23 16:05:48 +00:00
Jaroslav Tulach
16c1b74218
Enso Library Feature to execute (a bit of) Base_Tests (#9997) 2024-05-23 08:20:19 +02:00
Radosław Waśko
1e0649fda1
Improvements to Table.union (#9968)
- Closes #9952
2024-05-22 09:38:10 +00:00
AdRiley
c7476c10f3
Add data cleanse component (#9879)
* Signature

* More API work

* Red

* Still red

* Green

* Red

* Green

* Red

* Green

* Refactor

* Refactor

* Refactor

* Red

* Green

* Red

* Green

* Red

* Green

* More tests

* non-ascii

* Ordering tests

* Remove tabs

* Numbers and Letters

* Changelog.md

* Add documentation

* Tests for non-text columns

* Add punctuation

* Add symbols

* Fix

* Refactor

* Refactor

* Move to base

* Fix

* Start of in-db tests. Not working

* DB versions

* Update widget

* Fix widgets

* Move tests to base

* Code review changes
2024-05-20 09:33:18 +01:00
GregoryTravis
4d49b00375
Combine builders for Vector.build and Vector.new_builder (#9922)
We have decided to keep the old new_builder, and to combine the new and old builders into one builder.
2024-05-17 16:18:47 +00:00
Radosław Waśko
38ad7b0afa
Creating datalinks from code (#9957)
- Closes #9673
- Adds the ability to save an existing Postgres connection as a datalink into the Enso Cloud, automatically promoting plain-text passwords into a Secret.
- Fixes dataflow error propagation in `JS_Object.from_pairs`.
2024-05-16 13:29:41 +00:00
Radosław Waśko
a2481036e6
Prepare for the Path resolver change to GET (#9958)
- Prepares for the Cloud API change, together with a fallback for the old API to avoid problems during migration.
- This PR should be merged before the https://github.com/enso-org/cloud-v2/pull/1236 PR is _deployed_.
2024-05-15 13:00:32 +00:00
Michał Wawrzyniec Urbańczyk
7982935c2f
[CI] Fix issues with checking PRs created against repository forks (#9926)
This PR fixes several issues that were appearing when running CI jobs on PRs created against the repository forks:
* electron-builder on Windows and macOS will properly recognize that the secrets are missing and will not attempt to sign the artifacts;
* similarly, fixed the S3 library tests;
* test reporter step will be now skipped, as it does not support forks.
2024-05-14 18:38:44 +00:00
James Dunkerley
b2aeb9fc84
Moving away from Integer | Nothing to Rows_To_Read for limiting number of rows. (#9925)
- Added a new `Rows_To_Read` type with conversions from `Nothing` and integers.
- Updated `read` on `Table`, `Column`, `DB_Table` and `DB_Column`.
- Updated `Delimited_Format.Delimited` to use `Rows_To_Read` for `row_limit`.
- Updated `Excel_Format.Sheet` and `Excel_Format.Range` to use `Rows_To_Read` for `row_limit`.
- Updated `Excel_Workbook.read` to use `Rows_To_Read`.
- Updated `Connection.read` (in all connection types) to use `Rows_To_Read`.

![image](https://github.com/enso-org/enso/assets/4699705/553c027f-f4c3-4855-9f51-2c4bcaec48a0)

![image](https://github.com/enso-org/enso/assets/4699705/a06c3912-77e0-4c10-abb8-73aed667458d)
2024-05-14 16:31:26 +00:00
James Dunkerley
2d6735edc7
Allow a Table to be used to rename_columns. (#9940)
Following an example from Steve, adjusted `rename_column` to allow a `Table` to define the mapping.

- A single Text column table gives a list of new names.
- A two Text column table gives a map from old name to new name.
- If a `DB_Table` is passed errors and tells the user to materialize the table.

![image](https://github.com/enso-org/enso/assets/4699705/3ed98330-1fce-465e-bf96-4a86e0872dd3)
2024-05-14 15:57:10 +00:00
Radosław Waśko
5f0a16c87c
Audit Logs for Postgres connections opened through a data link (#9873)
- Closes #9599
- Implemented API for sending audit logs to the cloud on a background thread.
- If the Postgres connection is opened through a datalink, its internal JDBC connection is replaced by a wrapper that reports executed queries to the audit log.
- Also introduces `EnsoMeta` - a helper Java class that can be used in our helper libraries to access Enso types.
- I have replaced the common pattern scattered throughout the codebase with calls to this 'library' to avoid repetitive code.
- Refactored `Table.display` to share code between in-memory and DB - it was needed as the function stopped working for `DB_Table` after adding making the `Table` constructor `private`.
- Clearer error when reading a SQLite database from a remote file (tells the user to download it first).
- Follow up - correlate asset id of the data link:
#9869
- Follow up - include project name (once bug is fixed):
#9875
- Some problems/improvements of the audit log:
- The audit log system is not yet ready for high throughput of logs
#9870
- The logs may be lost if `System.exit` is used
#9871
2024-05-11 08:54:33 +00:00
James Dunkerley
d97754da17
Error messages for rename_columns and Vector.duplicates (#9917)
- Improve error message for `rename_columns`.
- Add `length` to `Set` and `Map`.
- Add `duplicates` to `Vector` (and `Array`).

![image](https://github.com/enso-org/enso/assets/4699705/623df253-52e8-4bdc-a69c-ac8dc3ca594e)
2024-05-10 17:43:50 +00:00
GregoryTravis
720d32cbe3
Convert most remaining uses of Vector.new_builder to Vector.build (#9891) 2024-05-09 15:52:53 +00:00