Commit Graph

394 Commits

Author SHA1 Message Date
James Dunkerley
b2545b8b96
Small tweaks from using Process.run (#11110)
* Once more unto the lint dear friends.

* Default the arguments for process run.
Allow delimiter to be used to convert Text to Table.
2024-09-17 14:27:25 +01:00
AdRiley
b8516f7b4a
Make all Common_Spec tests pass for SQLSerever (#10810)
* More SQL tests

* Remove duplicate

* Fix another test

* Temp fix

* More temporay fixes

* More green

* Another green

* Another one

* Fix counts

* Temporary change

* Fix tests

* Auto-commit work in progress before clean build on 2024-09-06 10:53:46

* Another fix

* Green

* Specialise ensure_query_has_no_holes

* cleanup

* Cleanup

* Cleanup

* Clean

* Code review changes
2024-09-12 10:08:55 +01:00
AdRiley
a666585afe
Add update mode to Running (#11045) 2024-09-11 17:41:33 +03:00
GregoryTravis
97a1628017
Add a banker's rounding primitive for Snowflake (#11037)
* log table_builder create

* fix log, 38,12

* fix scale in rounding tests

* shorten some decimals

* fix test

* dialect flags

* revert native bankers

* missing import

* sf builtin

* runs

* works on +/-2/4.5

* remove extreme precision limit tests

* remove another extreme precision test, and restore batching for bankers rounding tests

* cleanup

* cleanup

* cleanup, comment
2024-09-11 10:29:41 -04:00
James Dunkerley
7b4b635fa8
Add Row Number starts at 0 and other tweaks. (#11017)
- Change default to be from 0.
- Alter widgets so always shown.
- Tweaks to the Welcome screen following review with Product Team.
- Use 0 for default seed option in `Index_Sub_Range.Sample`.
- Add ALIASES for aggregate and running.
- Add more imports for `Widgets.enso` to fix extension methods not working.
2024-09-09 20:26:59 +00:00
GregoryTravis
2adb8bb86d
Use NUMBER(38, 12) as default range for Snowflake DECIMALs (#10971) 2024-09-09 18:31:09 +00:00
GregoryTravis
6f97e8041b
Upgrade SQLite to 3.46 (#10911)
Before 3.46, the SQLite parser had a limited stack, which could overflow for certain complex queries.

CTE optimizations make some of our queries much smaller, but also a little bit more deeply nested, causing the parser stack to overflow. 3.46 removes this stack limitation.

Closes #10910.
2024-09-07 17:58:59 +00:00
James Dunkerley
e6ea3b2607
Default Project, Templates and Docs (#10980)
- Linting updates.
- Add an `Examples.welcome` and adjust the start up project to use it.
- Merge all of Cass's work into the source code.
- Make example render in mono space font.
2024-09-05 14:36:08 +00:00
James Dunkerley
91226be378
Small tweaks from QA (#10941) 2024-08-31 09:04:52 +02:00
GregoryTravis
8260a9587f
Column-level lexically-scoped CTE expressions (#10826)
This implements `DB_Column.with`, which uses `WITH ... AS` SQL clauses to remove duplicates in the generated SQL.

After a discussion with @radeusgd, we concluded that we will probably want a more complete CTE implementation, so this one is useful for now to deal with big queries (like `round`).

# Important Notes
Still to do in this PR:

- [x] Rename `with` to `let` (or something similar)
- [x] tests
- [x] documentation
- [x] remove `State` hack by moving query generation into a class and using a `Ref` field for scoping

Results on `round_float`:

| --- | SQL length in characters (unprettified) | SQL length in lines (prettified) |
| --- | --- | --- |
| Without CTEs | 13193 | 851 |
| With CTEs | 3644 | 187 |

Compare the SQL:

[without-ctes.sql.txt](https://github.com/user-attachments/files/16629356/without-ctes.sql.txt)
[with-ctes.sql.txt](https://github.com/user-attachments/files/16629357/with-ctes.sql.txt)

Update, with name shortening:

| --- | SQL length in characters (unprettified) | SQL length in lines (prettified) |
| --- | --- | --- |
| Without CTEs | 13193 | 853 |
| With CTEs | 2427 | 176 |

[without-cte.txt](https://github.com/user-attachments/files/16694328/without-cte.txt)
[with-cte.txt](https://github.com/user-attachments/files/16694327/with-cte.txt)
2024-08-28 18:23:51 +00:00
Radosław Waśko
80922dcd01
First/Last support in Snowflake, and partially in SQLite (#10841)
- Closes #10411
- Also a similar technique allowed to add partial support for SQLite as well.
- Only `include_nothing=True` case is still not supported as the technique used for Snowflake did not port well to SQLite and I did not find yet a satisfactory workaround (it is doable, but too ugly to justify it for now IMHO).
2024-08-28 10:00:07 +00:00
Radosław Waśko
2e5f011f3a
Move Connection classes out of Internal (#10886)
- Closes #10848
2024-08-28 08:15:07 +00:00
marthasharkey
b2237f77c7
Add drilldown to XML (#10824)
This adds the drilldown behaviour to XML_Elements.

![10557-xml-drilldown](https://github.com/user-attachments/assets/062d7de9-c157-4ea2-8a9b-a39f91c26581)
2024-08-19 10:12:42 +00:00
James Dunkerley
d6ca3ea47b
Add Regex link to all regex functions. (#10825)
Adds links to Regex documentation for all regex functions.
2024-08-15 16:21:19 +00:00
James Dunkerley
422fa8c16b
Adding support for creating Atoms in expressions (#10820)
- Enables the `..` autoscoping style for creating Atoms in expressions.
- Add type checking to methods in columns.
- Auto wrap returns from method in expressions into a column as needed.
- Remove `Time_Period.Day` to remove confusion..
2024-08-15 15:52:30 +00:00
AdRiley
9fc6af12c5
Clean up Text option from read and query (#10790)
* Revert

* from

* Fix

* fix

* Last few details

* Fix

* Fix test

* Fix

* Cleanup

* Fix error message

* fix

* Fix

* Improve error message
2024-08-13 13:25:21 +03:00
marthasharkey
0d5d01a294
Wip/mk/drive drilldown from enso (#10724)
This moves some of the drilldown logic to the enso code and utilises the new 'get_rows' method for Table
2024-08-12 12:51:51 +00:00
AdRiley
3581fd8e21
Make sqlserver temp tables work (#10781)
* Temp table support

* Fix

* Add query

* Clean up

* Clean up

* Fix

* Code review changes

* Added comment

* Code review changes

* Add execute
2024-08-09 13:06:16 +03:00
AdRiley
0f688d0a25
Add newlines option to text cleanse/replace (#10761)
* Auto-commit work in progress before clean build on 2024-08-06 11:32:46

* Fixed Regex and additional test

* changelog

* .

* Make non-capturing
2024-08-06 18:59:54 +03:00
Radosław Waśko
3fd14642d9
Fix upload/delete transactions in Snowflake backend (#10738)
Fixes #10609 by rewriting all our upload-related operations to rely on `DDL_Transaction` - an abstraction that handles 'transactionality' of `CREATE TABLE` statements dependent on if a given backend allows DDLs within transactions or not (if not it emulates transactionality by creating the tables outside of transaction and then dropping them on rollback).
2024-08-06 08:14:44 +00:00
Radosław Waśko
6ad3faf03b
Refactor Upload_Table to be more readable: split into separate smaller submodules (#10713)
- First step of #10609 - before I actually modify it, I decided I need to refactor the `Upload_Table` logic as it was quite convoluted. Doing this as a separate PR for easier review. A big 600+ line file was replaced by several smaller ones grouped by topics.
- Practically no changes apart from moving stuff into separate modules.
- One small change - added `Missing_Argument` to `SQL_Query` as I noticed that lack of defaults was giving rise to confusing errors when working with `query` in the GUI.

Before:
![image](https://github.com/user-attachments/assets/b586caec-f25c-406e-be5a-d402f10feb86)
After:
![image](https://github.com/user-attachments/assets/6b1d4206-05b1-4587-b3e1-43ca95ea7c2e)
![image](https://github.com/user-attachments/assets/58c098bd-db0c-4ee2-823c-bf5c9e758ce4)
2024-07-31 09:54:17 +00:00
AdRiley
0c552489e3
Add Initial SQL Server support (#10624)
* Squash all commits to resolve merge conflicts

* Fix merge problems

* Merge fix

* Fix port

* Fix warning

* cargo fmt

* legal review

* Small fixes

* Update instructions

* Code review feedback

* Cleanup

* typo

* Fix

* Remove leftover snowflake code

* Remove comment

* Add underscore

* Type cleanup

* Code review fix

* Cleanup

* Add datetime roundtrip test

* add comment

* drop

* Refactor

* Refactor

* Fix merge

* Fix

* Fix

* fix

* Add comment
2024-07-30 11:13:08 +01:00
James Dunkerley
74acc1de24
Tweaks from client Demo (#10685)
- Adjusted Filter_Condition removing keep/drop from basic filters.
- Fix Is_In to have selector.
- Fix for Date simple expressions.
- Add get_row to Table and DB_Table.
2024-07-26 18:44:36 +00:00
Radosław Waśko
46e8bab429
Fix aggregate after sort in DB (#10677)
* add test for https://github.com/enso-org/enso/issues/10321

* wrap in subquery if ordering was present to fix the bug

* adding one more test just to be sure

* fix import
2024-07-26 08:43:19 +01:00
Kaz Wesley
e5b85bf16e
Space-precedence does not apply to value-level operators (#10597)
In a sequence of value-level operators, whitespace does not affect relative precedence. Functional operators still follow the space-precedence rules.

The "functional" operators are: `>> << |> |>> <| <<| : .`, application, and any operator containing `<-` or `->`. All other operators are considered value-level operators.

Asymmetric whitespace can still be used to form *operator sections* of value-level operators, e.g. `+2 * 3` is still equivalent to `x -> (x+2) * 3`.

Precedence of application is unchanged, so `f x+y` is still equivalent to `f (x + y)` and `f x+y * z` is still equivalent to `(f (x + y)) * z`.

Any attempt to use spacing to override value-level operator precedence will be caught by the new enso linter. Mixed spacing (for clarity) in value-operator expressions is allowed, as long as it is consistent with the precedences of the operators.

Closes #10366.

# Important Notes
Precedence warnings:
- The parser emits a warning if the whitespace in an expression is inconsistent with its effective precedence.
- A new enso linter can be run with `./run libraries lint`. It parses all `.enso` files in `distribution/lib` and `test`, and reports any errors or warnings. It can also be run on individual files: `cargo run --release --bin check_syntax -- file1 file2...` (the result may be easier to read than the `./run` output).
- The linter is also run as part of `./run lint`, so it is checked in CI.

Additional language change:
- The exponentiation operator (`^`) now has higher precedence than the multiplication class (`*`, `/`, `%`). This change did not affect any current enso files.

Library changes:
- The libraries have been updated. The new warnings were used to identify all affected code; the changes themselves have not been programmatically verified (in many cases their equivalence relies on the commutativity of string concatenation).
2024-07-24 10:55:44 +00:00
Radosław Waśko
7fd8701690
Snowflake Dialect pt. 6 - Union, Distinct and other improvements (#10576)
- Part of #9486
- Fixes `Table.union`, `merge` and `distinct` tests
- Replaces `distinct_on` in `Context` that was actually a Postgres specific addition leaking into the base with a more abstract `Context_Extension` mechanism.
- This allows us to implement the Snowflake-specific `DISTINCT` using `QUALIFY`.
2024-07-19 16:04:00 +00:00
James Dunkerley
15810a4675
Linting fixes. (#10589)
Generating Doc data so fixed linting.
2024-07-18 12:42:34 +00:00
James Dunkerley
771402acf6
Widget Improvements (#10575)
- New centralised `make_any_selector` for creating dropdown for various types.
- Updated `Filter_Condition.default_widget` to be auto-scoped.
![image](https://github.com/user-attachments/assets/2786d512-6716-4d4b-b119-15054611c11c)
- Nested structure for Simple_Expressions.
![image](https://github.com/user-attachments/assets/09eda332-0888-41e7-8462-fc8566959da5)
2024-07-17 19:53:17 +00:00
AdRiley
cd1d61710f
Rename run workflow button to write (#10568)
Request from Product team to not have a run button. Renaming it to write
2024-07-16 18:04:31 +00:00
Radosław Waśko
a30b0c60eb
Snowflake Dialect pt. 5 (#10528)
- Related to #9486
- Batching of expression tests
- Fixing arithmetic by simplifying `%` and `/` operations
- Trying to share some more tables, sometimes improving performance sometimes not really
- Adding sorting and other fixes to tests to make them pass: Missing_Values_Spec, Filter_Spec, Map_Spec
- Fixing warnings related to materialization of Decimal->Integer, thus fixing Join_Spec.
2024-07-16 09:38:57 +00:00
Radosław Waśko
632355f85b
Snowflake Dialect pt. 4 - reading a column of small integers as Integer type, other type mapping tests (#10518)
- Related to #9486
- Ensures that even though an integer column in Snowflake is represented by `Decimal` type, if the values are small enough, they are materialized as `Integer`.
- If the values are larger, they are still read in as `Decimal`.
- Adds tests for some other `Decimal` edge cases (various precisions and scales), and for `Float`.
2024-07-11 20:14:46 +00:00
Pavel Marek
0f9852aab2
Replace all from ... export all with explicit exports (#10369)
Replace all exports with explicit exports.
2024-07-11 19:34:25 +02:00
Jaroslav Tulach
220b40a1cd
Enforce conversion method return type & introduce Comparable.new (#10468) 2024-07-11 06:58:51 +02:00
Adam Obuchowicz
60c1a0e1f6
Record control tweaks (#10509)
Fixes #10388

[Screencast from 2024-07-10 13-55-46.webm](https://github.com/enso-org/enso/assets/3919101/4ad7c039-84f6-4e42-aad9-5e287ccd88bb)

# Important Notes
Also, removed unnecessary  setExecutionEnvironment call on startup
2024-07-10 20:19:23 +00:00
Radosław Waśko
48c17845a7
Fixing Database tests and Snowflake Dialect - part 3 out of ... (#10458)
- Related to #9486
- Fixes types in literal tables that are used throughout the tests
- Tries to makes testing faster by disabling some edge cases, trying batching some queries, re-using the main connection and trying to re-use tables more
- Implements date/time type mapping and operations for Snowflake
- Updates type mapping to correctly reflect what Snowflake does
- Disables warnings for Integer->Decimal coercion as that's too annoying and implicitly understood in Snowflake
- Allows to select a Decimal column with `..By_Type ..Integer` (only in Snowflake backend) because the Decimal column there is its 'de-facto' Integer column replacement.
2024-07-10 13:21:30 +00:00
GregoryTravis
71a6e2162e
Fix return type for Postgres Decimal division (#10479) 2024-07-09 15:22:14 +00:00
James Dunkerley
4b3e4ae15e
Rename Map to Dictionary and Set to Hashset. (#10474)
- Rename `Map` to `Dictionary`.
- Rename `Set` to `Hashset`.
- Add a deprecated place holder for the static method of `Map`.
2024-07-09 09:12:23 +00:00
James Dunkerley
4c0647ea29
Stop publishing First/Last as constructors and use auto-scoping for take and drop. (#10467)
- Removes `First` and `Last` from the `Standard.Base` exports.
- Enable auto-scoping for all `Index_Sub_Range` and `Text_Sub_Range`.
- Update all use of those methods to use auto-scoping.
2024-07-08 10:26:30 +00:00
James Dunkerley
018d4c312f
Stop publishing Postgres constructor, update Postgres_Details.Postgres to Postgres.Server. (#10466)
![image](https://github.com/enso-org/enso/assets/4699705/6d0d4167-e97b-4765-8079-650ad091ce60)

- Rename `Postgres_Details` to `Postgres`.
- Rename `Postgres` constructor to `Server`.
- Update SPI.
- Linting issues (indent, missing doc comment)
2024-07-08 07:58:08 +00:00
James Dunkerley
c2c4b95116
Final step removing the Problem_Behavior publishing. (#10461)
- Remove publishing the constructors.
- Fix any missed use in libs.
- Alter tests to generally use auto-scoped calls.
- `on_incomparable` to `on_problems`.
2024-07-05 18:41:36 +00:00
James Dunkerley
2124cd022a
Batch Two of Problem_Behavior (#10454)
Part 2 of removing the public constructors.
Completes review of the libraries.

Tests left to do then can remove the export.
2024-07-05 13:40:29 +00:00
AdRiley
fc93b4d121
Refactor database dialect types (#10437)
* Auto-commit work in progress before clean build on 2024-07-03 14:17:22

* Refactor

* Revert

* revert

* Code Review feedback

* Green

* 2 Red

* Green

* Renames

* Code review changes

* Code review changes
2024-07-05 13:08:25 +01:00
James Dunkerley
0661f17d1c
Tune Text.trim, fix for Text.split (#10445)
- Rename `Location.Start` to `Location.Left`.
- Rename `Location.End` to `Location.Right`.
- Use auto-scoping for `Location`.
- Tune widgets for `Text.trim`.
- Correct signature of `Text.split`.
- Adjist `generateLocallyUniqueIdent` to not fail on bad signature.
2024-07-04 22:24:56 +00:00
James Dunkerley
9a2aed92f1
Separating list from read function and other small tweaks. (#10434)
- Rename `Faker.string_value` to `Faker.text_value`.
- Remove `Regex.pattern_string` as duplicate of `Regex.pattern`.
- Sort the Date picker.
- Rename `Data.list_directory` to `Data.list`.
- Remove support for reading a directory.

![image](https://github.com/enso-org/enso/assets/4699705/b42bb3c9-e63b-49f2-8cdc-4666cb6d968e)

![image](https://github.com/enso-org/enso/assets/4699705/97f49891-5ae5-4f0a-9a41-6200888fcd86)
2024-07-03 22:00:53 +00:00
marthasharkey
ee39fd7f53
Visualise excelwork book sheets, db tables/schemas and add node adding gets to query them (#10362)
changes to current table (and db table) on index click:
- node added on double click
- tooltip added
- cursor changes to pointer
- styled as a link

![7473-table-w-links](https://github.com/enso-org/enso/assets/170310417/5e60c177-3f83-4db7-be86-fa8a9d493204)


Row table types:
Added on click to show value
![7473-row-links](https://github.com/enso-org/enso/assets/170310417/82f878ea-420d-4308-99bf-c77a6340a8c3)



Excel Workbook connections
- show sheet names in column
- sheet names are clickable that add a 'read' node for the corresponding sheet
- Shown in table with header title: Sheets

![7473-excel-links-sheets](https://github.com/enso-org/enso/assets/170310417/748f524e-5cca-4c20-b458-132af9a57ec1)


SQLite Connection:
- shows available schemas in column
- schemas are clickable that add a 'query' node for the corresponding schema/table
![7473-sql-lite-links](https://github.com/enso-org/enso/assets/170310417/21a8006f-c462-4128-874d-05380d3bab00)


Postgres Connection:
- shows available tables in a column
- tables are clickable and add a 'query' node for the corresponding table
![7473-postgres-links](https://github.com/enso-org/enso/assets/170310417/66b134ff-80fd-4995-b445-505919e25cfa)

JS_Object
- style keys as links
![7473-json-links](https://github.com/enso-org/enso/assets/170310417/9ec17c85-c1f4-42ab-b40a-06d6bc29d54f)
2024-07-03 13:58:07 +00:00
GregoryTravis
48fb999eb3
Implement Decimal support for Postgres backend (#10216)
* treat scale nothing as unspecifed

* cast to decimal

* float int biginteger

* conversion failure ints

* loss of decimal precision

* precision loss for mixed column to float

* mixed columns

* loss of precision on inexact float conversion

* cleanup, reuse

* changelog

* review

* no fits bd

* no warning on 0.1 conversion

* fmt

* big_decimal_fetcher

* default fetcher and statement setting

* round-trip d

* fix warning

* expr +10

* double builder retype to bigdecimal

* Use BD fetcher for underspecified postgres numeric column, not inferred builder, and do not use biginteger builder for integral bigdecimal values

* fix tests

* fix test

* cast_op_type

* no-ops for other dialects

* Types

* sum + avg

* avg + sum test

* fix test

* update agg type inference test

* wip

* is_int8, stddev

* more doc, overflow check

* fmt

* finish round-trip test

* wip
2024-07-02 15:01:55 -04:00
Radosław Waśko
db4f7ab3b5
Fixing Database tests and Snowflake Dialect - part 2 out of ... (#10319)
- Part of #9486
- Fixing our tests to not rely on deterministic ordering of created Tables in Database backends
- Before, SQLite and Postgres used to mostly return rows in the order they were inserted in, but Snowflake does not.
- Fixing various parts of Snowflake dialect.
2024-06-27 14:54:00 +00:00
James Dunkerley
d92078471b
Rename order_by to sort for Table and DB_Table. (#10372)
- Rename `order_by` to `sort` for `Table` and `DB_Table`.
- Added deprecated placeholder.
- Fixed a couple of minor deprecated mistakes.

![image](https://github.com/enso-org/enso/assets/4699705/96c32fa7-33e5-400a-9d3a-ebf330886911)
2024-06-26 17:46:09 +00:00
AdRiley
6bc7d4f786
Fix signatures for distinct and duplicates (#10365)
Fixes #10364
2024-06-25 12:59:27 +00:00
James Dunkerley
e6c8ec7ab5
Changes from session with Ned (#10349)
- Removed `second_row` and `second_column` from the `Table` and `DB_Table`.
- Added `first_value` and `last_value` to the `Table` and `DB_Table`.
- Fixed bug where negative index access wasn't allowed on `Column`.
- Added error if negative index access used on `DB_Column`. Tells user they have to materialize.
- Fix argument order for `Table.text_cleanse` and a couple of typo corrections.
- Rename `auto_value_type` to `auto_cast` on table and columns.
2024-06-24 12:47:14 +00:00