Commit Graph

392 Commits

Author SHA1 Message Date
AdRiley
1b6a1f990b
Enable SQLServer Sort Feature and associated tests (#11379)
* Auto-commit work in progress before clean build on 2024-10-15 12:59:18

* checkpoint

* Clean

* Cleaner

* cleaner

* Cleaner

* Green

* stash

* stash

* Fix sort

* Sub 300 failures

* More passing tests

* Auto-commit work in progress before clean build on 2024-10-22 09:00:41

* SQLServer green 265

* remove dead code

* Add doc

* Add generate_column

* Refactor

* refactor

* Refactor

* Refactor

* Add  is_operation_supported_fn

* Fix

* Fix

* Fix

* Code review changes

* Code review feedback

* typos

* Add comment
2024-10-25 13:16:52 +03:00
Radosław Waśko
ca9df70ebf
Saving data links to a DB_Table (#11371)
- Closes #11295
2024-10-24 13:18:49 +00:00
Kaz Wesley
33904912ee
Move annotations into fields of Function and ConstructorDefinition (#11374)
Move annotations into fields of Function and ConstructorDefinition.

# Important Notes
New syntax: Constructor argument-definition lines
- Each argument in a type-constructor definition may be specified on its own (indented) line.

Relaxed syntax: Unparenthesized arguments to annotations
- A generic annotation now uses the rest of the line as its argument expression; the expression no longer needs to be parenthesized.
2024-10-23 20:35:06 +00:00
Radosław Waśko
d75e20c1d2
Save Database connection as data link, SQL Server data link support (#11343)
- Closes #11294
2024-10-17 09:06:57 +00:00
Radosław Waśko
244effde0c
Enable audit logs for Snowflake backend (#11306)
- Closes #11292
- Tries to fix #11300
2024-10-14 14:30:42 +00:00
AdRiley
03369b9273
Refactor Dialect_Flag (#11273)
* Add property flags

* Update post initial design review

* Add more tests

* Fix tests

* Remove .LCK

* Fix
2024-10-14 10:32:19 +03:00
Radosław Waśko
2843dcbf4a
When connecting to a Postgres database through a datalink stored on Enso Cloud, its asset ID is included in the audit logs (#11291)
- Closes #9869
2024-10-10 15:18:47 +00:00
AdRiley
26c711ab14
Enable SQLServer select columns (#11270)
* Enable SQLServer select columns

* Code review feedback

* Code Review feedback
2024-10-08 22:57:36 +03:00
AdRiley
d0f0770ef4
Add DB_Table feature flags (#11163)
* Refactor

* Fix

* More fixes

* Fixes

* Fix

* Fix imports

* Fix

* Fix

* Fix Test

* Fix

* Fix

* is_feature_supported API

* Add SQLServer Tests

* SQLServer Select_columns functional

* SQLServer Filter functional

* Aggregate

* order feature

* Remove old order flag

* Join

* Union

* Distinct

* Text_cleanse

* Remove Nothing tests

* add_row_number and integration

* Make_table_from

* Feature flag failing tests

* Column_operations

* Set

* date_time off

* Conversion

* More flags

* Merge

* Expressions

* TakeDrop

* Missing

* Cross_join

* More flags

* Temporaily disable

* Fix merge issue

* Update other dbs

* Code review changes

* Code review changes

* Code revoew changes

* Fix

* Fix

* Fix

* Fix

* Fix

* Code review fix

* Typo

* Code review changes

* Casing

* Add comments

* Add comments

* Update

* Fix

* Fix imports
2024-10-07 16:05:09 +01:00
AdRiley
a4f800dbf0
Add Coalesce tests and fix In-Memory version (#11242)
Add tests for coalesce and fix the In-memory version

# Important Notes
The distribution/lib/Standard/Database/0.0.0-dev/src/Internal/IR/From_Spec.enso change isn't actually needed for this MR as I switched away from using literal tables for these tests as trying to mix 2 literal tables doesn't look to be supported. The change I added will let any future developers know that.
2024-10-07 09:02:11 +00:00
James Dunkerley
fd72ab7052
Remove some catch alls (#11254)
- Allow Interrupted Exceptions to float out of the web requests.
- Use `Type_Error` rather than Any when catching auto scoping resolving.
- Rename `Java_Exception` to `JException`
2024-10-04 21:32:56 +00:00
James Dunkerley
a2c853c6e3
Small Fixes (#11236)
- Left over from the last PR.
- Widget for `headers` in `URL.fetch` extension.
- Fix `Left` and Right` to allow columns or expressions in it.
2024-10-02 18:57:02 +00:00
Radosław Waśko
0399a4570d
Resolving ~ in Enso asset paths to user home (#11235)
- Closes #11226
2024-10-02 18:49:43 +00:00
James Dunkerley
6ea716f1b3
Widget for Database.connect (#11216)
- Use auto-scoping for Redshift, SQLServer and Snowflake.
![image](https://github.com/user-attachments/assets/2f5ff24a-44f4-4e87-909a-e064b8653511)
- Fix for widgets on Header functions.
![image](https://github.com/user-attachments/assets/4384efcf-a4da-48b1-b571-1167ca8d0134)
- Move `Snowflake_Details` to `Standard.Snowflake.Connection` namespace to make widgets work.
- Add widgets to `Snowflake_Details`.
![image](https://github.com/user-attachments/assets/b51d0126-a768-4f4a-9d87-c42c8e91e26b)
- Typo fix for SQLServer SPI.
- Change SQLServer port to be an Integer and added default.
- Reordered parameters on SQLServer **(potentially breaking change)**.
- Added widgets to SQLServer.
![image](https://github.com/user-attachments/assets/1c744da4-7913-4a87-9e64-fc10442a06eb)
- Added widget for JDBC options (as well as conversion from Vector to options).
![image](https://github.com/user-attachments/assets/4958b1e4-4cbc-43e3-8381-64e5ce7ea8ff)
- Added header alias to `use_first_row_as_names`.
- Added various aliases to `read` and `write`.
2024-10-01 08:43:03 +00:00
James Dunkerley
28bbc34257
Widget support for Data.fetch and Data.post. (#11199)
- Add `pretty` for `Date` and `Time`.
- Add constructors for `JS_Object` and `Dictionary` to the component browser.
- Add widgets to `Dictionary` methods.
![image](https://github.com/user-attachments/assets/4f6c58d5-9eb5-40e5-96c1-2e06e23051d0)
- Add conversion from Vector to Dictionary.
- Add `pair` method shorthand for `Pair.Value`.
- Created widget for `Header`.
- Added widgets for `Data.fetch` and `Data.post`.
- Added widgets for `Request_Body` constructors.
- Update the Forbidden Operation message to be friendlier.
![image](https://github.com/user-attachments/assets/eaac5def-a91f-450f-b814-d776311962e3)

Video before fixing Forbidden Message:

https://github.com/user-attachments/assets/f9c4bde4-3f0a-49f1-a3ca-a0aaa3219286
2024-09-27 18:08:12 +00:00
AdRiley
0f4fa42eea
Refactor unsupported db error messages (#11156)
* Refactor

* Fix

* More fixes

* Fixes

* Fix

* Fix imports

* Fix

* Fix

* Fix Test

* Fix

* Fix
2024-09-27 11:05:14 +03:00
Gregory Michael Travis
a0c3b901ac
Do not default the primary key in select_into_database_table (#11120) 2024-09-25 14:34:36 -04:00
Gregory Michael Travis
859b572242
Better error message for NULLs in primary key columns (#11055) 2024-09-18 11:56:21 -04:00
James Dunkerley
b2545b8b96
Small tweaks from using Process.run (#11110)
* Once more unto the lint dear friends.

* Default the arguments for process run.
Allow delimiter to be used to convert Text to Table.
2024-09-17 14:27:25 +01:00
AdRiley
b8516f7b4a
Make all Common_Spec tests pass for SQLSerever (#10810)
* More SQL tests

* Remove duplicate

* Fix another test

* Temp fix

* More temporay fixes

* More green

* Another green

* Another one

* Fix counts

* Temporary change

* Fix tests

* Auto-commit work in progress before clean build on 2024-09-06 10:53:46

* Another fix

* Green

* Specialise ensure_query_has_no_holes

* cleanup

* Cleanup

* Cleanup

* Clean

* Code review changes
2024-09-12 10:08:55 +01:00
AdRiley
a666585afe
Add update mode to Running (#11045) 2024-09-11 17:41:33 +03:00
GregoryTravis
97a1628017
Add a banker's rounding primitive for Snowflake (#11037)
* log table_builder create

* fix log, 38,12

* fix scale in rounding tests

* shorten some decimals

* fix test

* dialect flags

* revert native bankers

* missing import

* sf builtin

* runs

* works on +/-2/4.5

* remove extreme precision limit tests

* remove another extreme precision test, and restore batching for bankers rounding tests

* cleanup

* cleanup

* cleanup, comment
2024-09-11 10:29:41 -04:00
James Dunkerley
7b4b635fa8
Add Row Number starts at 0 and other tweaks. (#11017)
- Change default to be from 0.
- Alter widgets so always shown.
- Tweaks to the Welcome screen following review with Product Team.
- Use 0 for default seed option in `Index_Sub_Range.Sample`.
- Add ALIASES for aggregate and running.
- Add more imports for `Widgets.enso` to fix extension methods not working.
2024-09-09 20:26:59 +00:00
GregoryTravis
2adb8bb86d
Use NUMBER(38, 12) as default range for Snowflake DECIMALs (#10971) 2024-09-09 18:31:09 +00:00
GregoryTravis
6f97e8041b
Upgrade SQLite to 3.46 (#10911)
Before 3.46, the SQLite parser had a limited stack, which could overflow for certain complex queries.

CTE optimizations make some of our queries much smaller, but also a little bit more deeply nested, causing the parser stack to overflow. 3.46 removes this stack limitation.

Closes #10910.
2024-09-07 17:58:59 +00:00
James Dunkerley
e6ea3b2607
Default Project, Templates and Docs (#10980)
- Linting updates.
- Add an `Examples.welcome` and adjust the start up project to use it.
- Merge all of Cass's work into the source code.
- Make example render in mono space font.
2024-09-05 14:36:08 +00:00
James Dunkerley
91226be378
Small tweaks from QA (#10941) 2024-08-31 09:04:52 +02:00
GregoryTravis
8260a9587f
Column-level lexically-scoped CTE expressions (#10826)
This implements `DB_Column.with`, which uses `WITH ... AS` SQL clauses to remove duplicates in the generated SQL.

After a discussion with @radeusgd, we concluded that we will probably want a more complete CTE implementation, so this one is useful for now to deal with big queries (like `round`).

# Important Notes
Still to do in this PR:

- [x] Rename `with` to `let` (or something similar)
- [x] tests
- [x] documentation
- [x] remove `State` hack by moving query generation into a class and using a `Ref` field for scoping

Results on `round_float`:

| --- | SQL length in characters (unprettified) | SQL length in lines (prettified) |
| --- | --- | --- |
| Without CTEs | 13193 | 851 |
| With CTEs | 3644 | 187 |

Compare the SQL:

[without-ctes.sql.txt](https://github.com/user-attachments/files/16629356/without-ctes.sql.txt)
[with-ctes.sql.txt](https://github.com/user-attachments/files/16629357/with-ctes.sql.txt)

Update, with name shortening:

| --- | SQL length in characters (unprettified) | SQL length in lines (prettified) |
| --- | --- | --- |
| Without CTEs | 13193 | 853 |
| With CTEs | 2427 | 176 |

[without-cte.txt](https://github.com/user-attachments/files/16694328/without-cte.txt)
[with-cte.txt](https://github.com/user-attachments/files/16694327/with-cte.txt)
2024-08-28 18:23:51 +00:00
Radosław Waśko
80922dcd01
First/Last support in Snowflake, and partially in SQLite (#10841)
- Closes #10411
- Also a similar technique allowed to add partial support for SQLite as well.
- Only `include_nothing=True` case is still not supported as the technique used for Snowflake did not port well to SQLite and I did not find yet a satisfactory workaround (it is doable, but too ugly to justify it for now IMHO).
2024-08-28 10:00:07 +00:00
Radosław Waśko
2e5f011f3a
Move Connection classes out of Internal (#10886)
- Closes #10848
2024-08-28 08:15:07 +00:00
marthasharkey
b2237f77c7
Add drilldown to XML (#10824)
This adds the drilldown behaviour to XML_Elements.

![10557-xml-drilldown](https://github.com/user-attachments/assets/062d7de9-c157-4ea2-8a9b-a39f91c26581)
2024-08-19 10:12:42 +00:00
James Dunkerley
d6ca3ea47b
Add Regex link to all regex functions. (#10825)
Adds links to Regex documentation for all regex functions.
2024-08-15 16:21:19 +00:00
James Dunkerley
422fa8c16b
Adding support for creating Atoms in expressions (#10820)
- Enables the `..` autoscoping style for creating Atoms in expressions.
- Add type checking to methods in columns.
- Auto wrap returns from method in expressions into a column as needed.
- Remove `Time_Period.Day` to remove confusion..
2024-08-15 15:52:30 +00:00
AdRiley
9fc6af12c5
Clean up Text option from read and query (#10790)
* Revert

* from

* Fix

* fix

* Last few details

* Fix

* Fix test

* Fix

* Cleanup

* Fix error message

* fix

* Fix

* Improve error message
2024-08-13 13:25:21 +03:00
marthasharkey
0d5d01a294
Wip/mk/drive drilldown from enso (#10724)
This moves some of the drilldown logic to the enso code and utilises the new 'get_rows' method for Table
2024-08-12 12:51:51 +00:00
AdRiley
3581fd8e21
Make sqlserver temp tables work (#10781)
* Temp table support

* Fix

* Add query

* Clean up

* Clean up

* Fix

* Code review changes

* Added comment

* Code review changes

* Add execute
2024-08-09 13:06:16 +03:00
AdRiley
0f688d0a25
Add newlines option to text cleanse/replace (#10761)
* Auto-commit work in progress before clean build on 2024-08-06 11:32:46

* Fixed Regex and additional test

* changelog

* .

* Make non-capturing
2024-08-06 18:59:54 +03:00
Radosław Waśko
3fd14642d9
Fix upload/delete transactions in Snowflake backend (#10738)
Fixes #10609 by rewriting all our upload-related operations to rely on `DDL_Transaction` - an abstraction that handles 'transactionality' of `CREATE TABLE` statements dependent on if a given backend allows DDLs within transactions or not (if not it emulates transactionality by creating the tables outside of transaction and then dropping them on rollback).
2024-08-06 08:14:44 +00:00
Radosław Waśko
6ad3faf03b
Refactor Upload_Table to be more readable: split into separate smaller submodules (#10713)
- First step of #10609 - before I actually modify it, I decided I need to refactor the `Upload_Table` logic as it was quite convoluted. Doing this as a separate PR for easier review. A big 600+ line file was replaced by several smaller ones grouped by topics.
- Practically no changes apart from moving stuff into separate modules.
- One small change - added `Missing_Argument` to `SQL_Query` as I noticed that lack of defaults was giving rise to confusing errors when working with `query` in the GUI.

Before:
![image](https://github.com/user-attachments/assets/b586caec-f25c-406e-be5a-d402f10feb86)
After:
![image](https://github.com/user-attachments/assets/6b1d4206-05b1-4587-b3e1-43ca95ea7c2e)
![image](https://github.com/user-attachments/assets/58c098bd-db0c-4ee2-823c-bf5c9e758ce4)
2024-07-31 09:54:17 +00:00
AdRiley
0c552489e3
Add Initial SQL Server support (#10624)
* Squash all commits to resolve merge conflicts

* Fix merge problems

* Merge fix

* Fix port

* Fix warning

* cargo fmt

* legal review

* Small fixes

* Update instructions

* Code review feedback

* Cleanup

* typo

* Fix

* Remove leftover snowflake code

* Remove comment

* Add underscore

* Type cleanup

* Code review fix

* Cleanup

* Add datetime roundtrip test

* add comment

* drop

* Refactor

* Refactor

* Fix merge

* Fix

* Fix

* fix

* Add comment
2024-07-30 11:13:08 +01:00
James Dunkerley
74acc1de24
Tweaks from client Demo (#10685)
- Adjusted Filter_Condition removing keep/drop from basic filters.
- Fix Is_In to have selector.
- Fix for Date simple expressions.
- Add get_row to Table and DB_Table.
2024-07-26 18:44:36 +00:00
Radosław Waśko
46e8bab429
Fix aggregate after sort in DB (#10677)
* add test for https://github.com/enso-org/enso/issues/10321

* wrap in subquery if ordering was present to fix the bug

* adding one more test just to be sure

* fix import
2024-07-26 08:43:19 +01:00
Kaz Wesley
e5b85bf16e
Space-precedence does not apply to value-level operators (#10597)
In a sequence of value-level operators, whitespace does not affect relative precedence. Functional operators still follow the space-precedence rules.

The "functional" operators are: `>> << |> |>> <| <<| : .`, application, and any operator containing `<-` or `->`. All other operators are considered value-level operators.

Asymmetric whitespace can still be used to form *operator sections* of value-level operators, e.g. `+2 * 3` is still equivalent to `x -> (x+2) * 3`.

Precedence of application is unchanged, so `f x+y` is still equivalent to `f (x + y)` and `f x+y * z` is still equivalent to `(f (x + y)) * z`.

Any attempt to use spacing to override value-level operator precedence will be caught by the new enso linter. Mixed spacing (for clarity) in value-operator expressions is allowed, as long as it is consistent with the precedences of the operators.

Closes #10366.

# Important Notes
Precedence warnings:
- The parser emits a warning if the whitespace in an expression is inconsistent with its effective precedence.
- A new enso linter can be run with `./run libraries lint`. It parses all `.enso` files in `distribution/lib` and `test`, and reports any errors or warnings. It can also be run on individual files: `cargo run --release --bin check_syntax -- file1 file2...` (the result may be easier to read than the `./run` output).
- The linter is also run as part of `./run lint`, so it is checked in CI.

Additional language change:
- The exponentiation operator (`^`) now has higher precedence than the multiplication class (`*`, `/`, `%`). This change did not affect any current enso files.

Library changes:
- The libraries have been updated. The new warnings were used to identify all affected code; the changes themselves have not been programmatically verified (in many cases their equivalence relies on the commutativity of string concatenation).
2024-07-24 10:55:44 +00:00
Radosław Waśko
7fd8701690
Snowflake Dialect pt. 6 - Union, Distinct and other improvements (#10576)
- Part of #9486
- Fixes `Table.union`, `merge` and `distinct` tests
- Replaces `distinct_on` in `Context` that was actually a Postgres specific addition leaking into the base with a more abstract `Context_Extension` mechanism.
- This allows us to implement the Snowflake-specific `DISTINCT` using `QUALIFY`.
2024-07-19 16:04:00 +00:00
James Dunkerley
15810a4675
Linting fixes. (#10589)
Generating Doc data so fixed linting.
2024-07-18 12:42:34 +00:00
James Dunkerley
771402acf6
Widget Improvements (#10575)
- New centralised `make_any_selector` for creating dropdown for various types.
- Updated `Filter_Condition.default_widget` to be auto-scoped.
![image](https://github.com/user-attachments/assets/2786d512-6716-4d4b-b119-15054611c11c)
- Nested structure for Simple_Expressions.
![image](https://github.com/user-attachments/assets/09eda332-0888-41e7-8462-fc8566959da5)
2024-07-17 19:53:17 +00:00
AdRiley
cd1d61710f
Rename run workflow button to write (#10568)
Request from Product team to not have a run button. Renaming it to write
2024-07-16 18:04:31 +00:00
Radosław Waśko
a30b0c60eb
Snowflake Dialect pt. 5 (#10528)
- Related to #9486
- Batching of expression tests
- Fixing arithmetic by simplifying `%` and `/` operations
- Trying to share some more tables, sometimes improving performance sometimes not really
- Adding sorting and other fixes to tests to make them pass: Missing_Values_Spec, Filter_Spec, Map_Spec
- Fixing warnings related to materialization of Decimal->Integer, thus fixing Join_Spec.
2024-07-16 09:38:57 +00:00
Radosław Waśko
632355f85b
Snowflake Dialect pt. 4 - reading a column of small integers as Integer type, other type mapping tests (#10518)
- Related to #9486
- Ensures that even though an integer column in Snowflake is represented by `Decimal` type, if the values are small enough, they are materialized as `Integer`.
- If the values are larger, they are still read in as `Decimal`.
- Adds tests for some other `Decimal` edge cases (various precisions and scales), and for `Float`.
2024-07-11 20:14:46 +00:00
Pavel Marek
0f9852aab2
Replace all from ... export all with explicit exports (#10369)
Replace all exports with explicit exports.
2024-07-11 19:34:25 +02:00