Commit Graph

223 Commits

Author SHA1 Message Date
Radosław Waśko
0be572af6a
Only Join_Kind.Inner removes the common-named columns (#6564)
As requested by @jdunkerley
2023-05-05 08:12:07 +00:00
Radosław Waśko
41a8257e8d
Separating Redshift connector from Database library into a new AWS library (#6550)
Related to #5777
2023-05-04 17:36:51 +00:00
James Dunkerley
bb8f910be9
Continued Execution Context work and some little fixes (#6506)
- Add `with_disabled` shortcut for `Context`.
- Protect `Image.write` behind `Context.Output`.
- Correct text on the `Forbidden_Operation` error message.
- Remove context overrides from tests.
- Add `File` operations tests with `Context.Output` disabled.
- Add tests for `Text.write` operations with `Context.Output` disabled.
- Use a better method to make `File_Format` dropdown widget.
- Fix bug in `Invalid_Format.to_display_text`.
2023-05-03 08:01:16 +00:00
Radosław Waśko
5eb9c3a843
Implement create_database_table for Database Table (#6467)
Closes #6327
2023-05-02 14:01:44 +00:00
James Dunkerley
6b0c682b08
Add Execution Context control to Text.write (#6459)
- Adjusted `Context.is_enabled` to support default argument (moved built in so can have defaults).
- Made `environment` case-insensitive.
- Bug fix for play button.
- Short hand to execute within an enabled context.
- Forbid file writing if the Output context is disabled with a `Forbidden_Operation` error.
- Add temporary file support via `File.create_temporary_file` which is deleted on exit of JVM.
- Execution Context first pass in `Text.write`.
- Added dry run warning.
- Writes to a temporary file if disabled.
- Created a `DryRunFileManager` which will create and manage the temporary files.
- Added `format` dropdown to `File.read` and `Data.read`.
- Renamed `JSON_File` to `JSON_Format` to be consistent.

(still to unit test).
2023-04-29 08:39:18 +00:00
Radosław Waśko
cdd0065800
Change defaults for Connection.tables and ensure that Connection.query recognizes all available tables (#6443)
Closes #6398
2023-04-29 00:13:29 +00:00
Hubert Plociniczak
0dc9f3742c
Unbreak CI after outdated merge (#6466) 2023-04-28 14:35:11 +02:00
Hubert Plociniczak
c6790f1e9c
Report only unique warnings (#6372)
This change makes sure that reported warnings are unique, based on the value of internal clock tick and ignoring differences in reassignments.

Before:
![Screenshot from 2023-04-20 15-42-55](https://user-images.githubusercontent.com/292128/233415710-925c1045-37c7-49f5-9bc3-bfbfd30270a3.png)
After:
![Screenshot from 2023-04-20 15-27-27](https://user-images.githubusercontent.com/292128/233415807-8cb67bc2-ac37-4db7-924e-ae7619074b5b.png)

On the positive side, no further changes, like in LS, have to be done.


Closes #6257.
2023-04-28 07:16:00 +00:00
Radosław Waśko
462016a428
Create database table from memory (#6429) 2023-04-27 20:06:17 +00:00
James Dunkerley
0c7c3bdeaf
Fix for the massive number of warnings when renaming with invalid names. (#6450)
* Rename makeUnique overloads to avoid issue when Nothing is passed.
Suspend warnings when building the output table to avoid mass warning duplication.

* Add test for mixed invalid names.
Adjust so a single warning attached.

* PR comments.
2023-04-27 14:51:59 +01:00
James Dunkerley
0e51131809
Table Visualization and display text changes. (#6382)
- Missing tests from number parsing.
- Fix type signature on some warning methods.
- Fix warnings on `Standard.Database.Data.Table.parse_values`.
- Added test for `Nothing` and empty string on `use_first_row_as_names`.
- New API for `Number.format` taking a simple format string and `Locale`.
- Add ellipsis to truncated `Text.to_display_text`.
- Adjusted built-in `to_display_text` for numbers to not include type (but also to display BigInteger as value).
- Remove `Noise.Generator` interface type.
- Json: Added `to_display_text` to `JS_Object`.
- Time: Added `to_display_text` for `Date`, `Time_Of_Day`, `Date_Time`, `Duration` and `Period`.
- Text: Added `to_display_text` to `Locale`, `Case_Sensitivity`, `Encoding`, `Text_Sub_Range`, `Span`, `Utf_16_Span`.
- System: Added `to_display_text` to `File`, `File_Permissions`, `Process_Result` and `Exit_Code`.
- Network: Added `to_display_text` to `URI`, `HTTP_Status_Code` and `Header`.
- Added `to_display_text` to `Maybe`, `Regression`, `Pair`, `Range`, `Filter_Condition`.
- Added support for `to_js_object` and `to_display_text` to `Random_Number_Generator`.
- Verified all error types have `to_display_text`.
- Removed `BigInt`, `Date`, `Date_Time` and `Time_Of_Day` JS based rendering as using `to_display_text` now.
- Added support for rendering nested structures in the table viz.
2023-04-26 18:15:48 +00:00
GregoryTravis
dcfbf841b3
Replace Table should_equal with should_equal_verbose (#6405)
Table.should_equal_verbose shows the contents of the tables on failure; let's make this the standard comparison.
2023-04-26 14:01:49 +00:00
GregoryTravis
afd804d529
5127 Add Table.parse_to_columns to parse a single column to a set of columns. (#6383)
Implement Table.parse_to_columns
2023-04-24 15:21:38 +00:00
Radosław Waśko
f3873f9768
Infer SQLite types locally (#6381)
Closes #6208
2023-04-24 10:55:12 +00:00
Radosław Waśko
a43d524336
Add typechecks to Aggregate and Cross Tab (#6380)
Follow up of #6298 as it grew too much. Adds the needed typechecks to aggregate operations. Ensures that the DB operations report `Floating_Point_Equality` warning consistently with in-memory.
2023-04-24 08:55:54 +00:00
GregoryTravis
22f820feb7
Add Table.parse_text_to_table to convert Text to a Table. (#6294) 2023-04-21 17:43:19 +00:00
Radosław Waśko
8db2ad51a1
Adding typechecks to Column Operations (#6298)
Closes #6106
2023-04-21 12:20:12 +00:00
James Dunkerley
0350762386
Add replace, trim to Column. Better number parsing. (#6253)
- Add `replace` with same syntax as on `Text` to an in-memory `Column`.
- Add `trim` with same syntax as on `Text` to an in-memory `Column`.
- Add `trim` to in-database `Column`.
- Added `is_supported` to dialects and exposed the dialect consistently on the `Connection`.
- Add `write_table` support to `JSON_File` allowing `Table.write` to write JSON.
- Updated the parsing for integers and decimals:
- Support for currency symbols.
- Support for brackets for negative numbers.
- Automatic detection of decimal points and thousand separators.
- Tighter rules for scientific and thousand separated numbers.
- Remove `replace_text` from `Table`.
- Remove `write_json` from `Table`.
2023-04-20 16:04:59 +00:00
Pavel Marek
b42e910280
sort handles incomparable values (#5998)
* Update type ascriptions in some operators in Any

* Add @GenerateUncached to AnyToTextNode.

Will be used in another node with @GenerateUncached.

* Add tests for "sort handles incomparable types"

* Vector.sort handles incomparable types

* Implement sort handling for different comparators

* Comparison operators in Any do not throw Type_Error

* Fix some issues in Ordering_Spec

* Remove the remaining comparison operator overrides for numbers.

* Consolidate all sorting functionality into a single builtin node.

* Fix warnings attachment in sort

* PrimitiveValuesComparator handles other types than primitives

* Fix byFunc calling

* on function can be called from the builtin

* Fix build of native image

* Update changelog

* Add VectorSortTest

* Builtin method should not throw DataflowError.

If yes, the message is discarded (a bug?)

* TypeOfNode may not return only Type

* UnresolvedSymbol is not supported as `on` argument to Vector.sort_builtin

* Fix docs

* Fix bigint spec in LessThanNode

* Small fixes

* Small fixes

* Nothings and Nans are sorted at the end of default comparator group.

But not at the whole end of the resulting vector.

* Fix checking of `by` parameter - now accepts functions with default arguments.

* Fix changelog formatting

* Fix imports in DebuggingEnsoTest

* Remove Array.sort_builtin

* Add comparison operators to micro-distribution

* Remove Array.sort_builtin

* Replace Incomparable_Values by Type_Error in some tests

* Add on_incomparable argument to Vector.sort_builtin

* Fix after merge - Array.sort delegates to Vector.sort

* Add more tests for problem_behavior on Vector.sort

* SortVectorNode throws only Incomparable_Values.

* Delete Collections helper class

* Add test for expected failure for custom incomparable values

* Cosmetics.

* Fix test expecting different comparators warning

* isNothing is checked via interop

* Remove TruffleLogger from SortVectorNode

* Small review refactorings

* Revert "Remove the remaining comparison operator overrides for numbers."

This reverts commit 0df66b1080.

* Improve bench_download.py tool's `--compare` functionality.

- Output table is sorted by benchmark labels.
- Do not fail when there are different benchmark labels in both runs.

* Wrap potential interop values with `HostValueToEnsoNode`

* Use alter function in Vector_Spec

* Update docs

* Invalid comparison throws Incomparable_Values rather than Type_Error

* Number comparison builtin methods return Nothing in case of incomparables
2023-04-16 16:40:12 +02:00
GregoryTravis
4dcf5faddd
Add split and tokenize to the Table. (#6233)
Implement split and tokenize for tables.
2023-04-14 16:03:02 +00:00
Radosław Waśko
0f4f8a0542
Full-joins in SQLite (#6215)
Closes #5254

In #6189 the SQLite version was bumped to a newer release which has builtin support for Full and Right joins, so no workaround is no longer needed.
2023-04-06 16:49:14 +00:00
Radosław Waśko
f5db35af07
Adjust {Table|Column}.parse to use Value_Type (#6213)
Closes #5660
2023-04-06 10:58:55 +00:00
Radosław Waśko
83b10a2088
Implement Table.union for Database backend (#6204)
Closes #5235
2023-04-06 08:40:34 +00:00
James Dunkerley
f26bcf6ab6
Small issues from working with Ned (#6160)
- `Process.run` now returns a `Process_Result` allowing the easy capture of stdout and stderr.
- Joining a column with a column name does not warn if adding just the prefix.
- Stop the table viz from changing case and adding spaces to the headers.
2023-04-03 13:01:42 +00:00
Radosław Waśko
6ddcb553e5
Date/time support for Postgres. Year/month/day operations on Columns. (#6153)
Closes #6115
2023-03-31 18:37:04 +00:00
Radosław Waśko
6f86115498
Proper implementation of Value Types in Table (#6073)
This is the first part of the #5158 umbrella task. It closes #5158, follow-up tasks are listed as a comment in the issue.

- Updates all prototype methods dealing with `Value_Type` with a proper implementation.
- Adds a more precise mapping from in-memory storage to `Value_Type`.
- Adds a dialect-dependent mapping between `SQL_Type` and `Value_Type`.
- Removes obsolete methods and constants on `SQL_Type` that were not portable.
- Ensures that in the Database backend, operation results are computed based on what the Database is meaning to return (by asking the Database about expected types of each operation).
- But also ensures that the result types are sane.
- While SQLite does not officially support a BOOLEAN affinity, we add a set of type overrides to our operations to ensure that Boolean operations will return Boolean values and will not be changed to integers as SQLite would suggest.
- Some methods in SQLite fallback to a NUMERIC affinity unnecessarily, so stuff like `max(text, text)` will keep the `text` type instead of falling back to numeric as SQLite would suggest.
- Adds ability to use custom fetch / builder logic for various types, so that we can support vendor specific types (for example, Postgres dates).

# Important Notes
- There are some TODOs left in the code. I'm still aligning follow-up tasks - once done I will try to add references to relevant tasks in them.
2023-03-31 16:16:18 +00:00
GregoryTravis
6b9cbeacb2
Implement Regular Expression replace and update Text.replace to the new API (#5959)
Re-implement replace on top of Truffle regex.
2023-03-28 06:13:12 +00:00
James Dunkerley
58f2c7643f
Use new Enso Hash Codes and Comparable (#6060)
Enables `distinct`, `aggregate` and `cross_tab` to use the Enso hashing and equality operations.
Also, I rewired the way the ObjectComparators are obtained in polyglot code to be more consistent.

Add Comparator for `Day_Of_Week`, `Header`, `SQL_Type`, `Image` and `Matrix`.
Also, removed the custom `==` from these types as needed. (Closes #5626)
2023-03-24 15:02:25 +00:00
Hubert Plociniczak
8c6fd60aaf
Detect conflicts between exported types and FQNs (#5986)
Exporting types named the same as the module where they are defined in `Main` modules of library components may lead to accidental name conflicts. This became apparent when trying to access `Problem_Behavior` module via a fully qualified name and the compiler rejected it. This is due to the fact that `Main` module exported `Error` type defined in `Standard.Base.Error` module, thus making it impossible to access any other submodules of `Standard.Base.Error` via a fully qualified name.

This change adds a warning to FullyQualifiedNames pass that detects any such future problems.
While only `Error` module was affected, it was widely used in the stdlib, hence the number of changes.

Closes #5902.

# Important Notes
I left out the potential conflict in micro-distribution, thus ensuring we actually detect and report the warning.
2023-03-21 21:09:41 +00:00
Radosław Waśko
952beba8d1
Fix cross_tab column naming edge cases, add fill_empty (#5863)
Closes #5151 and adds some additional tests for `cross_tab` that verify duplicated and invalid names.

I decided that for empty or `Nothing` names, instead of replacing them with `Column` and implicitly losing connection with the value that was in the column, we should just error on such values.

To make handling of these easier, `fill_empty` was added allowing to easily replace the empty values with something else.

Also, `{is,fill}_missing` was renamed to `{is,fill}_nothing` to align with `Filter_Condition.Is_Nothing`.
2023-03-11 11:58:54 +00:00
Pavel Marek
5f7a4a5a39
Merge ordered and unordered comparators (#5845)
Merge _ordered_ and _unordered_ comparators into a single one.

# Important Notes
Comparator is now required to have only `compare` method:
```
type Comparator
comapre : T -> T -> (Ordering|Nothing)
hash : T -> Integer
```
2023-03-11 05:43:22 +00:00
Radosław Waśko
91ef8acf35
Review generated Column names (#5850)
Closes #5583 and closes #5157
2023-03-10 19:07:58 +00:00
Radosław Waśko
62e57f5557
Test some Mismatched Quote edge cases in Delimited reader (#5810)
Follow-up to #5113 - I add some more edge case tests as we discussed with @jdunkerley

When debugging some quoting issues, I also realised the current `Mismatched_Quote` error provided not enough information. So I amended it to at least include some context indicating which was the 'offending' cell.
2023-03-10 15:47:57 +00:00
Jaroslav Tulach
8bbdd1af5b
Meta.is_a consistent with case-type-of check (#5853)
Removing special handling of `AtomConstructor` in `Meta.is_a` check.

# Important Notes
A lot of tests are about to fail. Many of them indirectly call `Meta.is_a` with a constructor rather than type.
2023-03-10 07:41:04 +00:00
James Dunkerley
299bfd6b7d
Fixes from the Demo on 2nd March (#5823)
- Fix issue with Geo Map viz.
- Handle invalid format strings better in `Data_Formatter`.
- New constants for the ISO format strings (and a special ENSO_ZONED_DATE_TIME)
- Consistent Date Time format for parsing in all places.
- Avoid throwing exception in datetime parsing.
- Support for milliseconds (well nanoseconds) in Date_Time and Time_Of_Day.
- `Column.map` stays within Enso.
- Allow `Aggregate_Column.Group_By` in `cross_tab` group_by parameter.
2023-03-07 20:58:00 +00:00
Pavel Marek
b6e2319fcc
Comparators support partial ordering (#5778) 2023-03-07 04:16:38 +00:00
Radosław Waśko
da760aa27d
Review Text/Table.write problem behavior (#5816)
Closes #5114

Added tests for various problems scenarios when writing files.

And ensured that those tests are passing by fixing a few edge cases.
2023-03-07 02:25:13 +00:00
Radosław Waśko
2d29456ed1
Review File/Data read and read_text warnings (#5799)
Closes #5113

Fixes a bug where read-only files would be overwritten if File.write was used in backup mode, and added tests to avoid such regression. To implement it, introduced a `is_writable` property on `File`.
2023-03-06 03:43:38 +00:00
James Dunkerley
01fc34c18a
Improving Expression Support for In Database (#5790)
- Adjust Excel Workbook write behaviour.
- Support Nothing / Null constants.
- Deduce the type of arithmetic operations and `iif`.
- Allow Date_Time constants, treating as local timezone.
- Removed the `to_column_name` and `ensure_sane_name` code.
2023-03-03 12:03:05 +00:00
Radosław Waśko
b764b0b7b7
Improve error handling of Connection.query (#5693)
Closes #5252
2023-02-24 17:15:10 +00:00
Radosław Waśko
793eafc866
Improve Table.parse_values API (#5692)
Closes #5111
2023-02-24 13:35:01 +00:00
James Dunkerley
652b8d5db3
Update rename_columns to new API design, add first_row, second_row and last_row functions to the table. (#5719)
- Updates the `rename_columns` API.
- Add `first_row`, `second_row` and `last_row` to the Table types.
- New option for reading only last row of ResultSet.
2023-02-23 19:42:45 +00:00
Radosław Waśko
3027c6f3a2
Ensure entries containing newlines are quoted when writing Delimited Files (#5652)
Fixes #5638
2023-02-17 00:57:48 +00:00
James Dunkerley
1bc27501e6
Remove Column type from Aggregate_Column, simplify Column_Selector, some new File_Formats (#5646)
- Updated `Widget.Vector_Editor` ready for use by IDE team.
- Added `get` to `Row` to make API more aligned.
- Added `first_column`, `second_column` and `last_column` to `Table` APIs.
- Adjusted `Column_Selector` and associated methods to have simpler API.
- Removed `Column` from `Aggregate_Column` constructors.
- Added new `Excel_Workbook` type and added to `Excel_Section`.
- Added new `SQLiteFormatSPI` and `SQLite_Format`.
- Added new `IamgeFormatSPI` and `Image_Format`.
2023-02-16 15:15:49 +00:00
Radosław Waśko
a02eab451e
Implement basic warnings for column arithmetic, review warnings on expressions and filter (#5605)
Closes #5109

# Important Notes
- Currently the tests pass for the in-memory parts of Common_Table_Operations, but still some stuff not working on DB backends - in progress.
2023-02-14 09:33:04 +00:00
Pavel Marek
1f8511dab2
Add Comparator conversion for all types (#4067)
Add `Comparator` type class emulation for all types. Migrate all the types in stdlib to this new `Comparator` API. The main documentation is in `Ordering.enso`.

Fixes these pivotals:
- https://www.pivotaltracker.com/story/show/183945328
- https://www.pivotaltracker.com/story/show/183958734
- https://www.pivotaltracker.com/story/show/184380208

# Important Notes
- The new Comparator API forces users to specify both `equals` and `hash` methods on their custom comparators.
- All the `compare_to` overrides were replaced by definition of a custom _ordered_ comparator.
- All the call sites of `x.compare_to y` method were replaced with `Ordering.compare x y`.
- `Ordering.compare` is essentially a shortcut for `Comparable.from x . compare x y`.
- The default comparator for `Any` is `Default_Unordered_Comparator`, which just forwards to the builtin `EqualsNode` and `HashCodeNode` nodes.
- For `x`, one can get its hash with `Comparable.from x . hash x`.
- This makes `hash` as _hidden_ as possible. There are no other public methods to get a hash code of an object.
- Comparing `x` and `y` can be done either by `Ordering.compare x y` or `Comparable.from x . compare x y` instead of `x.compare_to y`.
2023-02-10 09:22:11 +00:00
James Dunkerley
1c821e22cf
Some fixed form the Anagrams experiment. (#5592)
- Fixes the display of Date, Time_Of_Day and Date_Time so doesn't wrap.
- Adjust serialization of large integer values for JS and display within table.
- Workaround for issue with using `.lines` in the Table (new bug filed).
- Disabled warning on no specified `separator` on `Concatenate`.

Does not include fix for aggregation on integer values outside of `long` range.
2023-02-08 22:17:00 +00:00
Radosław Waśko
4f90946d1e
Rework Invalid Aggregations (#5579)
Closes #5108
2023-02-08 18:39:09 +00:00
Radosław Waśko
3c72ab08c4
Review Missing_Input_Column and Column_Index_Out_Of_Range warnings (#4118)
Implements https://www.pivotaltracker.com/story/show/184226383
2023-02-06 19:52:25 +00:00
James Dunkerley
0790ce494f
New set function, parse a column (#4097)
- New `set` function design - takes a `Column` and works with that more easily and supports control of `Set_Mode`.
- New simple `parse` API on `Column`.
- Separated expression support for `filter` to new `filter_by_expression` on `Table`.
- New `compute` function allowing creation of a column from an expression.
- Added case sensitivity argument to `Column` based on `starts_with`, `ends_with` and `contains`.
- Added case sensitivity argument to `Filter_Condition` for `Starts_With`, `Ends_With`, `Contains` and `Not_Contains`.
- Fixed the issue in JS Table visualisation where JavaScript date was incorrectly set.
- Some dynamic dropdown expressions - experimenting with ways to use them.
- Fixed issue with `.pretty` that wasn't escaping `\`.
- Changed default Postgres DB to `postgres`.
- Fixed SQLite support for starts_with, ends_with and contains to be consistent (using GLOB not LIKE).
2023-01-31 20:48:16 +00:00