Commit Graph

287 Commits

Author SHA1 Message Date
James Dunkerley
f0f6deef2a
Load the File_Format types via a ServiceLoader (#3813)
Moves the File.read method into the `File` type.
Uses the ServiceLoader to find all types for the File_Format.
2022-10-24 09:55:18 +00:00
Radosław Waśko
cc76e7d36a
Add support for Blank_Columns to Table and Database (#3812)
Implements https://www.pivotaltracker.com/story/show/183390281 and https://www.pivotaltracker.com/story/show/183390394
2022-10-20 09:11:08 +00:00
Radosław Waśko
17f73988e8
Update drop_missing_rows to filter_blank_rows API. (#3805)
Implements https://www.pivotaltracker.com/story/show/183390042 and https://www.pivotaltracker.com/story/show/183390370
2022-10-18 15:58:50 +00:00
James Dunkerley
701c644d0e
Tidy up the remaining ones except Base... (#3797)
- Removed `Dubious constructor export` from Examples, Geo, Google_Api, Image and Test.
- Updated Google_Api project to meet newer code standards.
- Restructured `Standard.Test`:
- `Main.enso` now exports `Bench`, `Faker`, `Problems`, `Test`, `Test_Suite`
- `Test.Suite` methods moved into a `Test_Suite` type.
- Moved `Bench.measure` into `Bench` type.
- Separated the reporting to a `Test_Reporter` module.
- Moved `Faker` methods into `Faker` type.
- Removed `Verbs` and `.should` method.
- Added `should_start_with` and `should_contain` extensions to `Any`.
- Restructured `Standard.Image`:
- Merged Codecs methods into `Image`.
- Export `Image`, `Read_Flag`, `Write_Flag` and `Matrix` as types from `Main.enso`.
- Merged the internal methods into `Matrix` and `Image`.
- Fixed `Day_Of_Week` to be exported as a type and sort the `from` method.
2022-10-17 11:27:27 +00:00
Radosław Waśko
82de8f88bd
Add support for Is_In and Not_In to Filter_Condition (#3790)
Implements https://www.pivotaltracker.com/story/show/183389945
2022-10-15 11:29:59 +00:00
Paweł Grabarz
ce6267f098
Add replace_text method to In-Memory Table (#3793)
Implements https://www.pivotaltracker.com/n/projects/2539304/stories/183415329
2022-10-14 17:42:29 +02:00
Radosław Waśko
592a8516a8
Add Is_Empty, Not_Empty, Like and Not_Like to Filter_Condition (#3775)
Implements https://www.pivotaltracker.com/story/show/183389890
2022-10-10 23:11:04 +00:00
James Dunkerley
9301f2dcc5
Sort out statics in Database. (#3774)
- Moved `Standard.Database.connect` into `Standard.Database.Database.connect`, so can now just `from Standard.Database import ...`.
- Removed all `Dubious constructor export`s.
- Switched to using `project` for internal imports.
- Moved to using `Value` for private constructors and not re-exporting.
- Export types not modules from `Standard.Database`.
- Broke up `IR` into separate files (Context, Expression, From_Spec, Internal_Column, Join_Kind, Query).
- No longer use `IR.` instead via specific types.
- Broke up `SQL` into separate files (SQL_Type and SQL_Statement).

Additionally;
- Standard.Table: Moved `storage_types` into `Storage`.
- Standard.Table: Switched to using `project` for internal imports.
- Standard.Table.Excel: Renamed modules `Range` to `Excel_Range` and `Section` to `Excel_Section`.
- `Standard.Visualisation`: Switched to using `project` for internal imports.
- `Standard.Visualisation`: Moved to using `Value` for private constructors and not re-exporting.

# Important Notes
- Have not cleared up the `Errors` yet.
- Have not switched to type pattern matching.
2022-10-07 11:32:00 +00:00
Radosław Waśko
7afaf8c6cc
Add filtering by Filter_Condition to Vector, Range and List (#3770)
Implements https://www.pivotaltracker.com/story/show/183389901
2022-10-07 04:02:54 +00:00
James Dunkerley
185378f07c
Moving library statics to type for Table. (#3760)
- Generally export types not modules from the `Standard.Table` import.
- Moved `new`, `from_rows` the `Standard.Table` library into the `Table` type.
- Renames `Standard.Table.Data.Storage.Type` to `Standard.Table.Data.Storage.Storage`
- Removed the internal `from_columns` method.
- Removed `join` and `concat` and merged into instance methods.
- Removed `Table` and `Column` from the `Standard.Database` exports.
- Removed `Standard.Table.Data.Column.Aggregate_Column` as not used any more.
2022-10-06 17:01:18 +00:00
Radosław Waśko
503d3eb5a3
Update Table API with new filter design (#3750)
Implements https://www.pivotaltracker.com/story/show/183389855

# Important Notes
Implements basic filter operations both In-Memory and for the Database backend, ensuring that existing tests can be adapted and keep working. Not all `Filter_Condition`s are implemented yet.

Also implements significant part of https://www.pivotaltracker.com/story/show/183390314
2022-10-05 11:40:10 +00:00
James Dunkerley
eeec3bdf94
Adding a shorthand for the Table API until widgets available (#3753)
Allows using `Vector ColumnName` for the various table functions as short hand.

- `select_columns`, `remove_columns`,`reorder_columns`, `distinct` all map to an exact By_Name match.
- `rename_columns` does a positional rename on the Vector passed.
- `order_by` sorts ascending on each column passed in order.

# Important Notes
This may be reversed once widgets are available and working but this makes the APIs much more usable in current UI.
2022-10-04 11:37:14 +00:00
Hubert Plociniczak
ae66087bb9
By-type pattern matching (#3742)
This change brings by-type pattern matching to Enso.
One can pattern match on Enso types as well as on polyglot types.

For example,
```
case x of
_ : Integer -> ...
_ : Text -> ...
_ -> ...
```
as well as Java's types
```
case y of
_ : ArrayList -> ...
_ : List -> ...
_ : AbstractList -> ...
_ -> ..
```

It is no longer possible to match a value with a corresponding type constructor.
For example
```
case Date.now of
Date -> ...
```
will no longer match and one should match on the type (`_ : Date`) instead.
```
case Date of
Date -> ...
```
is fine though, as requested in the ticket.

The change required further changes to `type_of` logic which wasn't dealing well with polyglot values.

Implements https://www.pivotaltracker.com/story/show/183188846

# Important Notes
~I discovered late in the game that nested patterns involving type patterns, such as `Const (f : Foo) tail -> ...` are not possible due to the old parser logic.
I would prefer to add it in a separate PR because this one is already getting quite large.~ This is now supported!
2022-10-04 09:23:58 +00:00
Radosław Waśko
61a4120cfb
Fix date comparisons and test sorting of tables and vectors with dates (#3745)
Implements https://www.pivotaltracker.com/story/show/183402892

# Important Notes
- Fixes inconsistent `compare_to` vs `==` behaviour in date/time types and adds test for that.
- Adds test for `Table.order_by` on dates and custom types.
- Fixes an issue with `Table.order_by` for custom types.
- Unifies how incomparable objects are reported by `Table.order_by` and `Vector.sort`.
- Adds benchmarks comparing `Table.order_by` and `Vector.sort` performance.
2022-09-29 08:48:00 +00:00
Radosław Waśko
7da4d61484
Update Case Sensitivity arguments (#3741)
Implements https://www.pivotaltracker.com/story/show/183314956
2022-09-27 20:15:52 +00:00
Marcin Kostrzewa
726bfeb883
Wip/mk/statics (#3740)
Makes statics static. A type and its instances have different methods defined on them, as it should be. Constructors are now scoped in types, and can be imported/exported.

# Important Notes
The method of fixing stdlib chosen here is to just not. All the conses are exported to make all old code work. All such instances are marked with `TODO Dubious constructor export` so that it can be found and fixed.
2022-09-27 14:23:06 +00:00
Hubert Plociniczak
7a6ee0c200
Implement type_of (#3722)
This change implements a simple `type_of` method that returns a type of a given value, including for polyglot objects.

The change also allows for pattern matching on various time-related instances. It is a nice-to-have on its own, but it was primarily needed here to write some tests. For equality checks on types we currently can't use `==` due to a known _feature_ which essentially does wrong dispatching. This will be improved in the upcoming statics PR so we agreed that there is no point in duplicating that work and we can replace it later.

Also, note that this PR changes `Meta.is_same_object`. Comparing types revealed that it was wrong when comparing polyglot wrappers over the same value.
2022-09-26 16:01:39 +00:00
James Dunkerley
6f54e80970
Adjust Database connection to use query/read to access data. (#3727)
Adjust Database connection API to align with new [design](https://github.com/enso-org/design/blob/wip/jd/database-read/epics/basic-libraries/database-read/design.md#querying-tables).
- `query` replaces the old `access_table` and is expanded to support raw SQL queries.
- `read` replaces `execute_query` and matches the API of `query`.
- `to_dataframe` is renamed to `read`.

# Important Notes
Added support for `++` to concatenate a Text without wrapping in a `SQL.Code`.
2022-09-23 07:35:08 +00:00
James Dunkerley
d6346e9d66
Renaming various constructors and moving types around for Database. (#3715)
Repairing the constructor name following the types work. Some general tiding up as well.

- Remove `Standard.Database.Data.Column.Aggregate_Column_Builder`.
- Remove `Standard.Database.Data.Dialect.Dialect.Dialect_Data`.
- Remove unused imports and update some type definitions.
- Rename `Postgres.Postgres_Data` => `Postgres_Options.Postgres`.
- Rename `Redshift.Redshift_Data` => `Redshift_Options.Redshift`.
- Rename `SQLite.SQLite_Data` => `SQLite_Options.SQLite`.
- Rename `Credentials.Credentials_Data` => `Credentials.Username_And_Password`.
- Rename `Sql` to `SQL` across the board.
- Merge `Standard.Database.Data.Internal` into `Standard.Database.Internal`.
- Move dialects into `Internal` and merge the function in `Helpers` into `Base_Generator`.
2022-09-19 12:39:40 +00:00
Radosław Waśko
8fa8d12cc3
String functionality in std-table should use std-base (#3717)
Implements https://www.pivotaltracker.com/story/show/181754646
2022-09-17 14:38:02 +00:00
Hubert Plociniczak
0e5df935d3
Don't rename imported Main module that only imports names (#3710)
Turns that if you import a two-part import we had special code that would a) add Main submodule b) add an explicit rename.

b) is problematic because sometimes we only want to import specific names.
E.g.,
```
from Bar.Foo import Bar, Baz
```
would be translated to
```
from Bar.Foo.Main as Foo import Bar, Baz
```
and it should only be translated to
```
from Bar.Foo.Main import Bar, Baz
```

This change detects this scenario and does not add renames in that case.

Fixes [183276486](https://www.pivotaltracker.com/story/show/183276486).
2022-09-16 13:01:06 +00:00
Radosław Waśko
5ed388930e
Additional tests for handling Dates in Table (#3707)
Resolves https://www.pivotaltracker.com/story/show/183285801

@JaroslavTulach suggested the current implementation may not handle these correctly, which suggests that the logic is not completely trivial - so I added a test to ensure that it works as we'd expect. Fortunately, it did work - but it's good to keep the tests to avoid regressions.
2022-09-15 23:18:19 +00:00
James Dunkerley
0126f02e7b
Restructure File.read into the new design (#3701)
Changes following Marcin's work. Should be back to very similar public API as before.

- Add an "interface" type: `Standard.Base.System.File_Format.File_Format`.
- All `File_Format` types now have a `can_read` method to decide if they can read a file.
- Move `Standard.Table.IO.File_Format.Text.Text_Data` to `Standard.Base.System.File_Format.Plain_Text_Format.Plain_Text`.
- Move `Standard.Table.IO.File_Format.Bytes` to `Standard.Base.System.File_Format.Bytes`.
- Move `Standard.Table.IO.File_Format.Infer` to `Standard.Base.System.File_Format.Infer`. **(doesn't belong here...)**
- Move `Standard.Table.IO.File_Format.Unsupported_File_Type` to `Standard.Base.Error.Common.Unsupported_File_Type`.
- Add `Infer`, `File_Format`, `Bytes`, `Plain_Text`, `Plain_Text_Format` to `Standard.Base` exports.
- Fold extension methods of `Standard.Base.Meta.Unresolved_Symbol` into type.
- Move `Standard.Table.IO.File_Format.Auto` to `Standard.Table.IO.Auto_Detect.Auto_Detect`.
- Added a `types` Vector of all the built in formats.
- `Auto_Detect` asks each type if they `can_read` a file.
- Broke up and moved `Standard.Table.IO.Excel` into `Standard.Table.Excel`:
- Moved `Standard.Table.IO.File_Format.Excel.Excel_Data` to `Standard.Table.Excel.Excel_Format.Excel_Format.Excel`.
- Renamed `Sheet` to `Worksheet`.
- Internal types `Reader` and `Writer` providing the actual read and write methods.
- Created `Standard.Table.Delimited` with similar structure to `Standard.Table.Excel`:
- Moved `Standard.Table.IO.File_Format.Delimited.Delimited_Data` to `Standard.Table.Delimited.Delimited_Format.Delimited_Format.Delimited`.
- Moved `Standard.Table.IO.Quote_Style` to `Standard.Table.Delimited.Quote_Style`.
- Moved the `Reader` and `Writer` internal types into here. Renamed methods to have unique names.
- Add `Aggregate_Column`, `Auto_Detect`, `Delimited`, `Delimited_Format`, `Excel`, `Excel_Format`, `Sheet_Names`, `Range_Names`, `Worksheet` and `Cell_Range` to `Standard.Table` exports.
2022-09-15 14:48:46 +00:00
James Dunkerley
2b425f8e08
Restructuring Database.Connection to allow for database specific types. (#3632)
- Added `databases`, `database`, `set_database`.
- Added `schemas`, `schema`, `set_schema`.
- Added `table_types`,
- Added `tables`.
- Moved the vast majority of the connection work into a lower level `JDBC_Connection` object.
- `Connection` represents the standard API for database connections and provides a base JDBC implementation.
- `SQLite_Connection` has the `Connection` API but with custom `databases` and `schemas` methods for SQLite.
- `Postgres_Connection` has the `Connection` API but with custom `set_database`, `databases`, `set_schema` and `schemas` methods for Postgres.
- Updated `Redshift` - no public API change.
2022-09-07 17:32:28 +00:00
Radosław Waśko
551100af3b
Add Table.distinct function to In-Memory table (#3684)
Implements https://www.pivotaltracker.com/story/show/182307143

# Important Notes
- Modified standard library Java helpers dependencies so that `std-table` module depends on `std-base`, as a provided dependency. This is allowed, because `std-table` is used by the `Standard.Table` Enso module which depends on `Standard.Base` which ensures that the `std-base` is loaded onto the classpath, thus whenever `std-table` is loaded by `Standard.Table`, so is `std-base`. Thus we can rely on classes from `std-base` and its dependencies being _provided_ on the classpath. Thanks to that we can use utilities like `Text_Utils` also in `std-table`, avoiding code duplication. Additional advantage of that is that we don't need to specify ICU4J as a separate dependency for `std-table`, since it is 'taken' from `std-base` already - so we avoid including it in our build packages twice.
2022-09-07 12:28:41 +00:00
Radosław Waśko
eafba079d9
Make In Memory Table Aggregator types more specific where possible (#3679)
Many aggregation types fell back to the general `Any` type where they could have used the type of input column - for example `First` of a column of integers is guaranteed to fit the `Integer` storage type, so it doesn't have to fall back to `Any`. This PR fixes that and adds a test that checks this.
2022-09-05 09:17:41 +00:00
Radosław Waśko
65140f48ca
Add storage support for Date, Time and DateTime to InMemory table (#3673)
Implements https://www.pivotaltracker.com/story/show/183080911
2022-08-31 22:06:29 +00:00
Marcin Kostrzewa
4fc6dcced0
Get rid of free-floating atoms. Everything has a type now! (#3671)
This is a step towards the new language spec. The `type` keyword now means something. So we now have
```
type Maybe a
Some (from_some : a)
None
```
as a thing one may write. Also `Some` and `None` are not standalone types now – only `Maybe` is.
This halfway to static methods – we still allow for things like `Number + Number` for backwards compatibility. It will disappear in the next PR.

The concept of a type is now used for method dispatch – with great impact on interpreter code density.

Some APIs in the STDLIB may require re-thinking. I take this is going to be up to the libraries team – some choices are not as good with a semantically different language. I've strived to update stdlib with minimal changes – to make sure it still works as it did.

It is worth mentioning the conflicting constructor name convention I've used: if `Foo` only has one constructor, previously named `Foo`, we now have:
```
type Foo
Foo_Data f1 f2 f3
```

This is now necessary, because we still don't have proper statics. When they arrive, this can be changed (quite easily, with SED) to use them, and figure out the actual convention then.

I have also reworked large parts of the builtins system, because it did not work at all with the new concepts.

It also exposes the type variants in SuggestionBuilder, that was the original tiny PR this was based on.

PS I'm so sorry for the size of this. No idea how this could have been smaller. It's a breaking language change after all.
2022-08-30 22:54:53 +00:00
Radosław Waśko
e6e4692692
DataFormatter should infer datetime from values without seconds (#3668)
Fixes https://www.pivotaltracker.com/story/show/183033133
2022-08-26 21:10:52 +00:00
Radosław Waśko
d7ebc4a338
Add Table.take and Table.drop functions to In-Memory table (#3647)
Implements https://www.pivotaltracker.com/story/show/182307347
2022-08-26 19:41:36 +00:00
James Dunkerley
a20d43390e
Adding DateTime part functions (#3669)
- Added `Zone`, `Date_Time` and `Time_Of_Day` to `Standard.Base`.
- Renamed `Zone` to `Time_Zone`.
- Added `century`.
- Added `is_leap_year`.
- Added `length_of_year`.
- Added `length_of_month`.
- Added `quarter`.
- Added `day_of_year`.
- Added `Day_Of_Week` type and `day_of_week` function.
- Updated `week_of_year` to support ISO.

# Important Notes
- Had to pass locale to formatter for date/time tests to work on my PC.
- Changed default of `week_of_year` to use ISO.
2022-08-26 15:47:58 +00:00
Hubert Plociniczak
d87a32d019
Builtin Date_Time, Time_Of_Day, Zone (#3658)
* Builtin Date_Time, Time_Of_Day, Zone

Improved polyglot support for Date_Time (formerly Time), Time_Of_Day and
Zone. This follows the pattern introduced for Enso Date.

Minor caveat - in tests for Date, had to bend a lot for JS Date to pass.
This is because JS Date is not really only a Date, but also a Time and
Timezone, previously we just didn't consider the latter.
Also, JS Date does not deal well with setting timezones so the trick I
used is to first call foreign function returning a polyglot JS Date,
which is converted to ZonedDateTime and only then set the correct
timezone. That way none of the existing tests had to be changes or
special cased.

Additionally, JS deals with milliseconds rather than nanoseconds so
there is loss in precision, as noted in Time_Spec.

* Add tests for Java's LocalTime

* changelog

* Make date formatters in table happy

* PR review, add more tests for zone

* More tests and fixed a bug in column reader

Column reader didn't take into account timezone but that was a mistake
since then it wouldn't map to Enso's Date_Time.
Added tests that check it now.

* remove redundant conversion

* Update distribution/lib/Standard/Base/0.0.0-dev/src/Data/Time.enso

Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>

* First round of addressing PR review

* don't leak java exceptions in Zone

* Move Date_Time to top-level module

* PR review

Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
Co-authored-by: Jaroslav Tulach <jaroslav.tulach@enso.org>
2022-08-24 12:31:29 +02:00
James Dunkerley
684adcb7fb
Tidy up the default imports for Standard.Table (#3660)
- Removed various unnecessary `Standard.Base` imports still left behind.
- Added `Regex` to default `Standard.Base`.
- Removed aliasing from the examples as no longer needed (case coercion no long occurs).
- Remove `import Standard.Table` from within the Table library (directly importing types).
- Reviewed what was in `Standard.Database` - a few tweaks and removals.
- Removed various un-needed aliasing following Hubert's import work.
2022-08-22 19:21:54 +00:00
Radosław Waśko
bcca7f10d9
Add key functions to Table to make it act as [Column] (#3644)
Implements https://www.pivotaltracker.com/story/show/181370836
2022-08-18 12:33:02 +00:00
Radosław Waśko
fbf6c800f1
Short hand version for order_by (#3643)
Implements https://www.pivotaltracker.com/story/show/182868310
2022-08-16 15:41:37 +00:00
Radosław Waśko
3dca738cf7
Add Vector.take and Vector.drop functions (#3629)
Implements https://www.pivotaltracker.com/story/show/182307048
2022-08-10 16:02:02 +00:00
Radosław Waśko
0a2fea925c
Create Index_Sub_Range type and update Text.take and Text.drop (#3617) 2022-08-03 11:41:34 +00:00
Hubert Plociniczak
d59714a29d
Support module imports using a qualified name (#3608)
This change allows for importing modules using a qualified name and deals with any conflicts on the way.
Given a module C defined at `A/B/C.enso` with
```
type C
type C a
```
it is now possible to import it as
```
import project.A
...
val x = A.B.C 10
```

Given a module located at `A/B/C/D.enso`, we will generate
intermediate, synthetic, modules that only import and export the successor module along the path.
For example, the contents of a synthetic module B will look like
```
import <namespace>.<pkg-name>.A.B.C
export <namespace>.<pkg-name>.A.B.C
```
If module B is defined already by the developer, the compiler will _inject_ the above statements to the IR.

Also removed the last elements of some lowercase name resolution that managed to survive recent
changes (`Meta.Enso_Project` would now be ambiguous with `enso_project` method).

Finally, added a pass that detects shadowing of the synthetic module by the type defined along the path.
We print a warning in such a situation.

Related to https://www.pivotaltracker.com/n/projects/2539304

# Important Notes
There was an additional request to fix the annoying problem with `from` imports that would always bring
the module into the scope. The changes in stdlib demonstrate how it is now possible to avoid the workaround of
```
from X.Y.Z as Z_Module import A, B
```
(i.e. `as Z_Module` part is almost always unnecessary).
2022-07-29 14:19:07 +00:00
Hubert Plociniczak
f63e40df1b
Explicit self (#3569)
This change modifies the current language by requiring explicit `self` parameter declaration
for methods. Methods without `self` parameter in the first position should be treated as statics
although that is not yet part of this PR. We add an implicit self to all methods
This obviously required updating the whole stdlib and its components, tests etc but the change
is pretty straightforward in the diff.

Notice that this change **does not** change method dispatch, which was removed in the last changes.
This was done on purpose to simplify the implementation for now. We will likely still remove all
those implicit selfs to bring true statics.
Minor caveat - since `main` doesn't actually need self, already removed that which simplified
a lot of code.
2022-07-27 17:45:36 +00:00
James Dunkerley
a54a7d5553
Tidying up what is in Standard.Base (#3603)
- Added various of the types from the new APIs to the Standard.Base export.
- Removed Syntax_Error types for Regex and Uri and used the common one.
2022-07-27 13:28:00 +00:00
Radosław Waśko
ee91656f30
Remove duplicate Line_Ending_Style and update defaults (#3597)
Implements https://www.pivotaltracker.com/story/show/182749831
2022-07-27 09:43:51 +00:00
James Dunkerley
7090e1fb91
Docker file for testing Postgres SSL and updated Postgres Spec (#3607)
Adds a Dockerfile and `CreatePostgresSSL.sh` script, which makes an Alpine based Postgres server with a self signed certificate. The script will drop the generated `rootCA.crt` into the `data/transient` folder.

This can then be included in the test by setting the environment variable `ENSO_DATABASE_TEST_CA_CERT_FILE`.

Test has been updated to check the various SSL connection modes.
2022-07-26 13:28:43 +00:00
James Dunkerley
be311457bd
Add Linear Regression support for Vectors. (#3601)
Adds least squares regression APIs. Covers the basic 4 trend line types from Excel (doesn't cover Polynomial or Moving Average).
Removes the old `Model` from the `Standard.Table`.
2022-07-22 08:41:17 +00:00
Radosław Waśko
16fd038c1a
Add support for .pgpass to PostgreSQL (#3593)
Implements https://www.pivotaltracker.com/story/show/182582924
2022-07-21 13:32:37 +00:00
Jaroslav Tulach
4465d63dd8
Improved polyglot Date support (#3559)
Significantly improves the polyglot Date support (as introduced by #3374). It enhances the `Date_Spec` to run it in four flavors:
- with Enso Date (as of now)
- with JavaScript Date
- with JavaScript Date wrapped in (JavaScript) array
- with Java LocalDate allocated directly

The code is then improved by necessary modifications to make the `Date_Spec` pass.

# Important Notes
James has requested in [#181755990](https://www.pivotaltracker.com/n/projects/2539304/stories/181755990) - e.g. _Review and improve InMemory Table support for Dates, Times, DateTimes, BigIntegers_ the following program to work:
```
foreign js dateArr = """
return [1, new Date(), 7]

main =
IO.println <| (dateArr.at 1).week_of_year
```
the program works with here in provided changes and prints `27` as of today.

@jdunkerley has provided tests for proper behavior of date in `Table` and `Column`. Those tests are working as of [f16d07e](f16d07e640). One just needs to accept `List<Value>` and then query `Value` for `isDate()` when needed.

Last round of changes is related to **exception handling**. 8b686b12bd makes sure `makePolyglotError` accepts only polyglot values. Then it wraps plain Java exceptions into `WrapPlainException` with `has_type` method - 60da5e70ed - the remaining changes in the PR are only trying to get all tests working in the new setup.

The support for `Time` isn't part of this PR yet.
2022-07-21 06:32:40 +00:00
James Dunkerley
5e4083978f
Type name case fixes: (#3590)
- MacOS => Mac_OS
- PostgreSQL => Postgres
- SQLite => SQLite (align a few)
- InMemory => In_Memory
- PointData => Point_Data
- Io_Error => IO_Error
- Standard.Table.Io => Standard.Table.IO

In Tests:
- MyError => My_Error
- NotFoo => Not_Foo
2022-07-19 14:09:09 +00:00
Radosław Waśko
fc110659db
Implement should_succeed (#3586)
Implements https://www.pivotaltracker.com/story/show/182709976
2022-07-14 19:58:44 +00:00
Radosław Waśko
35ddd2a89e
Add new options to the Delimited format (#3581)
Implements https://www.pivotaltracker.com/story/show/182662195 and https://www.pivotaltracker.com/story/show/182651884
2022-07-14 15:01:26 +00:00
James Dunkerley
9578dc1e43
Move write_bytes to be part of Vector. (#3583)
Updates `write_bytes` API to be part of `Vector` and to conform to `write` APIs.

# Important Notes
Ensures doesn't touch the file if an invalid byte array.
2022-07-14 11:30:40 +00:00
James Dunkerley
e41936f436
Additional tests for Excel Append (#3580)
Add some additional scenarios to Excel append tests:
- Non-A1 start
- Name duplication
- Hitting another range

# Important Notes
Also fixed a warning in the Image library.
2022-07-13 13:02:39 +00:00
James Dunkerley
2527a7bdb2
Update SQLite, PostgreSQL and Redshift drivers (#3571)
Updated the SQLite, PostgreSQL and Redshift drivers.

# Important Notes
Updated the API for Redshift and proved able to connect without the ini file workaround.
2022-07-11 18:39:16 +00:00
Radosław Waśko
df10e4ba7c
Add appending support for Delimited files (#3573)
Implements https://www.pivotaltracker.com/story/show/182309839
2022-07-11 12:36:01 +00:00
Hubert Plociniczak
96e50648dd
Remove 'here' and make method name resolution case-sensitive (#3538)
Modified UppercaseNames to now resolve methods without an explicit `here` to point to the current module.
`here` was also often used instead of `self` which was allowed by the compiler.
Therefore UppercaseNames pass is now GlobalNames and does some extra work -
it translated method calls without an explicit target into proper applications.

# Important Notes
There was a long-standing bug in scopes usage when compiling standalone expressions.
This resulted in AliasAnalysis generating incorrect graphs and manifested itself only in unit tests
and when running `eval`, thus being a bit hard to locate.
See `runExpression` for details.

Additionally, method name resolution is now case-sensitive.

Obsolete passes like UndefinedVariables and ModuleThisToHere were removed. All tests have been adapted.
2022-07-07 10:31:06 +00:00
James Dunkerley
16e6f2fa08
Adding Append support to Excel.Write (#3558)
Adds support for appending to an existing Excel table.

# Important Notes
- Renamed `Column_Mapping` to `Column_Name_Mapping`
- Changed new type name to `Map_Column`
- Added last modified time and creation time to `File`.
2022-07-07 06:41:33 +00:00
Radosław Waśko
7c94fa6a77
Custom Encoding support when writing Delimited files (#3564)
Implements https://www.pivotaltracker.com/story/show/182545847
2022-07-07 00:20:00 +00:00
James Dunkerley
5174cc6ece
Update Database.connect to match new API (#3542)
Initial work restructuring the `Database.connect` API
- New SQLite API with support for InMemory.
- Updated PostgreSQL API with SSL and Client Certificate Support.
- Updated Redshift API.

# Important Notes
Follow up tasks:
- PostgreSQL SSL additional testing.
- Driver version updating.
- `.pgpass` support.
2022-07-04 20:26:44 +00:00
James Dunkerley
4ca2097488
Adding write support to File_Format.Excel (#3551)
Support for writing tables to Excel.

# Important Notes
Has custom support for Error mode as will allow appending a new table in this mode to the file.
2022-07-04 18:32:16 +00:00
Radosław Waśko
972b34d1a9
Implement value formatting and writing new files in Delimited format. (#3528)
Implements https://www.pivotaltracker.com/story/show/182309429 and https://www.pivotaltracker.com/story/show/182309573
2022-06-23 16:51:52 +00:00
James Dunkerley
7a2d304fa0
Update Excel reading API (#3523)
- Remove `from_xls` and `from_xlsx`.
- Add `headers` support to `File_Format.Excel`.
- Altered default read for Excel to be the first sheet.
- Altered behavior so that single cells grow down and right when reading sheet.
- Altered `Excel_Range` so knows if single cell or 1x1 range address.

# Important Notes
- Renamed `Range` to `Cell_Range` to avoid name clash.
2022-06-21 13:39:32 +00:00
Hubert Plociniczak
22a371a9c6
Substitute this with self (#3524)
A semi-manual s/this/self appied to the whole standard library.
Related to https://www.pivotaltracker.com/story/show/182328601

In the compiler promoted to use constants instead of hardcoded
`this`/`self` whenever possible.

# Important Notes
The PR **does not** require explicit `self` parameter declaration for methods as this part
of the design is still under consideration.
2022-06-21 10:53:52 +00:00
James Dunkerley
a0c6fa9c96
Removing old functions and tidy up of Table types (#3519)
- Removed `select` method.
- Removed `group` method.
- Removed `Aggregate_Table` type.
- Removed `Order_Rule` type.
- Removed `sort` method from Table.
- Expanded comments on `order_by`.
- Update comment on `aggregate` on Database.
- Update Visualisation to use new APIs.
- Updated Data Science examples to use new APIs.
- Moved Examples test out of Tests to own test.

# Important Notes
Need to get Examples_Tests added to CI.
2022-06-14 13:37:20 +00:00
Radosław Waśko
e83c36d9d6
Add scaffolding for Table.write function (#3521)
Implements https://www.pivotaltracker.com/story/show/182309559

This task implements common scaffolding for the `Table.write`, so that the particular implementations for Delimited and Excel file formats can be done in parallel.
2022-06-14 11:29:03 +00:00
Radosław Waśko
a04825a5ce
Add Text.write Function (#3518)
Implements https://www.pivotaltracker.com/story/show/182309026
2022-06-13 09:11:46 +00:00
James Dunkerley
e97d27e1e0
Adjusting First and Last order_by to use Sort_Column_Selector (#3517) 2022-06-10 09:59:03 +00:00
James Dunkerley
8afba43add
Implement In-Memory Table order_by (#3515)
Implemented the `order_by` function with support for all modes of operation.
Added support for case insensitive natural order.

# Important Notes
- Improved MultiValueIndex/Key to not create loads of arrays.
- Adjusted HashCode for MultiValueKey to have a simple algorithm.
- Added Text_Utils.compare_normalized_ignoring_case to allow for case insensitive comparisons.
- Fixed issues with ObjectComparator and added some unit tests for it.
2022-06-08 12:30:50 +00:00
Radosław Waśko
2af970fe52
Basic changes to File_Format (#3516)
Implements https://www.pivotaltracker.com/story/show/182308987
2022-06-08 09:53:18 +00:00
Radosław Waśko
a382e0c15e
Improve database Table.order_by (#3514)
Implements https://www.pivotaltracker.com/story/show/182195405

Adds support for the Postgres dialect and simple case insensitive collation for SQLite.
2022-06-07 12:31:55 +00:00
Radosław Waśko
7d94efa6f2
Implement Table.order_by for SQLite and the common scaffolding for all backends (#3502)
Implements the common and SQLite parts of https://www.pivotaltracker.com/story/show/182195405
2022-06-06 10:56:52 +00:00
James Dunkerley
ba5d6823a9
Merge the Unique Name Strategy with NameDeduplicator (#3490)
- Merge the two approaches and makes them consistent
- Add warning support into Reader

# Important Notes
- Added support for JUnit format XML generation on tests. Use `ENSO_TEST_JUNIT_DIR`
2022-06-01 12:52:23 +00:00
Radosław Waśko
f0f3a343eb
Adjust Table.sort_columns to use Text_Ordering design (#3487)
Implements https://www.pivotaltracker.com/story/show/182195306
2022-05-30 12:26:29 +00:00
Radosław Waśko
db611e1581
Remove obsolete Csv reading module (#3482)
Completes https://www.pivotaltracker.com/story/show/182037405

# Important Notes
- Some tests had to be adapted to the new parsing logic.
2022-05-28 10:01:14 +00:00
Radosław Waśko
8828d801ea
Implement Table from Text conversion (#3478)
Implements https://www.pivotaltracker.com/story/show/181824168
2022-05-26 12:04:25 +00:00
Radosław Waśko
7f572bf3e4
The user should be able to have the headers Inferred when reading a Delimited file (#3472)
Implements https://www.pivotaltracker.com/story/show/181986831
2022-05-25 13:29:17 +00:00
Radosław Waśko
ec1b072824
Integrate value parsing with Delimited file reading (#3463)
Implements https://www.pivotaltracker.com/story/show/182200028
2022-05-24 17:59:00 +02:00
Radosław Waśko
ff7700ebb1
Automatic inference of value types when parsing table columns (#3462)
Implements https://www.pivotaltracker.com/story/show/182199966
2022-05-20 15:08:36 +00:00
Radosław Waśko
8430ce2625
Parsing values with known types (#3455)
Implements https://www.pivotaltracker.com/story/show/181824146
2022-05-18 15:27:48 +00:00
Hubert Plociniczak
6b6b1430bc
Cleanup Ref - get/put (#3457)
The change promotes static methods of `Ref`, `get` and `put`, to be
methods of `Ref` type.
The change also removes `Ref` module from the default namespace.
Had to mostly c&p functional dispatch for now, in order for the methods
to be found. Will auto-generate that code as part of builtins system.

Related to https://www.pivotaltracker.com/story/show/182138899
2022-05-17 10:26:36 +00:00
James Dunkerley
4f3a76817c
Statistics on a Vector (#3442)
- Implements various statistics on Vector

# Important Notes
Some minor codebase improvements:
- Some tweaks to Any/Nothing to improve performance
- Fixed bug in ObjectComparator
- Added if_nothing
- Removed Group_By_Key
2022-05-11 13:25:06 +00:00
Radosław Waśko
64f178f7a8
Delimited File Encoding (#3430)
Implements https://www.pivotaltracker.com/story/show/181998375
2022-05-10 22:44:05 +00:00
James Dunkerley
078c665a60
File_Format.Excel work (#3425)
- Read in Excel files following the specification.
- Support for XLSX and XLS formats.
- Ability to select ranges and sheets.
- Skip Rows and Row Limits.

# Important Notes
- Minor fix to DelimitedReader for Windows
2022-05-06 13:21:10 +00:00
Radosław Waśko
8219dca400
Improve support for reading Delimited files (#3424)
Implements https://www.pivotaltracker.com/story/show/181823957
2022-04-29 17:12:19 +00:00
Radosław Waśko
14257d07aa
Data analysts should be able to use Text.split, Text.lines and Text.words to break up strings (#3415)
Implements https://www.pivotaltracker.com/story/show/181266184

### Important Notes

Changed example image download to only proceed if the file did not exist before - thus cutting on the build time (the build used to download it _every_ time - which completely failed the build if network is down). A redownload can be forced by performing a fresh repository checkout.
2022-04-26 17:22:53 +02:00
James Dunkerley
5a6b6749cc
Restructuring for File.read (#3390)
- Added Encoding type
- Added `Text.bytes`, `Text.from_bytes` with Encoding support
- Renamed `File.read` to `File.read_text`
- Renamed `File.write` to `File.write_text`
- Added Encoding support to `File.read_text` and `File.write_text`
- Added warnings to invalid encodings
2022-04-19 16:50:03 +00:00
Radosław Waśko
0ea5dc2a6f
Data analysts should be able to use Text.replace to substitute parts of the text (#3393)
Implements https://www.pivotaltracker.com/story/show/181266274
2022-04-13 19:21:47 +00:00
Radosław Waśko
891f064a6a
Extend Aggregate_Spec test suite with tests for missed edge-cases to ensure the feature is well-tested on all backends (#3383)
Implements https://www.pivotaltracker.com/story/show/181805693 and finishes the basic set of features of the Aggregate component.

Still not all aggregations are supported everywhere, because for example SQLite has quite limited support for aggregations. Currently the workaround is to bring the table into memory (if possible) and perform the computation locally. Later on, we may add more complex generator features to emulate the missing aggregations with complex sub-queries.
2022-04-12 11:02:01 +00:00
James Dunkerley
bade0c31de
First and Last ordering (#3380)
Add the missing `order_by` support to First and Last aggregations for InMemory table.
2022-04-06 12:36:46 +00:00
Radosław Waśko
a71db71645
Adding most of remaining aggregates to Database Table (#3375) 2022-04-06 10:06:50 +00:00
James Dunkerley
a4dbc9a37b
Moving Aggregation to Java (#3364) 2022-04-04 09:12:48 +00:00
Radosław Waśko
43265f10a8
Implement Error-Handling for Database aggregations, unify some error helpers across backends (#3371) 2022-03-31 12:10:22 +00:00
Radosław Waśko
20be5516a5
Aggregates in the Database library - MVP (#3353)
Implements infrastructure for new aggregations in the Database. It comes with only some basic aggregations and limited error-handling. More aggregations and problem handling will be added in subsequent PRs.

# Important Notes
This introduces basic aggregations using our existing codegen and sets-up our testing infrastructure to be able to use the same aggregate tests as in-memory backend for the database backends.

Many aggregations are not yet implemented - they will be added in subsequent tasks.

There are some TODOs left - they will be addressed in the next tasks.
2022-03-28 15:51:37 +00:00
James Dunkerley
02bcfbb2a8
Refactor Aggregate Column (#3349)
- Make it easier to understand the computations.
- Fix issue with First.
- Improve quote handling in Concatenate
- Added validation and warnings to input
2022-03-22 18:18:46 +00:00
James Dunkerley
6c1c4554f5
Refactor table.group_by to table.aggregate (#3339)
Following UX work move to `table.aggregate` function.
2022-03-15 15:23:36 +01:00
Radosław Waśko
dedd1eac96
Refactor library warnings to use the new system (#3337)
Implements https://www.pivotaltracker.com/story/show/181536964
2022-03-15 12:52:57 +01:00
James Dunkerley
65465fb8ef
Restructuring the Faker type and creating tests for Group_By (#3318)
- Added Minimum, Maximum, Longest. Shortest, Mode, Percentile
- Added first and last to Map
- Restructured Faker type more inline with FakerJS
- Created 2,500 row data set
- Tests for group_by
- Performance tests for group_by
2022-03-09 10:31:02 +00:00
James Dunkerley
738a691662
Table.group_by (#3305)
Functioning group_by based of Enso Map.

# Important Notes
This is an initial version which will be used to establish the API.
The grouping map will need to be moved to Java code for performance.
2022-03-01 16:18:11 +00:00
Radosław Waśko
b03416f907
Update Column_Selector and Column_Mapping to use Matcher over Matching_Strategy (#3299)
Implements https://www.pivotaltracker.com/story/show/181339748
2022-02-25 18:39:10 +00:00
Radosław Waśko
ae9d51555f
Data analysts should be able to use Text.contains to check for substring using various matcher techniques. (#3285)
* Add matching mode definitions

* Add stub for new method API and an initial test suite

* Fix tests, implement exact matching

* Implement Regex matching

* changelog

* Add benchmarks

* Wokraround for case insensitive regex locale support

* minor tweaks

* Unify Case_Insensitive

* Update edge cases

* Fix other affected places

* minor style change

* Add a problematic test

* Add a regex test for a similar situation

* Migrate to StringSearch:wq

* Add test cases for scharfes S edge case

* Add problematic Regex Unicode normalization test

* Document the regex accents peculiarity

* Do not apply the normalization in ASCII only mode

* cr
2022-02-22 15:41:56 +00:00
James Dunkerley
1814d3c4f1
Data analysts should be able to transform a Table using the rename_columns functions (#3249)
* Implement Natural_Order and sort_columns

* Starting on Rename

Align Column_Mapping

Add By_Position
Separating off the validation for By_Index so can reuse for rename

By_Position implemented

By_Index implemented
Adjusted behaviour following discussion with Ned, so that renames dominate untouched columns.

Moving to validation style checks for problems

Putting accumulator back

Rename work

* Add Range.find

* More work

* Regex support
Tidy of Unique Name Strategy

* Fix Regex support

* Warning messages
Tests for Unique Naming Strategy
Table rename working

* Database Table rename_columns
Fix for Table
**Must follow up on slice**

* Some tests

* More tests

* Complete test set
(and associated fixes)

* Functional use_first_row_as_names
Tests to go...

* Test for use_first_row_as_names

* Change log

* trailing space

Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
2022-02-11 10:18:51 +00:00
Radosław Waśko
8b24336604
Data analysts should be able to reorder columns into name order using sort_columns functions (#3250) 2022-02-08 17:28:46 +01:00
Radosław Waśko
d3c0f968fa
Data analysts should be able to transform a Table using the remove_columns and reorder_columns functions (#3240) 2022-02-03 15:18:47 +01:00
Radosław Waśko
b5fc87e618
Data analysts should be able to transform a Table using the select_columns function (#3230)
* Utility for mapping errors and warnings
* Imlpement By_Index
* Expose select_columns in InMem and DB. Need testing
* checkpoint: writing tests
* Fix minor issues, mock warning mapping for testing purposes
* Improve By_Index error handling
* A helper for testing problem handling
* More error handling
* docs
* changelog
* Fix matching test
* Add SQLite tests
* cleanup after test
* Rework problem handling
* small refactor
* add examples
* Add more test cases for regex matching
* Fix Regex.Patter.matches to match full string
* "Fix" tests
2022-02-02 09:04:06 +00:00
Radosław Waśko
107128aeec
A library developer should be able to select matching names given a list (#3220) 2022-01-20 11:11:43 +01:00
Ara Adkins
337f6c8ad4
Implement linear regression on tables (#2003) 2021-09-29 15:33:18 +01:00
Marcin Kostrzewa
4f4e472ddf
Statistical functions (#1990) 2021-09-06 14:48:09 +02:00
Ara Adkins
c12cab9bd9
Add Column.set_index (#1982) 2021-09-02 10:30:02 +01:00
Marcin Kostrzewa
4536ed9f9b
Stdlib Improvements (#1963) 2021-08-19 14:55:15 +02:00
Marcin Kostrzewa
98eab2873e
Allow specifying a cell range when reading spreadsheets (#1954) 2021-08-16 17:01:33 +02:00
Marcin Kostrzewa
ad0b677ed8
Entry point for writing tables (#1946) 2021-08-12 15:16:24 +02:00
Marcin Kostrzewa
ca8252c9cf
Table to JSON serialization (#1937) 2021-08-10 15:35:51 +02:00
Marcin Kostrzewa
9ce6eb0560
Write XLSX files (#1906) 2021-07-28 13:51:27 +02:00
Marcin Kostrzewa
ca52757c10
CSV Writing (#1894) 2021-07-22 15:13:00 +02:00
Marcin Kostrzewa
f55d66cb2c
XLS(X) Reading (#1879) 2021-07-20 13:32:19 +02:00
Marcin Kostrzewa
334a022ffd
Import syntax including namespace (#1806) 2021-06-24 12:42:24 +02:00
Marcin Kostrzewa
b4709ab529
Default visualization definitions (#1786) 2021-06-08 08:12:02 +02:00
Ara Adkins
c4c483683e
Improve error types in the standard library (#1734) 2021-05-11 10:19:30 +01:00
Ara Adkins
6060d31c79
Update examples for Standard.Base.Data.* (#1707) 2021-04-29 11:27:16 +01:00
Radosław Waśko
117ca51921
Improve how indexing in Table works (#1643) 2021-04-01 14:39:31 +01:00
Ara Adkins
9585080ab8
Clean up the standard library docs (#1641) 2021-04-01 12:20:36 +01:00
Dmitry Bushev
5cfd9284be
Convert GeoJSON to Table (#1632) 2021-03-30 15:06:22 +01:00
Ara Adkins
6ee0c19d53
Implement additional methods for table (#1628) 2021-03-29 17:34:06 +01:00
Radosław Waśko
49b30f2e9d
Database Visualization Support (#1582) 2021-03-18 14:28:52 +01:00
Ara Adkins
96697ddc97
Fix a crash due to shadowed project names (#1571) 2021-03-16 12:45:19 +00:00
Radosław Waśko
5f8af886e5
Connection and Materialization in the Database Library (#1546) 2021-03-09 19:52:42 +01:00
Marcin Kostrzewa
3dd348c1be
Table: Fix bool column sorting (#1505) 2021-02-24 17:36:24 +01:00
Marcin Kostrzewa
14dd4006bb
Table API: contatenation, index access, column aggregation, API unification (#1489) 2021-02-18 16:00:19 +01:00
Marcin Kostrzewa
05945ede90
Table Visualization Fixes (#1476) 2021-02-15 09:55:54 +01:00
Marcin Kostrzewa
93b6680d4f
Sorting Tables (#1471) 2021-02-11 16:50:07 +01:00
Ara Adkins
af1aab35aa
Improve dataflow errors in the standard library (#1446) 2021-02-02 12:31:33 +00:00
Marcin Kostrzewa
197190ceeb
Remove UFCS (#1398) 2021-01-14 21:53:04 +01:00
Marcin Kostrzewa
b751dfb3ec
Table: grouping (#1392) 2021-01-11 17:05:06 +01:00
Radosław Waśko
58346917eb
Implement Some Vectorized Text Operations And Dropping Missing (#1381) 2021-01-04 14:24:08 +01:00
Radosław Waśko
ab51bffd87
Implement fill_missing (#1372) 2020-12-22 23:10:27 +01:00
Marcin Kostrzewa
bf37754428
Table: maps, zips & more builtins (#1356) 2020-12-16 11:23:23 +01:00
Marcin Kostrzewa
a40989e7c6
Table: Indexes & Joins (#1317) 2020-11-30 16:21:55 +01:00
Marcin Kostrzewa
ab2c5ed097
Tables: column mapping & masking (#1297) 2020-11-18 15:09:43 +01:00
Marcin Kostrzewa
f420dd3702
Rename Unit to Nothing (#1269) 2020-11-06 12:44:11 +01:00
Marcin Kostrzewa
150771c0e2
Simple CSV parser (#1268) 2020-11-05 16:53:50 +01:00