Commit Graph

30 Commits

Author SHA1 Message Date
James Dunkerley
185378f07c
Moving library statics to type for Table. (#3760)
- Generally export types not modules from the `Standard.Table` import.
- Moved `new`, `from_rows` the `Standard.Table` library into the `Table` type.
- Renames `Standard.Table.Data.Storage.Type` to `Standard.Table.Data.Storage.Storage`
- Removed the internal `from_columns` method.
- Removed `join` and `concat` and merged into instance methods.
- Removed `Table` and `Column` from the `Standard.Database` exports.
- Removed `Standard.Table.Data.Column.Aggregate_Column` as not used any more.
2022-10-06 17:01:18 +00:00
Radosław Waśko
503d3eb5a3
Update Table API with new filter design (#3750)
Implements https://www.pivotaltracker.com/story/show/183389855

# Important Notes
Implements basic filter operations both In-Memory and for the Database backend, ensuring that existing tests can be adapted and keep working. Not all `Filter_Condition`s are implemented yet.

Also implements significant part of https://www.pivotaltracker.com/story/show/183390314
2022-10-05 11:40:10 +00:00
Marcin Kostrzewa
726bfeb883
Wip/mk/statics (#3740)
Makes statics static. A type and its instances have different methods defined on them, as it should be. Constructors are now scoped in types, and can be imported/exported.

# Important Notes
The method of fixing stdlib chosen here is to just not. All the conses are exported to make all old code work. All such instances are marked with `TODO Dubious constructor export` so that it can be found and fixed.
2022-09-27 14:23:06 +00:00
James Dunkerley
6f54e80970
Adjust Database connection to use query/read to access data. (#3727)
Adjust Database connection API to align with new [design](https://github.com/enso-org/design/blob/wip/jd/database-read/epics/basic-libraries/database-read/design.md#querying-tables).
- `query` replaces the old `access_table` and is expanded to support raw SQL queries.
- `read` replaces `execute_query` and matches the API of `query`.
- `to_dataframe` is renamed to `read`.

# Important Notes
Added support for `++` to concatenate a Text without wrapping in a `SQL.Code`.
2022-09-23 07:35:08 +00:00
James Dunkerley
d6346e9d66
Renaming various constructors and moving types around for Database. (#3715)
Repairing the constructor name following the types work. Some general tiding up as well.

- Remove `Standard.Database.Data.Column.Aggregate_Column_Builder`.
- Remove `Standard.Database.Data.Dialect.Dialect.Dialect_Data`.
- Remove unused imports and update some type definitions.
- Rename `Postgres.Postgres_Data` => `Postgres_Options.Postgres`.
- Rename `Redshift.Redshift_Data` => `Redshift_Options.Redshift`.
- Rename `SQLite.SQLite_Data` => `SQLite_Options.SQLite`.
- Rename `Credentials.Credentials_Data` => `Credentials.Username_And_Password`.
- Rename `Sql` to `SQL` across the board.
- Merge `Standard.Database.Data.Internal` into `Standard.Database.Internal`.
- Move dialects into `Internal` and merge the function in `Helpers` into `Base_Generator`.
2022-09-19 12:39:40 +00:00
Hubert Plociniczak
0e5df935d3
Don't rename imported Main module that only imports names (#3710)
Turns that if you import a two-part import we had special code that would a) add Main submodule b) add an explicit rename.

b) is problematic because sometimes we only want to import specific names.
E.g.,
```
from Bar.Foo import Bar, Baz
```
would be translated to
```
from Bar.Foo.Main as Foo import Bar, Baz
```
and it should only be translated to
```
from Bar.Foo.Main import Bar, Baz
```

This change detects this scenario and does not add renames in that case.

Fixes [183276486](https://www.pivotaltracker.com/story/show/183276486).
2022-09-16 13:01:06 +00:00
James Dunkerley
0126f02e7b
Restructure File.read into the new design (#3701)
Changes following Marcin's work. Should be back to very similar public API as before.

- Add an "interface" type: `Standard.Base.System.File_Format.File_Format`.
- All `File_Format` types now have a `can_read` method to decide if they can read a file.
- Move `Standard.Table.IO.File_Format.Text.Text_Data` to `Standard.Base.System.File_Format.Plain_Text_Format.Plain_Text`.
- Move `Standard.Table.IO.File_Format.Bytes` to `Standard.Base.System.File_Format.Bytes`.
- Move `Standard.Table.IO.File_Format.Infer` to `Standard.Base.System.File_Format.Infer`. **(doesn't belong here...)**
- Move `Standard.Table.IO.File_Format.Unsupported_File_Type` to `Standard.Base.Error.Common.Unsupported_File_Type`.
- Add `Infer`, `File_Format`, `Bytes`, `Plain_Text`, `Plain_Text_Format` to `Standard.Base` exports.
- Fold extension methods of `Standard.Base.Meta.Unresolved_Symbol` into type.
- Move `Standard.Table.IO.File_Format.Auto` to `Standard.Table.IO.Auto_Detect.Auto_Detect`.
- Added a `types` Vector of all the built in formats.
- `Auto_Detect` asks each type if they `can_read` a file.
- Broke up and moved `Standard.Table.IO.Excel` into `Standard.Table.Excel`:
- Moved `Standard.Table.IO.File_Format.Excel.Excel_Data` to `Standard.Table.Excel.Excel_Format.Excel_Format.Excel`.
- Renamed `Sheet` to `Worksheet`.
- Internal types `Reader` and `Writer` providing the actual read and write methods.
- Created `Standard.Table.Delimited` with similar structure to `Standard.Table.Excel`:
- Moved `Standard.Table.IO.File_Format.Delimited.Delimited_Data` to `Standard.Table.Delimited.Delimited_Format.Delimited_Format.Delimited`.
- Moved `Standard.Table.IO.Quote_Style` to `Standard.Table.Delimited.Quote_Style`.
- Moved the `Reader` and `Writer` internal types into here. Renamed methods to have unique names.
- Add `Aggregate_Column`, `Auto_Detect`, `Delimited`, `Delimited_Format`, `Excel`, `Excel_Format`, `Sheet_Names`, `Range_Names`, `Worksheet` and `Cell_Range` to `Standard.Table` exports.
2022-09-15 14:48:46 +00:00
Hubert Plociniczak
fba5047acc
Improved Vector/Array interop (#3667)
`Vector` type is now a builtin type. This requires a bunch of additional builtin methods for its creation:
- Use `Vector.from_array` to convert any array-like structure into a `Vector` [by copy](f628b28f5f)
- Use (already existing) `Vector.from_polyglot_array` to convert any array-like structure into a `Vector` **without** copying
- Use (already existing) `Vector.fill 1 item` to create a singleton `Vector`

Additional, for pattern matching purposes, we had to implement a `VectorBranchNode`. Use following to match on `x` being an instance of `Vector` type:
```
import Standard.Base.Data.Vector

size = case x of
Vector.Vector -> x.length
_ -> 0
```

Finally, `VectorLiterals` pass that transforms `[1,2,3]` to (roughly)
```
a1 = 1
a2 = 2
a3 = 3
Vector (Array (a1,a2, a3))
```
had to be modified to generate
```
a1 = 1
a2 = 2
a3 = 3
Vector.from_array (Array (a1, a2, a3))
```
instead to accomodate to the API changes. As of 025acaa676 all the known CI checks passes. Let's start the review.

# Important Notes
Matching in `case` statement is currently done via `Vector_Data`. Use:
```
case x of
Vector.Vector_Data -> True
```
until a better alternative is found.
2022-09-13 03:07:17 +00:00
Marcin Kostrzewa
4fc6dcced0
Get rid of free-floating atoms. Everything has a type now! (#3671)
This is a step towards the new language spec. The `type` keyword now means something. So we now have
```
type Maybe a
Some (from_some : a)
None
```
as a thing one may write. Also `Some` and `None` are not standalone types now – only `Maybe` is.
This halfway to static methods – we still allow for things like `Number + Number` for backwards compatibility. It will disappear in the next PR.

The concept of a type is now used for method dispatch – with great impact on interpreter code density.

Some APIs in the STDLIB may require re-thinking. I take this is going to be up to the libraries team – some choices are not as good with a semantically different language. I've strived to update stdlib with minimal changes – to make sure it still works as it did.

It is worth mentioning the conflicting constructor name convention I've used: if `Foo` only has one constructor, previously named `Foo`, we now have:
```
type Foo
Foo_Data f1 f2 f3
```

This is now necessary, because we still don't have proper statics. When they arrive, this can be changed (quite easily, with SED) to use them, and figure out the actual convention then.

I have also reworked large parts of the builtins system, because it did not work at all with the new concepts.

It also exposes the type variants in SuggestionBuilder, that was the original tiny PR this was based on.

PS I'm so sorry for the size of this. No idea how this could have been smaller. It's a breaking language change after all.
2022-08-30 22:54:53 +00:00
Radosław Waśko
fd318cfa96
Remove Array.set_at (#3634)
Implements https://www.pivotaltracker.com/story/show/182879865

# Important Notes
Note that removing `set_at` still does not make our arrays fully immutable - `Array.copy` can still be used to mutate them.
2022-08-26 09:34:33 +00:00
Hubert Plociniczak
4b9c91626e
Use Vector.from_polyglot_array to make Vectors backed by polyglot arrays (#3628)
Use Proxy_Polyglot_Array as a proxy for polyglot arrays, thus unifying
the way the underlying array is accessed in Vector.

Used the opportunity to cleanup builtin lookup, which now actually
respects what is defined in the body of @Builtin_Method annotation.

Also discovered that polyglot null values (in JS, Python and R) were leaking to Enso.
Fixed that by doing explicit translation to `Nothing`.

https://www.pivotaltracker.com/story/show/181123986
2022-08-23 21:13:16 +00:00
Jaroslav Tulach
2b9352d2fc
Lazy scatterplot for Vector & Table (#3655)
First of all this PR demonstrates how to implement _lazy visualization_:
- one needs to write/enhance Enso visualization libraries - this PR adds two optional parameters (`bounds` and `limit`) to `process_to_json_text` function.
- the `process_to_json_text` can be tested by standard Enso test harness which this PR also does
- then one has to modify JavaScript on the IDE side to construct `setPreprocessor` expression using the optional parameters

The idea of _scatter plot lazy visualization_ is to limit the amount of points the IDE requests. Initially the limit is set to `limit=1024`. The `Scatter_Plot.enso` then processes the data and selects/generates the `limit` subset. Right now it includes `min`, `max` in both `x`, `y` axis plus randomly chosen points up to the `limit`.

![Zooming In](https://user-images.githubusercontent.com/26887752/185336126-f4fbd914-7fd8-4f0b-8377-178095401f46.png)

The D3 visualization widget is capable of _zooming in_. When that happens the JavaScript widget composes new expression with `bounds` set to the newly visible area. By calling `setPreprocessor` the engine recomputes the visualization data, filters out any data outside of the `bounds` and selects another `limit` points from the new data. The IDE visualization then updates itself to display these more detailed data. Users can zoom-in to see the smallest detail where the number of points gets bellow `limit` or they can select _Fit all_ to see all the data without any `bounds`.

# Important Notes
Randomly selecting `limit` samples from the dataset may be misleading. Probably implementing _k-means clustering_ (where `k=limit`) would generate more representative approximation.
2022-08-23 12:12:22 +00:00
Hubert Plociniczak
f63e40df1b
Explicit self (#3569)
This change modifies the current language by requiring explicit `self` parameter declaration
for methods. Methods without `self` parameter in the first position should be treated as statics
although that is not yet part of this PR. We add an implicit self to all methods
This obviously required updating the whole stdlib and its components, tests etc but the change
is pretty straightforward in the diff.

Notice that this change **does not** change method dispatch, which was removed in the last changes.
This was done on purpose to simplify the implementation for now. We will likely still remove all
those implicit selfs to bring true statics.
Minor caveat - since `main` doesn't actually need self, already removed that which simplified
a lot of code.
2022-07-27 17:45:36 +00:00
James Dunkerley
a54a7d5553
Tidying up what is in Standard.Base (#3603)
- Added various of the types from the new APIs to the Standard.Base export.
- Removed Syntax_Error types for Regex and Uri and used the common one.
2022-07-27 13:28:00 +00:00
James Dunkerley
5e4083978f
Type name case fixes: (#3590)
- MacOS => Mac_OS
- PostgreSQL => Postgres
- SQLite => SQLite (align a few)
- InMemory => In_Memory
- PointData => Point_Data
- Io_Error => IO_Error
- Standard.Table.Io => Standard.Table.IO

In Tests:
- MyError => My_Error
- NotFoo => Not_Foo
2022-07-19 14:09:09 +00:00
Hubert Plociniczak
96e50648dd
Remove 'here' and make method name resolution case-sensitive (#3538)
Modified UppercaseNames to now resolve methods without an explicit `here` to point to the current module.
`here` was also often used instead of `self` which was allowed by the compiler.
Therefore UppercaseNames pass is now GlobalNames and does some extra work -
it translated method calls without an explicit target into proper applications.

# Important Notes
There was a long-standing bug in scopes usage when compiling standalone expressions.
This resulted in AliasAnalysis generating incorrect graphs and manifested itself only in unit tests
and when running `eval`, thus being a bit hard to locate.
See `runExpression` for details.

Additionally, method name resolution is now case-sensitive.

Obsolete passes like UndefinedVariables and ModuleThisToHere were removed. All tests have been adapted.
2022-07-07 10:31:06 +00:00
James Dunkerley
5174cc6ece
Update Database.connect to match new API (#3542)
Initial work restructuring the `Database.connect` API
- New SQLite API with support for InMemory.
- Updated PostgreSQL API with SSL and Client Certificate Support.
- Updated Redshift API.

# Important Notes
Follow up tasks:
- PostgreSQL SSL additional testing.
- Driver version updating.
- `.pgpass` support.
2022-07-04 20:26:44 +00:00
Hubert Plociniczak
22a371a9c6
Substitute this with self (#3524)
A semi-manual s/this/self appied to the whole standard library.
Related to https://www.pivotaltracker.com/story/show/182328601

In the compiler promoted to use constants instead of hardcoded
`this`/`self` whenever possible.

# Important Notes
The PR **does not** require explicit `self` parameter declaration for methods as this part
of the design is still under consideration.
2022-06-21 10:53:52 +00:00
James Dunkerley
a0c6fa9c96
Removing old functions and tidy up of Table types (#3519)
- Removed `select` method.
- Removed `group` method.
- Removed `Aggregate_Table` type.
- Removed `Order_Rule` type.
- Removed `sort` method from Table.
- Expanded comments on `order_by`.
- Update comment on `aggregate` on Database.
- Update Visualisation to use new APIs.
- Updated Data Science examples to use new APIs.
- Moved Examples test out of Tests to own test.

# Important Notes
Need to get Examples_Tests added to CI.
2022-06-14 13:37:20 +00:00
Hubert Plociniczak
4bbabc00be
Move Builtin Types and Methods to stdlib (#3363)
This PR replaces hard-coded `@Builtin_Method` and `@Builtin_Type` nodes in Builtins with an automated solution
that a) collects metadata from such annotations b) generates `BuiltinTypes` c) registers builtin methods with corresponding
constructors.
The main differences are:
1) The owner of the builtin method does not necessarily have to be a builtin type
2) You can now mix regular methods and builtin ones in stdlib 
3) No need to keep track of builtin methods and types in various places and register them by hand (a source of many typos or omissions as it found during the process of this PR)

Related to #181497846
Benchmarks also execute within the margin of error.

### Important Notes

The PR got a bit large over time as I was moving various builtin types and finding various corner cases.
Most of the changes however are rather simple c&p from Builtins.enso to the corresponding stdlib module.
Here is the list of the most crucial updates:
- `engine/runtime/src/main/java/org/enso/interpreter/runtime/builtin/Builtins.java` - the core of the changes. We no longer register individual builtin constructors and their methods by hand. Instead, the information about those is read from 2 metadata files generated by annotation processors. When the builtin method is encountered in stdlib, we do not ignore the method. Instead we lookup it up in the list of registered functions (see `getBuiltinFunction` and `IrToTruffle`)
- `engine/runtime/src/main/java/org/enso/interpreter/runtime/callable/atom/AtomConstructor.java` has now information whether it corresponds to the builtin type or not.
- `engine/runtime/src/main/scala/org/enso/compiler/codegen/RuntimeStubsGenerator.scala` - when runtime stubs generator encounters a builtin type, based on the @Builtin_Type annotation, it looks up an existing constructor for it and registers it in the provided scope, rather than creating a new one. The scope of the constructor is also changed to the one coming from stdlib, while ensuring that synthetic methods (for fields) also get assigned correctly
- `engine/runtime/src/main/scala/org/enso/compiler/codegen/IrToTruffle.scala` - when a builtin method is encountered in stdlib we don't generate a new function node for it, instead we look it up in the list of registered builtin methods. Note that Integer and Number present a bit of a challenge because they list a whole bunch of methods that don't have a corresponding method (instead delegating to small/big integer implementations).
During the translation new atom constructors get initialized but we don't want to do it for builtins which have gone through the process earlier, hence the exception
- `lib/scala/interpreter-dsl/src/main/java/org/enso/interpreter/dsl/MethodProcessor.java` - @Builtin_Method processor not only  generates the actual code fpr nodes but also collects and writes the info about them (name, class, params) to a metadata file that is read during builtins initialization 
- `lib/scala/interpreter-dsl/src/main/java/org/enso/interpreter/dsl/MethodProcessor.java` - @Builtin_Method processor no longer generates only (root) nodes but also collects and writes the info about them (name, class, params) to a metadata file that is read during builtins initialization
- `lib/scala/interpreter-dsl/src/main/java/org/enso/interpreter/dsl/TypeProcessor.java` - Similar to MethodProcessor but handles @Builtin_Type annotations. It doesn't, **yet**, generate any builtin objects.  It also collects the names, as present in stdlib, if any, so that we can generate the names automatically (see generated `types/ConstantsGen.java`)
- `engine/runtime/src/main/java/org/enso/interpreter/node/expression/builtin` - various classes annotated with @BuiltinType to ensure that the atom constructor is always properly registered for the builitn. Note that in order to support types fields in those, annotation takes optional `params` parameter (comma separated). 
- `engine/runtime/src/bench/scala/org/enso/interpreter/bench/fixtures/semantic/AtomFixtures.scala` - drop manual creation of test list which seemed to be a relict of the old design
2022-05-05 20:18:06 +02:00
Radosław Waśko
0ea5dc2a6f
Data analysts should be able to use Text.replace to substitute parts of the text (#3393)
Implements https://www.pivotaltracker.com/story/show/181266274
2022-04-13 19:21:47 +00:00
Radosław Waśko
43265f10a8
Implement Error-Handling for Database aggregations, unify some error helpers across backends (#3371) 2022-03-31 12:10:22 +00:00
Marcin Kostrzewa
334a022ffd
Import syntax including namespace (#1806) 2021-06-24 12:42:24 +02:00
Marcin Kostrzewa
b4709ab529
Default visualization definitions (#1786) 2021-06-08 08:12:02 +02:00
Ara Adkins
3890abe6fa
Update the protocol to support streaming files (#1757) 2021-05-26 15:08:41 +01:00
Ara Adkins
3080d8f6f7
Add .sum to Vector (#1702) 2021-04-28 10:47:57 +01:00
Ara Adkins
9585080ab8
Clean up the standard library docs (#1641) 2021-04-01 12:20:36 +01:00
Michał Wawrzyniec Urbańczyk
8d77a565eb
Case Insensitive Dataframe Support in Visualizations (#1634)
Ref https://github.com/enso-org/ide/issues/1391
2021-04-01 10:05:17 +02:00
Radosław Waśko
301672df24
Fix a Bug in the Database Join Implementation (#1614) 2021-03-26 00:34:16 +01:00
Michał Wawrzyniec Urbańczyk
5b57960da3
Histogram and Scatterplot visualizations support for Table (#1608) 2021-03-25 17:47:22 +01:00