Commit Graph

257 Commits

Author SHA1 Message Date
Radosław Waśko
7c94fa6a77
Custom Encoding support when writing Delimited files (#3564)
Implements https://www.pivotaltracker.com/story/show/182545847
2022-07-07 00:20:00 +00:00
James Dunkerley
5174cc6ece
Update Database.connect to match new API (#3542)
Initial work restructuring the `Database.connect` API
- New SQLite API with support for InMemory.
- Updated PostgreSQL API with SSL and Client Certificate Support.
- Updated Redshift API.

# Important Notes
Follow up tasks:
- PostgreSQL SSL additional testing.
- Driver version updating.
- `.pgpass` support.
2022-07-04 20:26:44 +00:00
James Dunkerley
4ca2097488
Adding write support to File_Format.Excel (#3551)
Support for writing tables to Excel.

# Important Notes
Has custom support for Error mode as will allow appending a new table in this mode to the file.
2022-07-04 18:32:16 +00:00
Jaroslav Tulach
096d8fdca0
Avoid crashing the engine when JS debugger statement is used (#3547)
Try following enso program:
```
main =
here.debug

foreign js debug = """
debugger;

```
it crashes the engine with exception:
```
Execution finished with an error: java.lang.ClassCastException: class com.oracle.truffle.js.runtime.builtins.JSFunctionObject$Unbound cannot be cast to class org.enso.interpreter.runtime.callable.CallerInfo (com.oracle.truffle.js.runtime.builtins.JSFunctionObject$Unbound and org.enso.interpreter.runtime.callable.CallerInfo are in unnamed module of loader com.oracle.graalvm.locator.GraalVMLocator$GuestLangToolsLoader @55cb6996)
at <java> org.enso.interpreter.runtime.callable.function.Function$ArgumentsHelper.getCallerInfo(Function.java:352)
at <java> org.enso.interpreter.instrument.ReplDebuggerInstrument$ReplExecutionEventNodeImpl.onEnter(ReplDebuggerInstrument.java:179)
at <java> org.graalvm.truffle/com.oracle.truffle.api.instrumentation.ProbeNode$EventProviderChainNode.innerOnEnter(ProbeNode.java:1397)
at <java> org.graalvm.truffle/com.oracle.truffle.api.instrumentation.ProbeNode$EventChainNode.onEnter(ProbeNode.java:912)
at <java> org.graalvm.truffle/com.oracle.truffle.api.instrumentation.ProbeNode.onEnter(ProbeNode.java:216)
at <java> com.oracle.truffle.js.nodes.JavaScriptNodeWrapper.execute(JavaScriptNodeWrapper.java:44)
at <java> com.oracle.truffle.js.nodes.control.DiscardResultNode.execute(DiscardResultNode.java:88)
at <java> com.oracle.truffle.js.nodes.function.FunctionBodyNode.execute(FunctionBodyNode.java:73)
at <java> com.oracle.truffle.js.nodes.JavaScriptNodeWrapper.execute(JavaScriptNodeWrapper.java:45)
at <java> com.oracle.truffle.js.nodes.function.FunctionRootNode.executeInRealm(FunctionRootNode.java:150)
at <java> com.oracle.truffle.js.runtime.JavaScriptRealmBoundaryRootNode.execute(JavaScriptRealmBoundaryRootNode.java:93)
at <js> <js> poly_enso_eval(Unknown)
at <epb> <epb> null(Unknown)
at <enso> Prg.debug(Prg.enso:27-28)
```
2022-06-25 05:58:24 +00:00
Hubert Plociniczak
8f9a0b33d5
Fix System.nanoTime definition (#3543)
While redesigning builtins a bug was accidentally introduced.
Stumbled upon accidentally while trying to run `test/Benchmarks`.
2022-06-24 10:46:31 +00:00
Radosław Waśko
972b34d1a9
Implement value formatting and writing new files in Delimited format. (#3528)
Implements https://www.pivotaltracker.com/story/show/182309429 and https://www.pivotaltracker.com/story/show/182309573
2022-06-23 16:51:52 +00:00
James Dunkerley
7a2d304fa0
Update Excel reading API (#3523)
- Remove `from_xls` and `from_xlsx`.
- Add `headers` support to `File_Format.Excel`.
- Altered default read for Excel to be the first sheet.
- Altered behavior so that single cells grow down and right when reading sheet.
- Altered `Excel_Range` so knows if single cell or 1x1 range address.

# Important Notes
- Renamed `Range` to `Cell_Range` to avoid name clash.
2022-06-21 13:39:32 +00:00
Hubert Plociniczak
22a371a9c6
Substitute this with self (#3524)
A semi-manual s/this/self appied to the whole standard library.
Related to https://www.pivotaltracker.com/story/show/182328601

In the compiler promoted to use constants instead of hardcoded
`this`/`self` whenever possible.

# Important Notes
The PR **does not** require explicit `self` parameter declaration for methods as this part
of the design is still under consideration.
2022-06-21 10:53:52 +00:00
Marcin Kostrzewa
ec3fa32fec
introduce a micro stdlib for testing (#3531)
This introduces a tiny alternative to our stdlib, that can be used for testing the interpreter. There are 2 main advantages of such a solution:
1. Performance: on my machine, `runtime-with-intstruments/test` drops from 146s to 65s, while `runtime/test` drops from 165s to 51s. >6 mins total becoming <2 mins total is awesome. This alone means I'll drink less coffee in these breaks and will be healthier.
2. Better separation of concepts – currently working on a feature that breaks _all_ enso code. The dependency of interpreter tests on the stdlib means I have no means of incremental testing – ALL of stdlib must compile. This is horrible, rendered my work impossible, and resulted in this PR.
2022-06-16 10:25:24 +00:00
James Dunkerley
a0c6fa9c96
Removing old functions and tidy up of Table types (#3519)
- Removed `select` method.
- Removed `group` method.
- Removed `Aggregate_Table` type.
- Removed `Order_Rule` type.
- Removed `sort` method from Table.
- Expanded comments on `order_by`.
- Update comment on `aggregate` on Database.
- Update Visualisation to use new APIs.
- Updated Data Science examples to use new APIs.
- Moved Examples test out of Tests to own test.

# Important Notes
Need to get Examples_Tests added to CI.
2022-06-14 13:37:20 +00:00
Radosław Waśko
e83c36d9d6
Add scaffolding for Table.write function (#3521)
Implements https://www.pivotaltracker.com/story/show/182309559

This task implements common scaffolding for the `Table.write`, so that the particular implementations for Delimited and Excel file formats can be done in parallel.
2022-06-14 11:29:03 +00:00
Hubert Plociniczak
fd46e84e8d
Towards a full-blown builtins DSL (part 3) (#3471)
Auto-generate all builtin methods for builtin `File` type from method signatures.
Similarly, for `ManagedResource` and `Warning`.
Additionally, support for specializations for overloaded and non-overloaded methods is added.
Coverage can be tracked by the number of hard-coded builtin classes that are now deleted.

## Important notes

Notice how `type File` now lacks `prim_file` field and we were able to get rid off all of those
propagating method calls without writing a single builtin node class.
Similarly `ManagedResource` and `Warning` are now builtins and `Prim_Warnings` stub is now gone.
2022-06-13 11:48:34 +00:00
Radosław Waśko
a04825a5ce
Add Text.write Function (#3518)
Implements https://www.pivotaltracker.com/story/show/182309026
2022-06-13 09:11:46 +00:00
James Dunkerley
e97d27e1e0
Adjusting First and Last order_by to use Sort_Column_Selector (#3517) 2022-06-10 09:59:03 +00:00
James Dunkerley
8afba43add
Implement In-Memory Table order_by (#3515)
Implemented the `order_by` function with support for all modes of operation.
Added support for case insensitive natural order.

# Important Notes
- Improved MultiValueIndex/Key to not create loads of arrays.
- Adjusted HashCode for MultiValueKey to have a simple algorithm.
- Added Text_Utils.compare_normalized_ignoring_case to allow for case insensitive comparisons.
- Fixed issues with ObjectComparator and added some unit tests for it.
2022-06-08 12:30:50 +00:00
Radosław Waśko
2af970fe52
Basic changes to File_Format (#3516)
Implements https://www.pivotaltracker.com/story/show/182308987
2022-06-08 09:53:18 +00:00
Radosław Waśko
a382e0c15e
Improve database Table.order_by (#3514)
Implements https://www.pivotaltracker.com/story/show/182195405

Adds support for the Postgres dialect and simple case insensitive collation for SQLite.
2022-06-07 12:31:55 +00:00
Radosław Waśko
7d94efa6f2
Implement Table.order_by for SQLite and the common scaffolding for all backends (#3502)
Implements the common and SQLite parts of https://www.pivotaltracker.com/story/show/182195405
2022-06-06 10:56:52 +00:00
Hubert Plociniczak
e43325bfe1
Short-circuiting || and && (#3492)
Short-circuiting || and && is typically taken for granted
by users of other PLs. This change makes it happen for Enso.

Related to https://www.pivotaltracker.com/story/show/182261401
2022-06-02 16:58:38 +00:00
James Dunkerley
ba5d6823a9
Merge the Unique Name Strategy with NameDeduplicator (#3490)
- Merge the two approaches and makes them consistent
- Add warning support into Reader

# Important Notes
- Added support for JUnit format XML generation on tests. Use `ENSO_TEST_JUNIT_DIR`
2022-06-01 12:52:23 +00:00
Michał Wawrzyniec Urbańczyk
9842b0e5f0
Allow trailing space in ENSO_HTTP_TEST_HTTPBIN_URL URL. (#3500) 2022-06-01 10:32:03 +03:00
James Dunkerley
1aa0bb3552
Rank Data, Correlation, Covariance, R Squared (#3484)
- Added new `Statistic`s: Covariance, Pearson, Spearman, R Squared
- Added `covariance_matrix` function
- Added `pearson_correlation` function to compute correlation matrix
- Added `rank_data` and Rank_Method type to create rankings of a Vector
- Added `spearman_correlation` function to compute Spearman Rank correlation matrix

# Important Notes
- Added `Panic.throw_wrapped_if_error` and `Panic.handle_wrapped_dataflow_error` to help with errors within a loop.
- Removed `Array.set_at` use from `Table.Vector_Builder`
2022-05-30 17:13:06 +00:00
Radosław Waśko
f0f3a343eb
Adjust Table.sort_columns to use Text_Ordering design (#3487)
Implements https://www.pivotaltracker.com/story/show/182195306
2022-05-30 12:26:29 +00:00
Radosław Waśko
db611e1581
Remove obsolete Csv reading module (#3482)
Completes https://www.pivotaltracker.com/story/show/182037405

# Important Notes
- Some tests had to be adapted to the new parsing logic.
2022-05-28 10:01:14 +00:00
Radosław Waśko
8828d801ea
Implement Table from Text conversion (#3478)
Implements https://www.pivotaltracker.com/story/show/181824168
2022-05-26 12:04:25 +00:00
Radosław Waśko
7f572bf3e4
The user should be able to have the headers Inferred when reading a Delimited file (#3472)
Implements https://www.pivotaltracker.com/story/show/181986831
2022-05-25 13:29:17 +00:00
Radosław Waśko
ec1b072824
Integrate value parsing with Delimited file reading (#3463)
Implements https://www.pivotaltracker.com/story/show/182200028
2022-05-24 17:59:00 +02:00
Radosław Waśko
ff7700ebb1
Automatic inference of value types when parsing table columns (#3462)
Implements https://www.pivotaltracker.com/story/show/182199966
2022-05-20 15:08:36 +00:00
Radosław Waśko
0073f461d9
Fix Dataflow Error propagation for Builtins accepting primitives (#3400)
[ci no changelog needed]

Fixes https://www.pivotaltracker.com/story/show/181652841
2022-05-19 15:25:30 +00:00
Radosław Waśko
8430ce2625
Parsing values with known types (#3455)
Implements https://www.pivotaltracker.com/story/show/181824146
2022-05-18 15:27:48 +00:00
Jaroslav Tulach
78e7d69198
Generator of natural numbers yields IllegalStateException (#3440) 2022-05-18 11:48:29 +00:00
Hubert Plociniczak
6b6b1430bc
Cleanup Ref - get/put (#3457)
The change promotes static methods of `Ref`, `get` and `put`, to be
methods of `Ref` type.
The change also removes `Ref` module from the default namespace.
Had to mostly c&p functional dispatch for now, in order for the methods
to be found. Will auto-generate that code as part of builtins system.

Related to https://www.pivotaltracker.com/story/show/182138899
2022-05-17 10:26:36 +00:00
James Dunkerley
4f3a76817c
Statistics on a Vector (#3442)
- Implements various statistics on Vector

# Important Notes
Some minor codebase improvements:
- Some tweaks to Any/Nothing to improve performance
- Fixed bug in ObjectComparator
- Added if_nothing
- Removed Group_By_Key
2022-05-11 13:25:06 +00:00
Radosław Waśko
64f178f7a8
Delimited File Encoding (#3430)
Implements https://www.pivotaltracker.com/story/show/181998375
2022-05-10 22:44:05 +00:00
James Dunkerley
078c665a60
File_Format.Excel work (#3425)
- Read in Excel files following the specification.
- Support for XLSX and XLS formats.
- Ability to select ranges and sheets.
- Skip Rows and Row Limits.

# Important Notes
- Minor fix to DelimitedReader for Windows
2022-05-06 13:21:10 +00:00
Hubert Plociniczak
4bbabc00be
Move Builtin Types and Methods to stdlib (#3363)
This PR replaces hard-coded `@Builtin_Method` and `@Builtin_Type` nodes in Builtins with an automated solution
that a) collects metadata from such annotations b) generates `BuiltinTypes` c) registers builtin methods with corresponding
constructors.
The main differences are:
1) The owner of the builtin method does not necessarily have to be a builtin type
2) You can now mix regular methods and builtin ones in stdlib 
3) No need to keep track of builtin methods and types in various places and register them by hand (a source of many typos or omissions as it found during the process of this PR)

Related to #181497846
Benchmarks also execute within the margin of error.

### Important Notes

The PR got a bit large over time as I was moving various builtin types and finding various corner cases.
Most of the changes however are rather simple c&p from Builtins.enso to the corresponding stdlib module.
Here is the list of the most crucial updates:
- `engine/runtime/src/main/java/org/enso/interpreter/runtime/builtin/Builtins.java` - the core of the changes. We no longer register individual builtin constructors and their methods by hand. Instead, the information about those is read from 2 metadata files generated by annotation processors. When the builtin method is encountered in stdlib, we do not ignore the method. Instead we lookup it up in the list of registered functions (see `getBuiltinFunction` and `IrToTruffle`)
- `engine/runtime/src/main/java/org/enso/interpreter/runtime/callable/atom/AtomConstructor.java` has now information whether it corresponds to the builtin type or not.
- `engine/runtime/src/main/scala/org/enso/compiler/codegen/RuntimeStubsGenerator.scala` - when runtime stubs generator encounters a builtin type, based on the @Builtin_Type annotation, it looks up an existing constructor for it and registers it in the provided scope, rather than creating a new one. The scope of the constructor is also changed to the one coming from stdlib, while ensuring that synthetic methods (for fields) also get assigned correctly
- `engine/runtime/src/main/scala/org/enso/compiler/codegen/IrToTruffle.scala` - when a builtin method is encountered in stdlib we don't generate a new function node for it, instead we look it up in the list of registered builtin methods. Note that Integer and Number present a bit of a challenge because they list a whole bunch of methods that don't have a corresponding method (instead delegating to small/big integer implementations).
During the translation new atom constructors get initialized but we don't want to do it for builtins which have gone through the process earlier, hence the exception
- `lib/scala/interpreter-dsl/src/main/java/org/enso/interpreter/dsl/MethodProcessor.java` - @Builtin_Method processor not only  generates the actual code fpr nodes but also collects and writes the info about them (name, class, params) to a metadata file that is read during builtins initialization 
- `lib/scala/interpreter-dsl/src/main/java/org/enso/interpreter/dsl/MethodProcessor.java` - @Builtin_Method processor no longer generates only (root) nodes but also collects and writes the info about them (name, class, params) to a metadata file that is read during builtins initialization
- `lib/scala/interpreter-dsl/src/main/java/org/enso/interpreter/dsl/TypeProcessor.java` - Similar to MethodProcessor but handles @Builtin_Type annotations. It doesn't, **yet**, generate any builtin objects.  It also collects the names, as present in stdlib, if any, so that we can generate the names automatically (see generated `types/ConstantsGen.java`)
- `engine/runtime/src/main/java/org/enso/interpreter/node/expression/builtin` - various classes annotated with @BuiltinType to ensure that the atom constructor is always properly registered for the builitn. Note that in order to support types fields in those, annotation takes optional `params` parameter (comma separated). 
- `engine/runtime/src/bench/scala/org/enso/interpreter/bench/fixtures/semantic/AtomFixtures.scala` - drop manual creation of test list which seemed to be a relict of the old design
2022-05-05 20:18:06 +02:00
Radosław Waśko
8219dca400
Improve support for reading Delimited files (#3424)
Implements https://www.pivotaltracker.com/story/show/181823957
2022-04-29 17:12:19 +00:00
Radosław Waśko
14257d07aa
Data analysts should be able to use Text.split, Text.lines and Text.words to break up strings (#3415)
Implements https://www.pivotaltracker.com/story/show/181266184

### Important Notes

Changed example image download to only proceed if the file did not exist before - thus cutting on the build time (the build used to download it _every_ time - which completely failed the build if network is down). A redownload can be forced by performing a fresh repository checkout.
2022-04-26 17:22:53 +02:00
Radosław Waśko
fecaa81551
Review Range and Interval, resolve infinite loop issue (#3408)
Implements: https://www.pivotaltracker.com/story/show/181652841
2022-04-20 16:22:01 +00:00
James Dunkerley
5a6b6749cc
Restructuring for File.read (#3390)
- Added Encoding type
- Added `Text.bytes`, `Text.from_bytes` with Encoding support
- Renamed `File.read` to `File.read_text`
- Renamed `File.write` to `File.write_text`
- Added Encoding support to `File.read_text` and `File.write_text`
- Added warnings to invalid encodings
2022-04-19 16:50:03 +00:00
Jaroslav Tulach
ab692b3b74
Enso Date shall be converted to java.time.LocalDate when passed to Java (#3374) 2022-04-15 06:02:05 +02:00
Radosław Waśko
0ea5dc2a6f
Data analysts should be able to use Text.replace to substitute parts of the text (#3393)
Implements https://www.pivotaltracker.com/story/show/181266274
2022-04-13 19:21:47 +00:00
Radosław Waśko
891f064a6a
Extend Aggregate_Spec test suite with tests for missed edge-cases to ensure the feature is well-tested on all backends (#3383)
Implements https://www.pivotaltracker.com/story/show/181805693 and finishes the basic set of features of the Aggregate component.

Still not all aggregations are supported everywhere, because for example SQLite has quite limited support for aggregations. Currently the workaround is to bring the table into memory (if possible) and perform the computation locally. Later on, we may add more complex generator features to emulate the missing aggregations with complex sub-queries.
2022-04-12 11:02:01 +00:00
Marcin Kostrzewa
4e51f31eb7
Always call defaulted atom arguments (#3358)
Solves the issue of defaulted args not being called in atoms. Doesn't solve the more general function issue.
2022-04-08 08:21:59 +00:00
James Dunkerley
bade0c31de
First and Last ordering (#3380)
Add the missing `order_by` support to First and Last aggregations for InMemory table.
2022-04-06 12:36:46 +00:00
Radosław Waśko
a71db71645
Adding most of remaining aggregates to Database Table (#3375) 2022-04-06 10:06:50 +00:00
Nikita Pekin
42ac28d0de
Add benchmark for Text.reverse with strings of varying length (#3381)
This pull request adds a benchmark for the `Text.reverse` function added in #3377 as part of https://www.pivotaltracker.com/n/projects/2539304/stories/181265419.

Per discussion with @jdunkerley on Discord it is useful to have this benchmark as this is a low-level item we want to track.
2022-04-05 17:59:21 +00:00
Nikita Pekin
22e3941371
Data analysts should be able to reverse strings using Text.reverse (#3377)
This commit implements `Text.reverse` as an extension on `Text`.
`Text.reverse` reverses strings. For example: `"Hello World!".reverse`
results in `"!dlroW olleH"`.

Strings are reversed by their Extended Grapheme Clusters not by their
characters. This has some performance implications because we need to
find these grapheme cluster boundaries when iterating. To do so,
`BreakIterator.getCharacterInstance` is used.

Implements: https://www.pivotaltracker.com/n/projects/2539304/stories/181265419
2022-04-05 16:45:56 +00:00
James Dunkerley
a4dbc9a37b
Moving Aggregation to Java (#3364) 2022-04-04 09:12:48 +00:00
Radosław Waśko
43265f10a8
Implement Error-Handling for Database aggregations, unify some error helpers across backends (#3371) 2022-03-31 12:10:22 +00:00
Radosław Waśko
20be5516a5
Aggregates in the Database library - MVP (#3353)
Implements infrastructure for new aggregations in the Database. It comes with only some basic aggregations and limited error-handling. More aggregations and problem handling will be added in subsequent PRs.

# Important Notes
This introduces basic aggregations using our existing codegen and sets-up our testing infrastructure to be able to use the same aggregate tests as in-memory backend for the database backends.

Many aggregations are not yet implemented - they will be added in subsequent tasks.

There are some TODOs left - they will be addressed in the next tasks.
2022-03-28 15:51:37 +00:00
Radosław Waśko
85a5770b7f
Quick-fix for Error.to_text CCE (#3357)
This is just a quick fix addressing an issue which was making debugging problematic.

The proper solution to the broader issue described at https://github.com/enso-org/enso/issues/1538#issuecomment-789645573 still needs to be done.
2022-03-24 13:12:53 +00:00
Radosław Waśko
85c09e7414
Make Resource.bracket not run the action if initializer failed with a dataflow error (#3356) 2022-03-23 16:36:35 +01:00
James Dunkerley
02bcfbb2a8
Refactor Aggregate Column (#3349)
- Make it easier to understand the computations.
- Fix issue with First.
- Improve quote handling in Concatenate
- Added validation and warnings to input
2022-03-22 18:18:46 +00:00
Hubert Plociniczak
66e2135b0d
Initialize AtomConstructor's fields via local vars (#3330)
The mechanism follows a similar approach to what is being in functions
with default arguments.
Additionally since InstantiateAtomNode wasn't a subtype of EnsoRootNode it
couldn't be used in the application, which was the primary reason for
issue #181449213.
Alternatively InstantiateAtomNode could have been enhanced to extend
EnsoRootNode rather than RootNode to carry scope info but the former
seemed simpler.

See test cases for previously crashing and invalid cases.
2022-03-21 09:15:14 +00:00
Radosław Waśko
cc7333812d
The library developer should be able to handle specific types of Panics while passing through others (#3344)
Implements https://www.pivotaltracker.com/story/show/181569176

Also ensures that Dataflow Errors have proper stack traces (earlier they did not point at the right location).
2022-03-18 16:57:06 +00:00
Radosław Waśko
08183f59f2
Minor fixes for Text (#3340)
* Avoid unnecessary copies

* Add tests for conversions

* Add guidelines for Text tests

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2022-03-15 16:11:46 +00:00
James Dunkerley
6c1c4554f5
Refactor table.group_by to table.aggregate (#3339)
Following UX work move to `table.aggregate` function.
2022-03-15 15:23:36 +01:00
Radosław Waśko
dedd1eac96
Refactor library warnings to use the new system (#3337)
Implements https://www.pivotaltracker.com/story/show/181536964
2022-03-15 12:52:57 +01:00
Radosław Waśko
247b284316
Data analysts should be able to use Text.location_of to find indexes within string using various matchers (#3324)
Implements https://www.pivotaltracker.com/n/projects/2539304/stories/181266029
2022-03-12 19:42:00 +00:00
Marcin Kostrzewa
4653bfeeab
Decorate values with arbitrary warnings (#3248) 2022-03-09 16:40:02 +01:00
James Dunkerley
65465fb8ef
Restructuring the Faker type and creating tests for Group_By (#3318)
- Added Minimum, Maximum, Longest. Shortest, Mode, Percentile
- Added first and last to Map
- Restructured Faker type more inline with FakerJS
- Created 2,500 row data set
- Tests for group_by
- Performance tests for group_by
2022-03-09 10:31:02 +00:00
Hubert Plociniczak
f92108158c
Added compare_to to True/False (#3317) 2022-03-08 14:24:04 +01:00
Hubert Plociniczak
8bdca89917
New Text.insert function (#3311)
Implements https://www.pivotaltracker.com/n/projects/2539304
2022-03-04 16:40:34 +01:00
James Dunkerley
fb68f18739
Within Vector, use Array.Copy wherever possible (#3236)
Following the Slice and Array.Copy experiment, took just the Array.Copy parts out and built into the Vector class.

This gives big performance wins in common operations:

| Test | Ref | New |
| --- | --- | --- |
| New Vector | 41.5 | 41.4 |
| Append Single | 26.6 | 4.2 |
| Append Large | 26.6 | 4.2 |
| Sum | 230.1 | 99.1 |
| Drop First 20 and Sum | 343.5 | 96.9 |
| Drop Last 20 and Sum | 311.7 | 96.9 |
| Filter | 240.2 | 92.5 |
| Filter With Index | 364.9 | 237.2 |
| Partition | 772.6 | 280.4 |
| Partition With Index | 912.3 | 427.9 |
| Each | 110.2 | 113.3 |

*Benchmarks run on an AWS EC2 r5a.xlarge with 1,000,000 item count, 100  iteration size run 10 times.*

# Important Notes
Have generally tried to push the `@Tail_Call` down from the Vector class and move to calling functions on the range class.

- Expanded benchmarks on Vector
- Added `take` method to Vector
- Added `each_with_index` method to Vector
- Added `filter_with_index` method to Vector
2022-03-03 15:40:48 +00:00
Radosław Waśko
500aed9d86
Fix the Test library ignoring dataflow errors (#3312)
Fixes https://www.pivotaltracker.com/story/show/181369176
2022-03-03 11:02:13 +01:00
James Dunkerley
ad1130587d
Updating Text.repeat and adding Text.* (#3310)
Updating the `Text.repeat` function:
- fix issue with negative count
- add * operator

Add tests of the function.
2022-03-02 19:00:47 +00:00
Radosław Waśko
40c851bf8b
Text.pad and Text.trim (#3309)
Implements https://www.pivotaltracker.com/story/show/181265516
2022-03-02 17:19:39 +00:00
James Dunkerley
738a691662
Table.group_by (#3305)
Functioning group_by based of Enso Map.

# Important Notes
This is an initial version which will be used to establish the API.
The grouping map will need to be moved to Java code for performance.
2022-03-01 16:18:11 +00:00
Radosław Waśko
0d96f59f44
Data analysts should be able to use Text.to_case to change the case of Text values (#3302)
* Move to_upper_case and to_lower_case into to_case

* Add an export, not sure about it

* Implement title case

TODO: some more tests would be good

* Add more tests

* explain title case

* fix todo

* changelog
2022-02-28 23:20:41 +00:00
Radosław Waśko
b03416f907
Update Column_Selector and Column_Mapping to use Matcher over Matching_Strategy (#3299)
Implements https://www.pivotaltracker.com/story/show/181339748
2022-02-25 18:39:10 +00:00
Radosław Waśko
2ae636f63c
Data analysts should be able to use Text.starts_with and Text.ends_with (#3292)
Implements https://www.pivotaltracker.com/story/show/181265900
2022-02-23 16:48:33 +00:00
James Dunkerley
2e2c5562a8
Text.take and Text.drop (#3287)
Implementation of the Text take and drop APIs
- Added `Range.contains` function
- Added `Text_Sub_Range` type
- Added `Text_Utils.index_of` and `Text_Utils.last_index_of` based on ICU StringSearcher
2022-02-22 18:50:59 +00:00
Radosław Waśko
ae9d51555f
Data analysts should be able to use Text.contains to check for substring using various matcher techniques. (#3285)
* Add matching mode definitions

* Add stub for new method API and an initial test suite

* Fix tests, implement exact matching

* Implement Regex matching

* changelog

* Add benchmarks

* Wokraround for case insensitive regex locale support

* minor tweaks

* Unify Case_Insensitive

* Update edge cases

* Fix other affected places

* minor style change

* Add a problematic test

* Add a regex test for a similar situation

* Migrate to StringSearch:wq

* Add test cases for scharfes S edge case

* Add problematic Regex Unicode normalization test

* Document the regex accents peculiarity

* Do not apply the normalization in ASCII only mode

* cr
2022-02-22 15:41:56 +00:00
Radosław Waśko
14f57271a2
Ensure that Text.compare_to compares strings according to grapheme clusters (#3282)
https://www.pivotaltracker.com/story/show/181175238
2022-02-17 17:09:41 +00:00
James Dunkerley
7afc8c48c5
Adding Integer.Parse (#3283)
* Integer parse via Longs

* Integer parse via Longs

* Benchmark for Number Parse

* CHANGELOG.md and Natural Order

* Expanded test set

* Number base tests

* Few more negative tests
2022-02-17 15:04:00 +00:00
James Dunkerley
68b85dea82
Improvement to the Natural Order Sort (#3276)
* Improved Natural Order
Data generator for benchmarking

* Missing Import
Benchmark script

* Update Natural_Order.enso

Restore missing ToDo

* Changelog

* PR Comments

* PR Comments

* Additional comments.

* Correction
2022-02-16 17:40:33 +00:00
Marcin Kostrzewa
67b4e59506
Properly expose stacktraces and related data to user code (#3271) 2022-02-16 10:36:19 +03:00
Radosław Waśko
fbf747d6cf
Implement Vector.flatten (#3259) 2022-02-15 16:16:08 +01:00
James Dunkerley
585afd83ce
Adding Text.at and Text.is_digit functions (#3269)
* Add Text.at function

* Add tests for Text.at

* Add tests for Text.is_digit

* Change log

* Avoid memory allocation
2022-02-14 09:03:55 +00:00
Edward Kmett
0c25ee736c
Upgrade Truffle and Graal to Version 21.3.0 (#3258) 2022-02-11 19:05:13 +03:00
James Dunkerley
1814d3c4f1
Data analysts should be able to transform a Table using the rename_columns functions (#3249)
* Implement Natural_Order and sort_columns

* Starting on Rename

Align Column_Mapping

Add By_Position
Separating off the validation for By_Index so can reuse for rename

By_Position implemented

By_Index implemented
Adjusted behaviour following discussion with Ned, so that renames dominate untouched columns.

Moving to validation style checks for problems

Putting accumulator back

Rename work

* Add Range.find

* More work

* Regex support
Tidy of Unique Name Strategy

* Fix Regex support

* Warning messages
Tests for Unique Naming Strategy
Table rename working

* Database Table rename_columns
Fix for Table
**Must follow up on slice**

* Some tests

* More tests

* Complete test set
(and associated fixes)

* Functional use_first_row_as_names
Tests to go...

* Test for use_first_row_as_names

* Change log

* trailing space

Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
2022-02-11 10:18:51 +00:00
Marcin Kostrzewa
ee8df25fd5
Fix vector sorting with TCO comparators (#3256) 2022-02-09 22:17:43 +01:00
Radosław Waśko
8b24336604
Data analysts should be able to reorder columns into name order using sort_columns functions (#3250) 2022-02-08 17:28:46 +01:00
Edward Kmett
8a70debb59
Implement conversions (#180312665) (#3227)
* Implement conversions

start wip branch for conversion methods for collaborating with marcin

add conversions to MethodDispatchLibrary (wip)

start MethodDispatchLibrary implementations

conversions for atoms and functions

Implement a bunch of missing conversion lookups

final bug fixes for merged methoddispatchlibrary implementations

UnresolvedConversion.resolveFor

progress on invokeConversion

start extracting constructors (still not working)

fix a bug

add some initial conversion tests

fix a bug in qualified name resolution, test conversions accross modules

implement error reporting, discover a ton of ignored errors...

start fixing errors that we exposed in the standard library

fix remaining standard lib type errors not caused by the inability to parse type signatures for operators

TODO: fix type signatures for operators. all of them are broken

fix type signature parsing for operators

test cases for meta & polyglot

play nice with polyglot

start pretending unresolved conversions are unresolved symbols

treat UnresolvedConversons as UnresolvedSymbols in enso user land

* update RELEASES.md

* disable test error about from conversions being tail calls. (pivotal issue #181113110)

* add changelog entry

* fix OverloadsResolutionTest

* fix MethodDefinitionsTest

* fix DataflowAnalysisTest

* the field name for a from conversion must be 'that'. Fix remaining tests that aren't ExpressionUpdates vs. ExecutionUpdate behavioral changes

* fix ModuleThisToHereTest

* feat: suppress compilation errors from Builtins

* Revert "feat: suppress compilation errors from Builtins"

This reverts commit 63d069bd4f.

* fix tests

* fix: formatting

Co-authored-by: Dmitry Bushev <bushevdv@gmail.com>
Co-authored-by: Marcin Kostrzewa <marckostrzewa@gmail.com>
2022-02-06 04:02:09 -05:00
Radosław Waśko
d3c0f968fa
Data analysts should be able to transform a Table using the remove_columns and reorder_columns functions (#3240) 2022-02-03 15:18:47 +01:00
Radosław Waśko
b5fc87e618
Data analysts should be able to transform a Table using the select_columns function (#3230)
* Utility for mapping errors and warnings
* Imlpement By_Index
* Expose select_columns in InMem and DB. Need testing
* checkpoint: writing tests
* Fix minor issues, mock warning mapping for testing purposes
* Improve By_Index error handling
* A helper for testing problem handling
* More error handling
* docs
* changelog
* Fix matching test
* Add SQLite tests
* cleanup after test
* Rework problem handling
* small refactor
* add examples
* Add more test cases for regex matching
* Fix Regex.Patter.matches to match full string
* "Fix" tests
2022-02-02 09:04:06 +00:00
Radosław Waśko
cfdb33bc68
Improve Vector (#3232) 2022-01-25 18:29:39 +01:00
James Dunkerley
8387375d83
Moving distinct to Map (#3229)
* Moving distinct to Map

* Mixed Type Comparable Wrapper

* Missing Bracket
Still an issue with `Integer` in the mixed vector test

* PR comments

* Use naive approach for mixed types

* Enable pending test

* Performance timing function

* Handle incomparable types cleanly

* Tidy up the time_execution function

* PR comments.

* Change log
2022-01-25 09:57:30 +00:00
Radosław Waśko
107128aeec
A library developer should be able to select matching names given a list (#3220) 2022-01-20 11:11:43 +01:00
Michał Wawrzyniec Urbańczyk
ed0e918bff
Fix the new engine CI workflow (#180855729) (#3219)
Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
2022-01-17 19:21:34 +01:00
Radosław Waśko
66082ea554
The user should be able to remove duplicate elements from a Vector (#3224) 2022-01-17 12:51:56 +03:00
Dmitry Bushev
c14a2d8169
Fix codec spec (#3185) 2021-12-09 15:01:47 +03:00
Dmitry Bushev
93f7362199
Set Locale in Tests (#3158) 2021-11-16 17:18:25 +03:00
Ara Adkins
337f6c8ad4
Implement linear regression on tables (#2003) 2021-09-29 15:33:18 +01:00
Ara Adkins
d6465e9e97
Implement a --compile command for the engine runner (#1998) 2021-09-24 12:24:44 +01:00
Ara Adkins
1cd2706ba8
Load IR Caches from Disk (#1996) 2021-09-18 13:48:13 +01:00
Ara Adkins
ab8b2a2d4a
Implement writing of IR caches (#1991) 2021-09-08 17:15:42 +01:00
Marcin Kostrzewa
4f4e472ddf
Statistical functions (#1990) 2021-09-06 14:48:09 +02:00
Marcin Kostrzewa
a81257b402
Google Spreadsheet Reading (#1976) 2021-09-03 21:41:12 +02:00