Commit Graph

925 Commits

Author SHA1 Message Date
AdRiley
a9ec0e36a2
Enable Union feature and make tests pass (#11587) 2024-11-20 19:48:00 +00:00
AdRiley
7327df88c3
Complete migration to Dialect/Feature flags + SQLServer support for Datetime (#11527)
* Supports_Sort_Digits_As_Numbers

* case_insensitive_ordering

* Case_Insensitive_Ordering

* Case_Insensitive_Ordering

* Enable tests

* Case_Insensitive_Non_Ascii

* Order_By_Unicode_Normalization_By_Default

* Allows_Mixed_Type_Comparisons

* Supports_Unicode_Normalization

* NaN_Non_Comparable

* Distinct_Returns_First_Row_From_Group_If_Ordered

* Datetime support

* refactor

* refactor

* Refactor

* Move remaining flags

* Cleanup

* sbt javafmtAll

* Fix

* Fix tests

* Code review changes

* More date_diff month tests

* Fix sowflake

* Fix snowflake tests

Bypassing failing test.
2024-11-20 10:41:49 +00:00
James Dunkerley
d5f0e9ed8c
Fixes a few bits... (#11581)
- Minor fixes from past PR.
- Always use Table viz for Table.
- Fix display to `Always`  for any AWS required parameters.
- Add widget for `Redshift` credentials and alter `Username_And_Password` to require arguments.
- Fix display in `Data` module methods for required parameters.
- Fix `Statistic.bulk_widget`.
- Alter `Google_Analytics` so credentials the first parameter.
- Add support for storing the credential file in Enso cloud in `Google_Analytics`.

![image](https://github.com/user-attachments/assets/f34c4231-9f9f-468b-90e9-bbc7f2374d22)
2024-11-18 22:29:34 +00:00
James Dunkerley
0543cfaec5
New Table.input function (#11562)
* New Table.input function for GUI allowing auto parsing and type control

* CHANGELOG.

* Missing import.

* Char not Text.

* Fix imports.
2024-11-15 10:50:21 +00:00
Gregory Michael Travis
159a7a32c8
Add Warnings whenever a floating-point value is used as a Dictionary key (#11417) 2024-11-14 19:57:48 +00:00
James Dunkerley
d0f3d7f098
skip_rows for Excel_Workbook.read. (#11560)
- Adds `skip_rows` to `Excel_Workbook.read`.
- Support for read by index.
2024-11-14 19:31:47 +00:00
James Dunkerley
7363377331
Linting and Widgets (#11551)
- Fix any issues identified by the doc writer.
- Alter all `default_widget` functions: all now take display and all use the type check signature.
- Review widgets on AWS APIs and make sure display is correct.
2024-11-14 12:01:04 +00:00
James Dunkerley
6b544650b3
New NumberParser for Table parsing (#11499)
Replaces the Regex based number parser with a new parser which works out the same by working out each part as it sees and example of it.

Close #7398 - performance of reading the large CSV now about 2s (down from 15-20s).
2024-11-13 19:08:23 +00:00
Gregory Michael Travis
fb50a8f24f
HTTP cache size limit environment variables (#11530) 2024-11-13 13:40:54 -05:00
Hubert Plociniczak
2f2eeafceb
Fix Logger's name in stdlib (#11519)
* Fix Logger's name in stdlib

Somehow SLF4J is able to recognize correctly the provided Logger's name
and print it to the user. Java's Logger is
not.
In addition, we setup SLF4J's configuration, meaning that log-levels are
correctly respected.

For a simple project:
```
from Standard.Base import all
from Standard.Base.Logging import all

type Foo

main =
    IO.println "Hello World!"
    Foo.log_message level=..Warning "I should warn you about something..."
    Foo.log_message level=..Info "Should be seen? By default we only show up-to warnings level"
    Foo.log_message level=..Severe "Something went really bad!"
```

This change demonstrates the fix.

Before:
```
> enso --run simple-logging.enso
Hello World!
Nov 08, 2024 6:08:07 PM com.oracle.truffle.host.HostMethodDesc$SingleMethod$MHBase invokeHandle
WARNING: I should warn you about something...
Nov 08, 2024 6:08:07 PM com.oracle.truffle.host.HostMethodDesc$SingleMethod$MHBase invokeHandle
INFO: Should be seen? By default we only show up-to warnings level
Nov 08, 2024 6:08:07 PM com.oracle.truffle.host.HostMethodDesc$SingleMethod$MHBase invokeHandle
SEVERE: Something went really bad!
Foo
```

After:
```
> enso --run simple-logging.enso
Hello World!
[WARN] [2024-11-08T18:03:37+01:00] [simple-logging.Foo] I should warn you about something...
[ERROR] [2024-11-08T18:03:37+01:00] [simple-logging.Foo] Something went really bad!
Foo
```

* Update distribution/lib/Standard/Base/0.0.0-dev/src/Logging.enso

Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>

* Test stdlib logs by using MemoryAppender

Added `getEvents` member to `MemoryAppender` so that it is possible to
retrieve individual log messages from tests and test their presence.
Required opening up to some modules to retrieve internals of loggers.

* nit

* small tweaks to eliminate module warnings

---------

Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
2024-11-13 10:20:41 +01:00
Gregory Michael Travis
c23ff074e4
Make round_integer work for positive decimal places (#11457) 2024-11-12 23:29:25 +00:00
Gregory Michael Travis
978a009133
Create Regex benchmarks (#11504) 2024-11-12 22:43:28 +00:00
Radosław Waśko
e76fe907d3
Initial implementation of Data.read_many (#11490)
- Part of #11311
- Adds ability to read a list of files (Vector, Column, Table) into a Vector.
- Reading into a Table of objects or merged will come in a next PR.
2024-11-08 19:03:47 +00:00
marthasharkey
f2037ee9d0
Add data quality indicators to table Viz (#11307) 2024-11-08 16:59:09 +00:00
AdRiley
29faf201fc
Make SQLServer join work (#11511)
* Make join work

* Code Review Changes
2024-11-08 15:48:55 +00:00
AdRiley
014a2a7e6b
Refactor (#11506) 2024-11-07 16:02:54 +00:00
AdRiley
676a7d4256
SQLServer enable filter (#11471)
* checkpoint

* Passing by.integer.comparisons

* More passing tests

* by empty text

* Add flag

* 2 more green

* Back to 6 red

* 5 red

* 2 red

* 1 red

* Green

* Refactor

* refactor

* Refactor

* refactor

* Refactor

* clean up

* Green

* Refactor

* Cleanup

* Refactor

* Clean up

* Cleanup

* Clean up

* Fix tests

* Fix

* Fix

* Code review

* Refactor

* Code review changes

* Add literals

* Code review changes

* Code review changes

* Checkpoint

* checkpoint

* checkpoint

* Cleanup

* Refactor

* Refactor

* refactor

* Fix type name

* Fix
2024-11-07 15:42:55 +00:00
Pavel Marek
701bba6504
Convert Array_Like_Helpers.map to a builtin to reduce stack size (#11363)
The ultimate goal is to reduce the method calls necessary for `Vector.map`.

# Important Notes
- I managed to reduce the number of Java stack frames needed for each `Vector.map` call from **150** to **22** (See https://github.com/enso-org/enso/pull/11363#issuecomment-2432996902)
- Introduced `Stack_Size_Spec` regression test that will ensure that Java stack frames needed for `Vector.map` method call does not exceed **40**.
2024-11-06 11:14:48 +00:00
James Dunkerley
c5734a8fc8
Improved Google Analytics integration (#11484)
- Enhanced Google Analytics API.
- Now published as a type with static methods not a module.
- Bump version and add Admin API.
- Moved the reading logic to Java from Enso.
- Add dependency on Standard Table allowing report to be built into a Java Table directly.
- New `Google_Credential.new` method.
![image](https://github.com/user-attachments/assets/54e3ad87-045f-4e40-b609-337d827c5d02)
- Ability to list accounts for a credential (`Google_Analytics.list_accounts`).
![image](https://github.com/user-attachments/assets/296c6dcc-3b24-43fa-b909-5e74c40d77a1)
- Ability to list properties (either for an account or for all) (`Google_Analytics.list_properties`).
![image](https://github.com/user-attachments/assets/e420c824-d08e-48d0-b21c-560b4c7c4809)
- Simple object structure of `Google_Analytics_Account`, `Google_Analytics_Property` and `Google_Analytics_Field` with some helper methods.
- Widget for `account`, `property` and `credentials`.
![image](https://github.com/user-attachments/assets/221c1450-964d-4fce-af8b-2273aa8739a1)
![image](https://github.com/user-attachments/assets/e1daf1dd-2ade-4c33-875c-4e3cb1544fe6)
![image](https://github.com/user-attachments/assets/cd37b018-4fad-4771-9f48-1448f0076ef9)
- Widget for `dimensions` and `metrics` with defaults and then reading from Admin API.
![image](https://github.com/user-attachments/assets/3a4b1d42-9555-499d-90da-04d7586ab4c1)
![image](https://github.com/user-attachments/assets/16efcb11-3547-4eaf-9f28-944fa21c4aa2)
- Added widget for `start_date` and `end_date` on `Google_Analytics.read`.
- Bug fix for `parse` with auto type by reordering to allow numeric dates to be parsed.
- **ToDo**: better exception handling.
2024-11-05 10:11:42 +00:00
Radosław Waśko
aad1107a8e
Add widget for name_filter (#11455)
- Closes #11310
2024-10-31 14:05:08 +00:00
Radosław Waśko
2619399799
Enso_File integration update: Multi-part upload and presigned URL for download (#11440)
- Closes #11330
- Closes #11331
2024-10-31 13:11:42 +00:00
Radosław Waśko
cf5326fbd1
Fixing small bugs uncovered by type checker (#11422)
Once our libraries and tests are compiled with basic inference of method types, some warnings were reported:
```
X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Database\0.0.0-dev\src\DB_Column.enso:1003:19: warning: Calling member method `div` on type Number will result in a No_Such_Method error in runtime.
1003 |         halfway = scale.div 2
|                   ^~~~~~~~~~~
X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Image\0.0.0-dev\src\Matrix.enso:381:21: warning: Invoking a value that has a non-function type (type Image) will result in a Not_Invokable error in runtime.
381 |     to_image self = Image (Image.from_vector self.normalize.to_vector self.rows self.channels)
|                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Table\0.0.0-dev\src\Internal\Multi_Value_Key.enso:94:22: warning: Calling static method `new` on (type Illegal_State) will result in a No_Such_Method error in runtime.
94 |         Error.throw (Illegal_State.new "Ordered_Multi_Value_Key is not intended for usage in unordered collections.")
|                      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```

This PR attempts to fix them.

There was also an expected error in the Examples:
```
X:\NBO\enso\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\lib\Standard\Examples\0.0.0-dev\src\Main.enso:139:36: warning: Calling static method `frobnicate` on (type No_Methods) will result in a No_Such_Method error in runtime.
139 | no_such_method = Panic.recover Any No_Methods.frobnicate . catch
|                                    ^~~~~~~~~~~~~~~~~~~~~
```
To avoid getting a warning, I wrapped this in `(_ : Any)` to make the warning go away. The behaviour of the function is unchanged.
2024-10-31 12:49:50 +00:00
James Dunkerley
610ee5fdec
Expand and improve pretty for core data types, vector and table. (#11438)
-  Alter default `Any.pretty` so constructor is prefixed with type name (as needed now).
![image](https://github.com/user-attachments/assets/72d5ff2f-b567-47e2-becf-2e4acd4d089d)
-  Tests for `pretty` on `Date`.
- `pretty` for  `Date_Time` and  `Time_Of_Day` improved to not have as much noise.
- `pretty` for  `Period`,  `Date_Range` and  `Range`.
- Added custom `pretty` for  `Vector` and  `Array` as built-in method doesn't call through to overrides.
- Added custom `pretty` for  `Column` and  `Table`.
- Bug fix for `pretty` in `Time_Zone` so calls through to `pretty` of the zone_id to ensure safely escaped.
- Initial `default_widget` for `Date` and `Time_Of_Day`.
- Improve widget for `Date.to_date_time`.
![image](https://github.com/user-attachments/assets/18bc1d88-8ea9-42d0-8a9c-bc873e5d6835)
- `to_text`, `to_display_text` and `pretty` for `Enso_Secret`
![image](https://github.com/user-attachments/assets/d850c109-d1af-4b6f-a450-013c4d137805)
- private constructor for `Enso_Secret` as can't be correctly built directly.
- Use `_` for the testing methods in `HTTP` to clarify they shouldn't be used in general code.
2024-10-31 10:04:52 +00:00
Gregory Michael Travis
dc50a7e369
HTTP response caching, with TTL and LRU logic (#11342) 2024-10-30 12:50:35 +00:00
Gregory Michael Travis
442123bba0
Fix Float.parse benchmark regression (#11402) 2024-10-29 22:39:32 +00:00
James Dunkerley
78d9e34840
Excel before 1900 and AWS signed requests. (#11373) 2024-10-28 20:20:06 +00:00
Jaroslav Tulach
9196db414b
Execute Base_Internal_Tests with native enso executable (#11403)
Another set of tests running with _native image_ `enso` executable.
2024-10-25 17:58:35 +00:00
AdRiley
1b6a1f990b
Enable SQLServer Sort Feature and associated tests (#11379)
* Auto-commit work in progress before clean build on 2024-10-15 12:59:18

* checkpoint

* Clean

* Cleaner

* cleaner

* Cleaner

* Green

* stash

* stash

* Fix sort

* Sub 300 failures

* More passing tests

* Auto-commit work in progress before clean build on 2024-10-22 09:00:41

* SQLServer green 265

* remove dead code

* Add doc

* Add generate_column

* Refactor

* refactor

* Refactor

* Refactor

* Add  is_operation_supported_fn

* Fix

* Fix

* Fix

* Code review changes

* Code review feedback

* typos

* Add comment
2024-10-25 13:16:52 +03:00
Jaroslav Tulach
66c09a96af
Execute test/Geo_Tests in enso binary launcher (#11394) 2024-10-25 06:42:39 +02:00
Radosław Waśko
ca9df70ebf
Saving data links to a DB_Table (#11371)
- Closes #11295
2024-10-24 13:18:49 +00:00
Kaz Wesley
33904912ee
Move annotations into fields of Function and ConstructorDefinition (#11374)
Move annotations into fields of Function and ConstructorDefinition.

# Important Notes
New syntax: Constructor argument-definition lines
- Each argument in a type-constructor definition may be specified on its own (indented) line.

Relaxed syntax: Unparenthesized arguments to annotations
- A generic annotation now uses the rest of the line as its argument expression; the expression no longer needs to be parenthesized.
2024-10-23 20:35:06 +00:00
marthasharkey
27a535f6d0
Display error message when Scatterplot Visualization errors out (#11284) 2024-10-18 11:30:14 +00:00
James Dunkerley
5f44c512a8
Handle mixed Date and Date Time column within Excel (#11349)
- If a column contains both Date and DateTime in Excel, we create a DateTime column.
- If the column contains numbers or text as well then we end up with a mixed column.
![image](https://github.com/user-attachments/assets/b0b98d1c-c5c5-41db-8af5-0c946d8a5b92)
2024-10-17 16:42:42 +00:00
Radosław Waśko
d75e20c1d2
Save Database connection as data link, SQL Server data link support (#11343)
- Closes #11294
2024-10-17 09:06:57 +00:00
James Dunkerley
96fa2ee35a
Small fixes.. (#11338)
- Handle calculated header rows in Excel.
![image](https://github.com/user-attachments/assets/e0b307b1-90c2-435b-8d78-9e1b0e8d3932)
- Better reporting of Incomparable Values (bug fix in handler, add catch to add_row_number and running).
![image](https://github.com/user-attachments/assets/9d2ee953-ae5f-45f3-b3fa-6d593529bfc9)
- Remove default from `tokenize` as it was generating hundreds of rows.
- Added error to `Data.read` if no path provided.
![image](https://github.com/user-attachments/assets/71c8cd5f-ec40-4d8c-9972-94aa6fb9d3de)
2024-10-16 17:31:26 +00:00
Radosław Waśko
069e2362dc
Revert "Add Meta.engine_version" (#11336) 2024-10-16 12:09:32 +00:00
Radosław Waśko
21bd05f318
Add Meta.engine_version (#11320)
Numerous times I wasn't sure when running the IDE if I'm running the bundled engine or a development build. Usually this depends on if I launch the standalone IDE or use a development build of project-manager.

Still it's not always obvious, and making sure that your IDE is running the right engine version is very often the first step when debugging issues with e.g. engine changes not showing up properly.

Thus I thought it may be worth to add this method (currently hidden to users in component browser by marking as `PRIVATE`, one has to type it in manually):
![image](https://github.com/user-attachments/assets/13af3df4-49ff-49bb-9b19-601258a8ca02)

I think it should be a helpful tool for debugging.
2024-10-15 13:22:55 +00:00
Radosław Waśko
244effde0c
Enable audit logs for Snowflake backend (#11306)
- Closes #11292
- Tries to fix #11300
2024-10-14 14:30:42 +00:00
AdRiley
03369b9273
Refactor Dialect_Flag (#11273)
* Add property flags

* Update post initial design review

* Add more tests

* Fix tests

* Remove .LCK

* Fix
2024-10-14 10:32:19 +03:00
Radosław Waśko
2843dcbf4a
When connecting to a Postgres database through a datalink stored on Enso Cloud, its asset ID is included in the audit logs (#11291)
- Closes #9869
2024-10-10 15:18:47 +00:00
Radosław Waśko
3458fe4fe1
Accessing and modifying description and labels of Enso Cloud assets (#11255)
- Closes #11227
- Additionally, it should fix #11278 by ensuring that every scheduled message goes to the desired endpoint, by splitting each batch by endpoint.
2024-10-10 12:11:10 +00:00
Hubert Plociniczak
468b643aad
Add retries to Data.read (#11269)
* Add retries to HTTP Get requests

A quick solution to random network failures for GET HTTP requests.
Should reduce the number of IOExceptions that users see while fetching
data.

* Use homemade retry logic for http requests

* Add retries to whole Data.read

Previously, we added retries only to fetch HTTP_Request. That was
insufficient as intermittent errors might happen while reading body's
stream.

Enhanced our simple server's crash endpoint to allow for different kind
of failures as well as simulate random failures.

* Remove retries from Java

Increased the scope of retries in the previous commit.

* nit

* Address PR comments

* PR comments

* Remove builtin
2024-10-10 10:25:16 +02:00
James Dunkerley
9d9c03eeaa
Some Additional Error checks for Enso_File (#11275)
- Check the validity of `Enso_Path` when making an `Enso_File`.
- Check that not an error when fetching asset reference.
- Added display text for `Enso_User`.

![image](https://github.com/user-attachments/assets/77e4bfe8-d4f4-4de5-9ccc-a7c6519d4759)

Check for `to_text` use within Enso_Cloud.
2024-10-09 11:12:25 +00:00
AdRiley
26c711ab14
Enable SQLServer select columns (#11270)
* Enable SQLServer select columns

* Code review feedback

* Code Review feedback
2024-10-08 22:57:36 +03:00
marthasharkey
1ae646419f
add method for xml doc drilldown (#11258) 2024-10-08 15:23:41 +00:00
Gregory Michael Travis
47bf591b5a
Add format parameter to Float.parse (#11229) 2024-10-07 15:43:57 +00:00
Gregory Michael Travis
cce50fab3a
Extend the range of int/float arguments to outside the range of Java long, in ceil, floor, and truncate (#11135) 2024-10-07 11:36:17 -04:00
AdRiley
d0f0770ef4
Add DB_Table feature flags (#11163)
* Refactor

* Fix

* More fixes

* Fixes

* Fix

* Fix imports

* Fix

* Fix

* Fix Test

* Fix

* Fix

* is_feature_supported API

* Add SQLServer Tests

* SQLServer Select_columns functional

* SQLServer Filter functional

* Aggregate

* order feature

* Remove old order flag

* Join

* Union

* Distinct

* Text_cleanse

* Remove Nothing tests

* add_row_number and integration

* Make_table_from

* Feature flag failing tests

* Column_operations

* Set

* date_time off

* Conversion

* More flags

* Merge

* Expressions

* TakeDrop

* Missing

* Cross_join

* More flags

* Temporaily disable

* Fix merge issue

* Update other dbs

* Code review changes

* Code review changes

* Code revoew changes

* Fix

* Fix

* Fix

* Fix

* Fix

* Code review fix

* Typo

* Code review changes

* Casing

* Add comments

* Add comments

* Update

* Fix

* Fix imports
2024-10-07 16:05:09 +01:00
AdRiley
a4f800dbf0
Add Coalesce tests and fix In-Memory version (#11242)
Add tests for coalesce and fix the In-memory version

# Important Notes
The distribution/lib/Standard/Database/0.0.0-dev/src/Internal/IR/From_Spec.enso change isn't actually needed for this MR as I switched away from using literal tables for these tests as trying to mix 2 literal tables doesn't look to be supported. The change I added will let any future developers know that.
2024-10-07 09:02:11 +00:00
Radosław Waśko
f1c2015970
Log failure to refresh the Cloud tokens (#11261)
- Trying to debug #11145, I'm adding logging of when token is refreshed and when that fails, with possibly some details that are included in the response.
- Also, the expired case is now narrowed down from 400-499 to just 401 as that's what [OAuth spec](https://www.rfc-editor.org/rfc/rfc6750.html#section-3.1) says is the correct status code for expired token. Other errors are reported as unexpected error, with more details.
2024-10-07 08:30:41 +00:00