Commit Graph

104 Commits

Author SHA1 Message Date
James Dunkerley
85c8f76805
Fix for Data Quality perf and issue other tweaks. (#11711)
- Fix bug where `DB_Table` data quality indicators broke deserialization in the table viz.
- Memorization of the untrimmed data quality indicator and move to it being an operation and column function.
- If more than 10,000 rows then use a sample for untrimmed.
- ALIASes for blank functions.
- Fix for Snowflake drill down.
- Bug fix for Long and Double columns with Nothings at end.
2024-11-29 17:54:40 +00:00
Radosław Waśko
dc55b4e9ce
Support for Table output in Data.read_many (#11546)
- Closes #11311
2024-11-29 12:06:30 +00:00
marthasharkey
582a9aaa3d
Drive data quality metrics from enso code (#11638)
If a data quality metric is added to the array sent in the table viz json for a table/column the metric will be added to the columns tooltip without the need for any frontend/ts changes.

This doesn't change anything for the users but here is a screenshot to show the same functionality:
![dqm-enso-driven](https://github.com/user-attachments/assets/7bf83d35-0d63-49ac-8d70-1f86dbedc169)
2024-11-25 23:01:09 +00:00
James Dunkerley
d5f0e9ed8c
Fixes a few bits... (#11581)
- Minor fixes from past PR.
- Always use Table viz for Table.
- Fix display to `Always`  for any AWS required parameters.
- Add widget for `Redshift` credentials and alter `Username_And_Password` to require arguments.
- Fix display in `Data` module methods for required parameters.
- Fix `Statistic.bulk_widget`.
- Alter `Google_Analytics` so credentials the first parameter.
- Add support for storing the credential file in Enso cloud in `Google_Analytics`.

![image](https://github.com/user-attachments/assets/f34c4231-9f9f-468b-90e9-bbc7f2374d22)
2024-11-18 22:29:34 +00:00
marthasharkey
f2037ee9d0
Add data quality indicators to table Viz (#11307) 2024-11-08 16:59:09 +00:00
Radosław Waśko
aad1107a8e
Add widget for name_filter (#11455)
- Closes #11310
2024-10-31 14:05:08 +00:00
marthasharkey
27a535f6d0
Display error message when Scatterplot Visualization errors out (#11284) 2024-10-18 11:30:14 +00:00
marthasharkey
a04ae1c9ad
scatterplot- on point click (#11142) 2024-09-24 09:14:39 +00:00
marthasharkey
b53d7b0521
scatterplot- time series support (#11085) 2024-09-20 15:48:28 +00:00
marthasharkey
836953289e
Wip/mk/scatter plot visual improvements (#10959)
* add mutliseries support to d3

* fix types

* fix scales when multiseries

* fix tests

* fix all tests

* basic hardcoded legend working, on only one plot at a time

* basic legend

* legend populating correctly with wrong label names

* legend with tooltip for full length names

* remove unused variale

* key on all nodes

* tooltips on points and labels

* turn on/off d3 'brush' so tool tip is availible to the user

* use colourScale

* fix type problems

* use default colour

* rename variables and remove unneeded code

* fix label

* sort labels in legend

* is multiseries show legend and dont show y label

* address comments

* fix type problem

* Update distribution/lib/Standard/Visualization/0.0.0-dev/src/Scatter_Plot.enso

Co-authored-by: James Dunkerley <jdunkerley@users.noreply.github.com>

---------

Co-authored-by: James Dunkerley <jdunkerley@users.noreply.github.com>
2024-09-06 10:12:55 +01:00
marthasharkey
069cf61b95
Initial enso changes to ScatterPlot (#10871) 2024-08-28 13:39:55 +00:00
marthasharkey
b2237f77c7
Add drilldown to XML (#10824)
This adds the drilldown behaviour to XML_Elements.

![10557-xml-drilldown](https://github.com/user-attachments/assets/062d7de9-c157-4ea2-8a9b-a39f91c26581)
2024-08-19 10:12:42 +00:00
AdRiley
9fc6af12c5
Clean up Text option from read and query (#10790)
* Revert

* from

* Fix

* fix

* Last few details

* Fix

* Fix test

* Fix

* Cleanup

* Fix error message

* fix

* Fix

* Improve error message
2024-08-13 13:25:21 +03:00
marthasharkey
0d5d01a294
Wip/mk/drive drilldown from enso (#10724)
This moves some of the drilldown logic to the enso code and utilises the new 'get_rows' method for Table
2024-08-12 12:51:51 +00:00
James Dunkerley
4c0647ea29
Stop publishing First/Last as constructors and use auto-scoping for take and drop. (#10467)
- Removes `First` and `Last` from the `Standard.Base` exports.
- Enable auto-scoping for all `Index_Sub_Range` and `Text_Sub_Range`.
- Update all use of those methods to use auto-scoping.
2024-07-08 10:26:30 +00:00
marthasharkey
ee39fd7f53
Visualise excelwork book sheets, db tables/schemas and add node adding gets to query them (#10362)
changes to current table (and db table) on index click:
- node added on double click
- tooltip added
- cursor changes to pointer
- styled as a link

![7473-table-w-links](https://github.com/enso-org/enso/assets/170310417/5e60c177-3f83-4db7-be86-fa8a9d493204)


Row table types:
Added on click to show value
![7473-row-links](https://github.com/enso-org/enso/assets/170310417/82f878ea-420d-4308-99bf-c77a6340a8c3)



Excel Workbook connections
- show sheet names in column
- sheet names are clickable that add a 'read' node for the corresponding sheet
- Shown in table with header title: Sheets

![7473-excel-links-sheets](https://github.com/enso-org/enso/assets/170310417/748f524e-5cca-4c20-b458-132af9a57ec1)


SQLite Connection:
- shows available schemas in column
- schemas are clickable that add a 'query' node for the corresponding schema/table
![7473-sql-lite-links](https://github.com/enso-org/enso/assets/170310417/21a8006f-c462-4128-874d-05380d3bab00)


Postgres Connection:
- shows available tables in a column
- tables are clickable and add a 'query' node for the corresponding table
![7473-postgres-links](https://github.com/enso-org/enso/assets/170310417/66b134ff-80fd-4995-b445-505919e25cfa)

JS_Object
- style keys as links
![7473-json-links](https://github.com/enso-org/enso/assets/170310417/9ec17c85-c1f4-42ab-b40a-06d6bc29d54f)
2024-07-03 13:58:07 +00:00
marthasharkey
bb16db9c79
Remove legacy matrix and legacy objects from table visualization and remove the index column (#10260)
- remove legacy matrix and object types from vue component and any further code relating to those
- remove the index and index header being sent in the json for tables
- add has_index_col flag for json hat previously sent 'indicies_header' and 'indicies' so that index/# column is still rendered where required
2024-06-17 09:17:20 +00:00
marthasharkey
55af1b9ffd
Add icon to indicate value type in table (#10225)
closes #10018

Sends the value type within the json for table visualisation
Header uses a html template to show it's value type to the right of the title
Displays the value type also in a tooltip, this displays the types "display_text"

<img width="392" alt="image" src="https://github.com/enso-org/enso/assets/170310417/0828e6a2-b30f-4ac7-9a8f-46b4a9cfac91">


tooltip:
<img width="498" alt="image" src="https://github.com/enso-org/enso/assets/170310417/f9964f90-9337-42d3-a0ef-3c58f6d74621">
2024-06-12 09:06:55 +00:00
AdRiley
af4177e61f
Make table default visualisation (#10128)
Makes table the default vis type.

Before:
![image](https://github.com/enso-org/enso/assets/1720119/8a5d113a-ac06-4dd8-afdb-151cdbc149ab)

After:
![image](https://github.com/enso-org/enso/assets/1720119/d7c33cf7-355d-4517-8967-1963ff506f97)

Closes #10093
2024-05-31 15:02:02 +00:00
James Dunkerley
c0fd6eed2d
Small tweaks from Ned, Mark, Cass and Steve's feedback. (#9859)
- Rename `.info` to `.column_info`.
- Rename `.select_by_type` to `.select_columns_by_type`. (to be merged into `select_columns` functionality)
- Rename `.remove_by_type` to `.remove_columns_by_type`. (to be merged into `remove_columns` functionality)
- Adding "field" `ALIAS`es.
- Making `Table.Value` private (and hence hiding `java_table`).
- Making `Column.Value` private (and hence hiding `java_column`, added method to `Storage` to allow tests to work).
- Making `DB_Table.Value` private (and hence hiding `connection`, `context` and `internal_columns`).
- Making `DB_Column.Value` private (and hence hiding `connection`, `sql_type_reference`, `expression` and `context`).
- Add widgets for everywhere `Filter_Condition` used (fix missing entry for lower, can't use auto-scoping).
![image](https://github.com/enso-org/enso/assets/4699705/bad7d5c8-cd01-4384-a1c8-dbcd7a5ad92b)
- Fix for widgets in `Simple_Calculation`: `date_add` and `format`.
![image](https://github.com/enso-org/enso/assets/4699705/14eeef2e-5069-4a88-8b68-eb675497addf)
![image](https://github.com/enso-org/enso/assets/4699705/75d4e097-f171-4bf6-a27e-f9d396c44a92)
- Fix for nested `Row`, `Column` and `DB_Column` rendering in Table Viz.
![image](https://github.com/enso-org/enso/assets/4699705/8c50c519-7c34-4228-bab7-be938a6ce1bf)
![image](https://github.com/enso-org/enso/assets/4699705/c83ad301-f31e-451d-a8be-b022e1456b4e)

# Important Notes
- Can't use `.` to access a private constructor field when mapping, have to use a lambda.
- We should agree convention for private backing fields which are wrapped by a public getter (went with `internal_` for this pass).
2024-05-06 13:49:12 +00:00
James Dunkerley
d2e6ff260e
Restructure SQLite_Details. (#9832)
```
type SQLite_Details
SQLite location:File|In_Memory

type In_Memory
```
to
```
type SQLite
From_File location:File

In_Memory
```

# Important Notes
Splits the In-Memory entry for Database Connect but still works nicely.

![image](https://github.com/enso-org/enso/assets/4699705/ec798ce0-9f41-4903-a2fd-722a9e37743c)

![image](https://github.com/enso-org/enso/assets/4699705/f233b055-893e-4c56-a23d-562e982560f6)
2024-05-01 22:15:41 +00:00
Jaroslav Tulach
ff62c1e25f
Access non-self arguments when widget annotations are computed (#9410) 2024-04-24 03:58:32 +02:00
James Dunkerley
e262801daa
Restructure Standard.Table. (#9559)
Move the types from `Standard.Table.Data` to `Standard.Table`.

Exceptions:
- `Standard.Table.Data.Report_Unmatched` => `Standard.Table.Constants`.
- `Standard.Table.Data.Join_Kind_Cross` => `Standard.Table.Internal.Join_Kind_Cross`.
Also removed constructor as an atom type.
- `Standard.Table.Extensions.Table_Ref` => `Standard.Table.Internal.Table_Ref`.
- `Standard.Table.Data.Type.Value_Type_Helpers` => `Standard.Table.Internal.Value_Type_Helpers`.
- `Standard.Table.Data.Type.Enso_Types` => `Standard.Table.Internal.Value_Type_Helpers`.
- `Standard.Table.Data.Type.Storage` => `Standard.Table.Internal.Storage`.

Changed all `Standard.Table` imports inside project to be project.
Favoured importing from `Standard.Table.Main` in `Standard.Database`.
Also fixed some linting in Enso_File.
2024-03-27 17:10:43 +00:00
Radosław Waśko
a3bf5a0be5
Split Excel_Format into 3 constructors (#9308)
Splitting the `Excel_Format` into 3 constructors.
2024-03-08 14:26:30 +00:00
James Dunkerley
c7d693dfc8
Move Standard.Database.Data to Standard.Database. (#9321)
Moves the types out of `Data`.
2024-03-07 14:43:38 +00:00
Pavel Marek
21e1284086
Enso tests can be run with filter from cmdline (#9065)
Simplify the `Test.Suite.run_with_filter` to accept a single filter parameter that searches for all the groups and specs that matches that filter. This filter can be a simple text provided from the command line.

# Important Notes
- Pending groups are now printed at the end of the run
- `Test.Suite.run_with_filter` is simplified to accept a single filter parameter that is either `Text` or `Nothing`. See the docs.
- Passing a filter from the command line is therefore straightforward, it is treated as a regex.
- For convenience, I have left all the `main` methods in all the test sources. I have just refactored them to accept the `filter` argument from the command line.
- For example, to run only a single spec from `Vector_Spec.enso`, invoke `enso --run test/Base_Tests/src/Data/Vector_Spec.enso "should allow vector creation with a programmatic constructor"`
- **Majority of the PR is a regex replace** of `^main =` for `main filter=Nothing =` and of `suite.run_with_filter` for `suite.run_with_filter filter`.
- **Fixed some internal engine bugs:**
- `AtomWithHole` allows to specify only one hole - https://github.com/enso-org/enso/pull/9065/files#diff-0f7bb7e85cf86a965de133aa7e6b5958ceb889bd1921c01e00d3a9ceb19626ef
- NaN keys in hash maps are handled in polyglot maps as well - c5257f6c2b78f893214ff67300893b593ea05e21..db4b3c0e9828ee79208d52e02586b24bb845b0d6
2024-02-22 12:31:44 +00:00
James Dunkerley
fa6fccb99e
Refactoring Table.set for GUI2 and other GUI fixes (#9128)
- Added `regex` function to `Standard.Base` as a way to easily and cleanly make regular expressions.
- Added `expr` and `Expression.Value` to distinguish expressions from text values.
- Fixed issues with `Table.join` widget so dropdown exists.
- Needed fully qualified name.
- Added default empty text values for right column to provided text input boxes.
- Deprecate `Table.filter_by_expression` and allow `Table.filter` to take an `Expression`.
- Added `Simple_Expression` and deprecated `Column_Operation`. Changes the order so takes a column then a calculation.
- Rename `column` to `value` and `new_name` to `as` on `Table.set`.
- Rename `name_column` to `names` on `Table.cross_tab`.
- Removed `Column_Ref.Expression` in favour of using `Expression.Value`.
2024-02-21 23:51:41 +00:00
AdRiley
4cbdc9dfb6
Remove table aliases (#9098)
Now Table and DB_Table have different names we can remove a lot of aliases. This closes #8981
2024-02-20 01:03:57 +00:00
AdRiley
27ec236c19
rename database Table to DB_Table (#9067)
Rename database.Table to database.DB_Table to avoid name collisions as part of https://github.com/enso-org/enso/issues/8981
2024-02-19 17:31:58 +00:00
James Dunkerley
8c197f325b
Update the aggregate API to take a separate group_by (#9027)
Separate out the `Group_By` from the column definition in `aggregate`.
![image](https://github.com/enso-org/enso/assets/4699705/6b4f03bc-1c4a-4582-b38a-ba528ae94167)

Supports the old API with a warning attached about deprecation:
![image](https://github.com/enso-org/enso/assets/4699705/0cc42ff7-6047-41a5-bb99-c717d06d0d93)

Widgets have been updated with `Group_By` removed from the dropdown.
2024-02-13 10:23:59 +00:00
Pavel Marek
f3f0697d56
Merge Test_New into Test (#8991)
Merges the temporary `Test_New` library into `Test`. This is the last PR in the series of PRs that refactor all the stdlib tests to the builder API.
2024-02-08 11:25:13 +00:00
James Dunkerley
0c39f8ec04
Allow Filter_Condition to be inverted. (#8861)
- Various linting fixes (doc comments and type annotations etc.).
- Add an action to determine if a `Filter_Condition` is keep or remove.

https://github.com/enso-org/enso/assets/4699705/69ba2bd3-8893-4237-acc4-eb01f534a209

- Remove `Not_In`, `Not_Contains` and `Not_Like` from `Filter_Condition`.

- Ability to use an `Expression` as a `Column_Ref`.

https://github.com/enso-org/enso/assets/4699705/16a2e030-f8f9-4f59-beca-2646f56fcb90
2024-02-07 14:36:14 +00:00
Pavel Marek
83fffd9c05
Refactor stdlib tests to the builder API (#8968)
Follow-up of #8890

Refactor the rest of the tests to the builder API (`Test_New`):
- `Image_Tests`
- `Geo_Tests`
- `Google_Api_Test`
- `Examples_Test`
- `AWS_Tests`
- `Meta_Test_Suite_Tests`
- `Visualization_Tests`

# Important Notes
- Unrelated: Fix NPE in `File.new "/" . name`
2024-02-07 13:22:17 +00:00
Jaroslav Tulach
7f0cb88fa1
Consistent simple and qualified type name (#8448)
Fixes #8255 by unifying `get_qualified_type_name` and `get_simple_type_name` implementations.
2023-12-06 04:30:24 +00:00
Jaroslav Tulach
81f06456bf
400x faster with linear hashing of the hash map entries (#8425)
Fixes #5233 by removing `EconomicMap` & co. and using plain old good _linear hashing_. Fixes #8090 by introducing `StorageEntry.removed()` rather than copying the builder on each removal.
2023-12-01 06:43:13 +00:00
Jaroslav Tulach
477dd82670
Drop-down widgets for extension functions via UnresolvedSymbol (#7115)
Fixes #6955 by:
- using `visualisationModule` to specify the module where the visualization is to be used
- referring to method in `Meta.get_annotation` with `.method_name` - e.g. unresolved symbol notation
- evaluating arguments to `Meta.get_annotation` in the context of the user module (which can access the extension functions)
2023-06-27 15:19:42 +00:00
Hubert Plociniczak
4e5cb9cca6
Consistent naming of visualization (#7124) 2023-06-26 17:04:35 +00:00
Pavel Marek
67821bf8df
Add compiler pass that discovers ambiguous imports (#6868)
Add a new compiler pass that analyses duplicated and ambiguous symbols from imports
2023-06-14 12:18:57 +02:00
Radosław Waśko
b513839418
Refactor create_database_table into Connection.create_table and select_into_database_table, implement Set. (#6925)
First part for #6498 - refactoring of the upload infrastructure, in preparation for `update_database_table`.

Implemented a `Set` data structure which was long needed.

The APIs are added and an initial implementation is created, but it is not complete - but it has grown significantly already so the remaining implementation will be done as a separate PR.

Adds some basic ability for a function to ensure that it is only executed from within a transaction.
2023-06-06 10:36:05 +00:00
James Dunkerley
db96bd2e2c
Small fixes from book club. (#6933)
- Add the missing dropdowns for `Locale` and `Encoding`.
- Correct a few mismatched type signatures.
- Adjust `order_by` calls with a single `Sort_Column` to call in a Vector.
- Adjust parameter names for `transpose`.
- Fix for the table viz: escape HTML and `suppressFieldDotNotation`.
- Use `Filter_Condition.Equal True` for the default filter.
- Adjust `Data.fetch` to return the response on success when parse fails. Rename `parse` to `try_auto_parse`.
- Add various aliases for methods.
- Add tests for `Table.set` when using a `Vector`, `Range` or `Date_Range`.
- Add check for mismatched length on `Table.set`.

![image](https://github.com/enso-org/enso/assets/4699705/23ea0ba3-2b05-4af8-afd9-f35b55446c24)

![image](https://github.com/enso-org/enso/assets/4699705/8b0253e6-e9e8-490a-9607-0da51ab5a215)
2023-06-05 13:57:30 +00:00
James Dunkerley
343b5fb085
Execution control for Table.write and various widget tweaks... (#6835)
- Adds execution control to `Table.write`.
- Refactored the `Text.write` to make part reusable.
- Tidied up some legacy mess in tests.
- Add easier flow to go from `Text` to an `URI` to fetching data.
- Add decode functions to `Response` and `Response_Body`.
- Fix issue with 0 length regex matches (using same as Python and .Net approach).
- Add various ALIAS entries to make function discovery easier.
- Sort a lot of drop down and vector editors out (including switch to fully qualified names).
2023-06-01 22:10:03 +00:00
Jaroslav Tulach
c6cb937c0a
Runtime check of ascribed types (#6790) 2023-05-30 11:07:26 +02:00
Radosław Waśko
f5071a17fd
Improving widgets for take/drop (#6641)
Related to #6410

# Important Notes
- Updated some `Meta` methods (needed for error handling):
- `Meta.Type` now has `name` and `qualified_name`.
- `Meta.Constructor` has `declaring_type` allowing to get the type that this constructor is associated with.
2023-05-12 15:33:15 +00:00
James Dunkerley
bc0db18a6e
Small changes from Book Club issues (#6533)
- Add dropdown to tokenize and split `column`.
- Remove the custom `Join_Kind` dropdown.
- Adjust split and tokenize names to start numbering from 1, not 0.
- Add JS_Object serialization for Period.
- Add `days_until` and `until` to `Date`.
- Add `Date_Period.Day` and create `next` and `previous` on `Date`.
- Use simple names with `File_Format` dropdown.
- Avoid using `Main.enso` based imports in `Standard.Base.Data.Map` and `Standard.Base.Data.Text.Helpers`.
- Remove an incorrect import from `Standard.Database.Data.Table`.

From #6587:

A few small changes, lots of lines because this affected lots of tests:
- `Table.join` now defaults to `Join_Kind.Left_Outer`, to avoid losing rows in the left table unexpectedly. If the user really wants to have an Inner join, they can switch to it.
- `Table.join` now defaults to joining columns by name not by index - it looks in the right table for a column with the same name as the first column in left table.
- Missing Input Column errors now specify which table they refer to in the join.
- The unique name suffix in column renaming / default column names when loading from file is now a space instead of underscore.
2023-05-06 10:10:24 +00:00
Radosław Waśko
41a8257e8d
Separating Redshift connector from Database library into a new AWS library (#6550)
Related to #5777
2023-05-04 17:36:51 +00:00
James Dunkerley
bb8f910be9
Continued Execution Context work and some little fixes (#6506)
- Add `with_disabled` shortcut for `Context`.
- Protect `Image.write` behind `Context.Output`.
- Correct text on the `Forbidden_Operation` error message.
- Remove context overrides from tests.
- Add `File` operations tests with `Context.Output` disabled.
- Add tests for `Text.write` operations with `Context.Output` disabled.
- Use a better method to make `File_Format` dropdown widget.
- Fix bug in `Invalid_Format.to_display_text`.
2023-05-03 08:01:16 +00:00
Pavel Marek
3a42d0ce76
Reimplement enso_project as a proper builtin (#6352)
Remove the magical code generation of `enso_project` method from codegen phase and reimplement it as a proper builtin method.

The old behavior of `enso_project` was special, and violated the language semantics (regarding the `self` argument):
- It was implicitly declared in every module, so it could be called without a self argument.
- It can be called with explicit module as self argument, e.g. `Base.enso_project`, or `Visualizations.enso_project`.

Let's avoid implicit methods on modules and let's be explicit. Let's reimplement the `enso_project` as a builtin method. To comply with the language semantics, we will have to change the signature a bit:
- `enso_project` is a static method in the `Standard.Base.Meta.Enso_Project` module.
- It takes an optional `project` argument (instead of taking it as an explicit self argument).

Having the `enso_project` defined as a (shadowed) builtin method, we will automatically have suggestions created for it.

# Important Notes
- Truffle nodes are no longer generated in codegen phase for the `enso_project` method. It is a standard builtin now.
- The minimal import to use `enso_project` is now `from Standard.Base.Meta.Enso_Project import enso_project`.
- Tested implicitly by `org.enso.compiler.ExecCompilerTest#testInvalidEnsoProjectRef`.
2023-05-02 16:40:58 +00:00
James Dunkerley
6b0c682b08
Add Execution Context control to Text.write (#6459)
- Adjusted `Context.is_enabled` to support default argument (moved built in so can have defaults).
- Made `environment` case-insensitive.
- Bug fix for play button.
- Short hand to execute within an enabled context.
- Forbid file writing if the Output context is disabled with a `Forbidden_Operation` error.
- Add temporary file support via `File.create_temporary_file` which is deleted on exit of JVM.
- Execution Context first pass in `Text.write`.
- Added dry run warning.
- Writes to a temporary file if disabled.
- Created a `DryRunFileManager` which will create and manage the temporary files.
- Added `format` dropdown to `File.read` and `Data.read`.
- Renamed `JSON_File` to `JSON_Format` to be consistent.

(still to unit test).
2023-04-29 08:39:18 +00:00
Radosław Waśko
cdd0065800
Change defaults for Connection.tables and ensure that Connection.query recognizes all available tables (#6443)
Closes #6398
2023-04-29 00:13:29 +00:00
Radosław Waśko
462016a428
Create database table from memory (#6429) 2023-04-27 20:06:17 +00:00