enso/gui/docs/product/visualizations.md

Visualization workflow

Purpose of visualizations

Visualizations have two main purposes:

• Display results of nodes
  Each node can be assigned with one or more visualization. After a node computes its new value, the visualization shows it in an understandable way to the user. Please note that a single node can be assigned with multiple visualizations at the same time. For example, a node might want to display a map of locations and their list at the same time next to each other.

• Provide interactive way to generate new data
  In a widget mode (described in detail later), visualizations provide users with an interactive GUI to define data. For example, a map visualization can both display locations, as wekk as allow the user to pick locations by cliking with a mouse. Similarly, the histogram can both display a list of numbers, and can be manually draw with the mouse producing such a list. Several numbers can be visualized as a table of sliders, which can also be used to interactively generate a table of numbers. Image visualizations can behave like an image editor, etc.

Visualization Display Forms

Visualizations can be displayed in the following ways:

• Attached to nodes
  In this mode, visualizations display the most recent result of the node. They behave like an integrated part of the node. Whenever you move the node, the visualization moves as well. This mode can be toggled by tapping the spacebar.

• Fullscreen
  Visualization attached to node can grow (animate) to ocupy full IDE visual space. This mode can be triggered on the recently selected node (in case many nodes are selected, the last selected node will be used) by either pressing keeping the spacebar pressed for longer than approx 0.5s, or by tapping it twice. In the former case, the visualization shrinks to each original form whenever we release space, in the later, whenever we press space again.

• Detached
  Visualizations attached to nodes can be detached, scaled, and placed freely across the visual canvas (we might introduce a special place where you can put such visualizations). This is useful when defining dashboards or reports. We also plan to provide a notebook-like experience where you can write text mixed with visualizations (including widgets for an interactive experience).

• Widgets
  In this mode visualizations behave like nodes but do not display expressions. They have one input and one output port. If the input port is connected, the visualization displays its value and passes its to the output port. In case it is not connected, the visualization becomes an interactive widget allowing the user to specify data. For example, a map visualization will allow the user to manually pick locations. After each change, the new locations will be sent to the output port. Under the hood, widgets are represented as nodes and their code lines are assigned with a dedicated "visualization" metadata. Visualizations generate expressions always in the form of name = data, where data is a hardcoded data produced from the visualization. For example, when user clicks the map to define locations, the data could be a string literal containing locations encoded in JSON.

Choosing a Visualization Type.

When a new data is provided to a visualization, the visualization registry searches for all visualizations that match it (see visualization registry to learn more). For example, when a data of type [Int] (list of ints) is produced, all visualizations which matches [Int], like [Int], [a], or a will be found. Each type can be associated with a default visualization. For example, [Int] might define that its default visualization is a plot. If no default visualization is defined, a JSON visualization is used. Each visualization has a drop-down menu allowinh the user switching to another visualization type.

Active Visualizations

When visualizations are displayed on the stage, they are not active by default, which means, they do not capture keyboard shortcuts. Visualization becomes active when user clicks it. Visualizations are deactivated by clicking in the background of the node editor. When a visualization is active, all other elements should be slightly dimmed, or the visualization should get a selection border (to be defined). Active visualizations capture all keyboard shortcuts, but the space bar presses. Fullscreen visualizations are considered active by default.

HTML and Native Visualizations

There are two main types of visualizations - Html and Native. The later uses the BaseGL shape API to draw on the screen. We prefer the later as it integrates tightly with our framework and allows for much better performance. However, there is already many visualizations in HTML/JS and we need to provide support for them as well. HTML visualizations are required to be displayed in dedicated div elements. This has several consequences. Firstly, the browser needs to layout them, taking into account the current camera view, etc. It is costly. Refreshing CSS3D styles of 100 visualizations can absolutely kill the interactive performance. On the other hand, refreshing the position of 10k Native visualizations is almost free. Secondly, they need to be handled by our engine in such way that we can interact with them. For that purpose, the current Scene implementation defines three layers - top HTML layer, middle WebGL layer, and bottom HTML layer. The HTML visualizations are created and displayed on the bottom layer by default. Whenever an HTML visualization gets active, it should be moved to the top layer.

Visualization Registry

Visualizations are user-defined. Enso ships with a set of predefined visualizations, but they are in no way different than user-defined, they are just defined for you. Visualizations can be defined either as HTML or native visualization and can be defined in JS or WASM (or any language that compiles to one of these). Visualizations are stored on disk on the server-side and are provided to the GUI by the server. Users can upload their custom visualizations as well. Each visualization is registered in the visualization map. The map maps an Enso type to a set of visualizations defined for that type. The type might be very generic, like [a] (which in Enso terms means list of any elements).

Defining a Visualization

This needs to be described in detail. For now, we can just assume that the user is allowed to create visualization and register it in Enso.

Sending Data to Visualizations

Lazy Visualizations

Very important information is how visualization architecture works to make them interactive and fast. Whenever new data is computed by the compiler and visualization is attached to it, it is sent to GUI to be displayed. However, sending really big chunks of data will kill the performance. When defining a visualization user is capable of defining a chunk of Luna code (as a string). This code is part of the visualization definition and is stored server-side. Visualizations are allowed to change the code at runtime. This code defines an Enso function, which will be run by the compiler on data the visualization is attached to. Only the results of this code will be sent to the GUI. For example, imagine you want to display a heatmap of 10 million points on a map. And these points change rapidly. Sending such amount of information via WebSocket could be too much, and you (as the visualization author) might decide that the visualization image should be generated on the server, and your visualization is meant only to display the resulting image. In such a scenario, you can define in your visualization an Enso function which will compute the image on the server!

Binary and Text (JSON) Formats

Each visualization can choose whether it supports either binary or JSON input. The input format defaults to JSON. The data from the server is always sent to GUI in a binary channel, however, when JSON format is selected, it is first converted to JSON representation on the server side. We can assume that all Enso data types have defined conversion to JSON by default. If the visualization input is defined as JSON input, the binary stream will be converted to JSON by the GUI engine before passing to visualization. It is up to the visualization author to handle the textual or binary form.