enso/gui/docs/product/visualizations.md

layout	title	category	tags
developer-doc	Visualization Workflow	product	product

Visualization Workflow

Purpose of visualizations

Visualizations have two main purposes:

• Display results of nodes
  Each node can be assigned with one or more visualization. After a node computes its new value, the visualization shows it in an understandable way to the user. Please note that a single node can be assigned with multiple visualizations at the same time. For example, a node might want to display a map of locations, and their list at the same time next to each other.

• Provide interactive way to generate new data
  In a widget mode (described in detail later), visualizations provide users with an interactive GUI to define data. For example, a map visualization can both display locations, as well as allowing the user to pick locations by clicking with a mouse. Similarly, the histogram can both display a list of numbers, and can be manually draw with the mouse producing such a list. Several numbers can be visualized as a table of sliders, which can also be used to interactively generate a table of numbers. Image visualizations can behave like an image editor, etc.

Visualization Display Forms

Visualizations can be displayed in the following ways:

• Attached to nodes In this mode, visualizations display the most recent result of the node. They behave like an integrated part of the node. Whenever you move the node, the visualization moves as well. This mode can be toggled by tapping the spacebar.

• Fullscreen
  Visualization attached to node can grow (animate) to ocupy full IDE visual space. This mode can be triggered on the recently selected node (in case many nodes are selected, the last selected node will be used) by either pressing keeping the spacebar pressed for longer than approx 0.5s, or by tapping it twice. In the former case, the visualization shrinks to each original form whenever we release space, in the later, whenever we press space again.

• Detached
  Visualizations attached to nodes can be detached, scaled, and placed freely across the visual canvas (we might introduce a special place where you can put such visualizations). This is useful when defining dashboards or reports. We also plan to provide a notebook-like experience where you can write text mixed with visualizations (including widgets for an interactive experience).

• Widgets
  In this mode visualizations behave like nodes but do not display expressions. They have one input and one output port. If the input port is connected, the visualization displays its value and passes its to the output port. In case it is not connected, the visualization becomes an interactive widget allowing the user to specify data. For example, a map visualization will allow the user to manually pick locations. After each change, the new locations will be sent to the output port. Under the hood, widgets are represented as nodes and their code lines are assigned with a dedicated "visualization" metadata. Visualizations generate expressions always in the form of name = data, where data is a hardcoded data produced from the visualization. For example, when user clicks the map to define locations, the data could be a string literal containing locations encoded in JSON.

Choosing a Visualization Type.

When a new data is provided to a visualization, the visualization registry searches for all visualizations that match it (see visualization registry to learn more). For example, when a data of type [Int] (list of ints) is produced, all visualizations which matches [Int], like [Int], [a], or a will be found. Each type can be associated with a default visualization. For example, [Int] might define that its default visualization is a plot. If no default visualization is defined, a JSON visualization is used. Each visualization has a drop-down menu allowinh the user switching to another visualization type.

Active Visualizations

When visualizations are displayed on the stage, they are not active by default, which means, they do not capture keyboard shortcuts. Visualization becomes active when user clicks it. Visualizations are deactivated by clicking in the background of the node editor. When a visualization is active, all other elements should be slightly dimmed, or the visualization should get a selection border (to be defined). Active visualizations capture all keyboard shortcuts, but the space bar presses. Fullscreen visualizations are considered active by default.

HTML and Native Visualizations

There are two main types of visualizations - Html and Native. The later uses the BaseGL shape API to draw on the screen. We prefer the later as it integrates tightly with our framework and allows for much better performance. However, there is already many visualizations in HTML/JS and we need to provide support for them as well. HTML visualizations are required to be displayed in dedicated div elements. This has several consequences. Firstly, the browser needs to layout them, taking into account the current camera view, etc. It is costly. Refreshing CSS3D styles of 100 visualizations can absolutely kill the interactive performance. On the other hand, refreshing the position of 10k Native visualizations is almost free. Secondly, they need to be handled by our engine in such way that we can interact with them. For that purpose, the current Scene implementation defines three layers - top HTML layer, middle WebGL layer, and bottom HTML layer. The HTML visualizations are created and displayed on the bottom layer by default. Whenever an HTML visualization gets active, it should be moved to the top layer.

Visualization Registry

Visualizations are user-defined. Enso ships with a set of predefined visualizations, but they are in no way different than user-defined, they are just defined for you. Visualizations can be defined either as HTML or native visualization and can be defined in JS or WASM (or any language that compiles to one of these). Visualizations are stored on disk on the server-side and are provided to the GUI by the server. Users can upload their custom visualizations as well. Each visualization is registered in the visualization map. The map maps an Enso type to a set of visualizations defined for that type. The type might be very generic, like [a] (which in Enso terms means list of any elements).

Defining a Visualization

Visualizations is planned to be defined both with Enso and JavaScript but, for now, only JavaScript visualizations are supported.

Because IDE lacks support for editing any other file besides Main.enso, the user has to create it outside of IDE in the visualization folder of the Enso project, as demonstrated bellow.

Custom Visualization Example

Every visualization must reside in the visualization folder of the user's project. For instance:

└─ ProjectName
   ├─ src
   │  └─ Main.enso
   └─ visualization
      └─ bubble.js

Visualizations can be defined as a JavaScript function which returns a class of a shape specified below. Consider the following definition:

console.log("Hi, this definition is being registered now!")

return class BubbleVisualization extends Visualization {
    static inputType = "Any"

    onDataReceived(data) {
        const xmlns = "http://www.w3.org/2000/svg";
        while (this.dom.firstChild) {
            this.dom.removeChild(this.dom.lastChild);
        }
        const width   = this.dom.getAttributeNS(null, "width");
        const height  = this.dom.getAttributeNS(null, "height");
        const svgElem = document.createElementNS(xmlns, "svg");
        svgElem.setAttributeNS(null, "id"     , "vis-svg");
        svgElem.setAttributeNS(null, "viewBox", "0 0 " + width + " " + height);
        svgElem.setAttributeNS(null, "width"  , "100%");
        svgElem.setAttributeNS(null, "height" , "100%");
        this.dom.appendChild(svgElem);
        data.forEach(data => {
            const bubble = document.createElementNS(xmlns,"circle");
            bubble.setAttributeNS(null,"stroke", "black");
            bubble.setAttributeNS(null,"fill"  , "red");
            bubble.setAttributeNS(null,"r"     , data[2]);
            bubble.setAttributeNS(null,"cx"    , data[0]);
            bubble.setAttributeNS(null,"cy"    , data[1]);
            svgElem.appendChild(bubble);
        });
    }

    setSize(size) {
        this.dom.setAttributeNS(null, "width", size[0]);
        this.dom.setAttributeNS(null, "height", size[1]);
    }
}

In particular:

• [Required] Source code

  Visualization definition has to be a valid body of JavaScript function which returns a class definition. Instances of that class will be considered separate visualizations. You are allowed to use global variables / global state across visualizations of the same type, but you are highly advised not to do so.

• [Required] Visualization superclass

  The class returned by the definition function should extend the predefined Visualization class. Classes which do not extend it, will not be registered as visualizations. The superclass defines a default constructor and a set of utils:

  • The setPreprocessor(code) method allowing setting an Enso code which will be evaluated on server-side before sending data to visualization.
  • The dom field, which will be initialized in the constructor to the DOM symbol used to host the visualization content. You are free to modify the DOM element, including adding other elements as its children.

• [Optional] Field label

  The static field label is an user-facing name used to identify the visualization. You are not allowed to define several visualizations of the same name in the same Enso library. In case the field is missing, the name will be inferred from the class name by splitting the camel-case name into chunks and converting them to lowercase string.

• [Optional] Field inputType

  The static field inputType is used to determine which Enso data types this visualization can be used for. Its value should be a valid Enso type, like "String | Int". In case the field is an empty string or it is missing, it will default to "Any", which is a type containing all other types. It is a rare case when you want to define a visualization which is able to work with just any data type, so you are highly advised to provide the type definition.

• [Optional] Field inputFormat

  The static field inputFormat is used to determine what format the data should be provided to the onDataReceived function. Currently, the only valid option is "json", but it will be possible to set it to "binary" in the future. In the later case, it is up to the visualization author to manage the binary stream received from the server.

• [Optional] Constructor

  The visualization will be instantiated by providing the constructor with a configuration object. The shape of the configuration object is not part of the public API and can change between releases of this library. You have to pass it unchanged to the superclass constructor.

• [Optional] Function onDataReceived

  The onDataReceived(data) method is called on every new data chunk received from the server. Note that the visualization will receive the "full data" if you are not using the setPreprocessor method.

• [Optional] Function setSize

  The setSize(size) method is called on every size change of the visualization. You should not draw outside of the provided area, however, if you do so, it will be clipped to the provided area automatically. The size parameter contains two fields width and height expressed in pixels.

Sending Data to Visualizations

Lazy Visualizations

Very important information is how visualization architecture works to make them interactive and fast. Whenever new data is computed by the compiler and visualization is attached to it, it is sent to GUI to be displayed. However, sending really big chunks of data will kill the performance. When defining a visualization user is capable of defining a chunk of Luna code (as a string). This code is part of the visualization definition and is stored server-side. Visualizations are allowed to change the code at runtime. This code defines an Enso function, which will be run by the compiler on data the visualization is attached to. Only the results of this code will be sent to the GUI. For example, imagine you want to display a heatmap of 10 million points on a map. And these points change rapidly. Sending such amount of information via WebSocket could be too much, and you (as the visualization author) might decide that the visualization image should be generated on the server, and your visualization is meant only to display the resulting image. In such a scenario, you can define in your visualization an Enso function which will compute the image on the server!

Binary and Text (JSON) Formats

Each visualization can choose whether it supports either binary or JSON input. The input format defaults to JSON. The data from the server is always sent to GUI in a binary channel, however, when JSON format is selected, it is first converted to JSON representation on the server side. We can assume that all Enso data types have defined conversion to JSON by default. If the visualization input is defined as JSON input, the binary stream will be converted to JSON by the GUI engine before passing to visualization. It is up to the visualization author to handle the textual or binary form.