Add org.enso.compiler.dumpIr system prop (#10740)

Working on compiler IR is a daunting task. I have therefore added a new system property `enso.compiler.dumpIr` that will help with that. It dumps the encountered IRs to `ir-dumps` directory in the [GraphViz](www.graphviz.org) format. More info in updated docs.

Note that all the functionality to dump IRs to `dot` files was already implemented. This PR just adds the command line option and updates docs.

# Important Notes
- `--dump-graphs` cmd line option is removed as per [Jaroslav's request](https://github.com/enso-org/enso/pull/10740#pullrequestreview-2216676140).
- To dump graphs, use `-Dgraal.Dump=Truffle:2` system property passed via `JAVA_OPTS` env var.

If you run `env JAVA_OPTS='-Denso.compiler.dumpIr=true' enso --run tmp.enso` where `tmp.enso` is, e.g.:
```
from Standard.Base import all
main = 42
```
You will then have something like:
```
$ ls ir-dumps
Standard.Base.Data.Filter_Condition.dot     Standard.Base.Data.Time.dot              Standard.Base.System.Advanced.dot       Standard.Base.Warning.dot
Standard.Base.Data.Locale.dot               Standard.Base.Enso_Cloud.Enso_File.dot   Standard.Base.System.File.Advanced.dot  tmp.dot
Standard.Base.Data.Numeric.dot              Standard.Base.Errors.dot                 Standard.Base.System.File.dot
Standard.Base.Data.Numeric.Internal.dot     Standard.Base.Network.HTTP.Internal.dot  Standard.Base.System.File.Generic.dot
Standard.Base.Data.Text.Regex.Internal.dot  Standard.Base.Runtime.dot                Standard.Base.System.Internal.dot
```
You can then visualize any of these with `dot -Tsvg -O ir-dumps/tmp.dot`.

An example how that could look like is
![image.svg](https://github.com/user-attachments/assets/26ab8415-72cf-46da-bc63-f475e9fa628e)

distribution/bin/enso

@@ -1,12 +1,5 @@
 COMP_PATH=$(dirname "$0")/../component
 
-EXTRA_OPTS="-Dgraal.PrintGraph=Network"
-for opt in "$@"; do
-  if [ "$opt" = "--dump-graphs" ]; then
-    EXTRA_OPTS="$EXTRA_OPTS -Dgraal.Dump=Truffle:1"
-  fi
-done
-
 JAVA_OPTS="--add-opens=java.base/java.nio=ALL-UNNAMED $JAVA_OPTS"
-exec java --module-path $COMP_PATH $EXTRA_OPTS $JAVA_OPTS -m org.enso.runtime/org.enso.EngineRunnerBootLoader "$@"
+exec java --module-path $COMP_PATH $JAVA_OPTS -m org.enso.runtime/org.enso.EngineRunnerBootLoader "$@"
 exit

distribution/bin/enso.bat

@@ -1,11 +1,5 @@
 @echo off
 set comp-dir=%~dp0\..\component
-set EXTRA_OPTS=-Dgraal.PrintGraph=Network
-FOR %%A in (%*) DO (
-if /I %%A==--dump-graphs (
-set EXTRA_OPTS=%EXTRA_OPTS% -Dgraal.Dump=Truffle:1
-)
-)
 set JAVA_OPTS=%JAVA_OPTS% --add-opens=java.base/java.nio=ALL-UNNAMED
-java --module-path %comp-dir% -Dpolyglot.compiler.IterativePartialEscape=true %EXTRA_OPTS% %JAVA_OPTS% -m org.enso.runtime/org.enso.EngineRunnerBootLoader %*
+java --module-path %comp-dir% -Dpolyglot.compiler.IterativePartialEscape=true %JAVA_OPTS% -m org.enso.runtime/org.enso.EngineRunnerBootLoader %*
 exit /B %errorlevel%

docs/runtime-guide.md

@@ -200,17 +200,16 @@ design choices. Here's a list with some explanations:
 7. **Microbenchmarks**: There are some microbenchmarks for tiny Enso programs
    for basic language constructs. They are located in
    [this directory](https://github.com/enso-org/enso/tree/develop/engine/runtime/src/bench).
-   They can be run through `sbt runtime/bench`. Each run will generate (or
+   They can be run through `sbt runtime-benchmarks/bench`. Each run will
-   append to) the `bench-report.xml` file. It will also fail the benchmark suite
+   generate (or append to) the `bench-report.xml` file. See
-   if any benchmark is more than 20% slower than the fastest recorded run. Don't
+   [Benchmarks](infrastructure/benchmarks.md) for more information about the
-   use your computer when running these. It is also worth noting that these can
+   benchmarking infrastructure.
-   be run through the `withDebug` utility, which allows you to test truffle
-   compilations (and e.g. watch the graphs in IGV with
    [Enso Language Support](../tools/enso4igv/README.md)).
 8. **Tests**: There are scalatests that comprehensively test all of the language
-   semantics and compiler passes. These are run with `sbt runtime/test`. For
+   semantics and compiler passes. These are run with
-   newer functionalities, we prefer adding tests to the `Tests` project in the
+   `sbt runtime-integration-tests/test`. For newer functionalities, we prefer
-   standard library test. At this point, Enso is mature enough to self-test.
+   adding tests to the `Tests` project in the standard library test. At this
+   point, Enso is mature enough to self-test.
 
 ### Language Server
 

docs/runtime/README.md

@@ -50,5 +50,6 @@ broken up as follows:
 - [**Searcher:**](./searcher.md) An explanation of how the searcher works.
 - [**Instruments:**](./instruments.md) An explanation of how we compile Truffle
   instrumentation.
+- [**Compiler IR:**](./compiler-ir.md) An explanation of the Enso compiler IR.
 - [**IR Caching:**](./ir-caching.md) An explanation of how we cache the compiler
   IR to reduce startup times.

docs/runtime/compiler-ir.md

@@ -0,0 +1,23 @@
+# Enso Compiler IR
+
+Enso IR, currently implemented in Scala, with base class
+`org.enso.compiler.core.IR`, is created from the output of the native
+[parser](../parser/README.md). The IR is an immutable annotated AST subjected to
+multiple passes. Every pass is a class implementing the
+`org.enso.compiler.pass.IRPass` interface.
+
+See [Runtime roadmap - static analysis](../runtime-roadmap.md#static-analysis)
+for future goals.
+
+## Visualization
+
+The IR can be visualized using `-Denso.compiler.dumpIr` system property. This
+will output a `.dot` file in [GraphViz](www.graphviz.org) format in the
+`ir-dumps` directory for each IR in the program. The _dot_ file format is a
+minimal textual format, that can be converted to a graphical representation
+using the `dot` command from the GraphViz package. For example, on Ubuntu,
+install `dot` with `sudo apt install graphviz`. Then, convert the `.dot` file to
+a `.svg` image with `dot -Tsvg -o <output>.svg <input>.dot`. An example is:
+![image.svg](https://github.com/user-attachments/assets/26ab8415-72cf-46da-bc63-f475e9fa628e)
+
+See `org.enso.compiler.dump.IRDumper`.

engine/runner/src/main/java/org/enso/runner/Main.java

@@ -12,6 +12,7 @@ import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Collections;
 import java.util.HashMap;
+import java.util.List;
 import java.util.concurrent.TimeUnit;
 import java.util.function.Function;
 import java.util.stream.Collectors;
@@ -60,7 +61,6 @@ public class Main {
   private static final String JVM_OPTION = "jvm";
   private static final String RUN_OPTION = "run";
   private static final String INSPECT_OPTION = "inspect";
-  private static final String DUMP_GRAPHS_OPTION = "dump-graphs";
   private static final String HELP_OPTION = "help";
   private static final String NEW_OPTION = "new";
   private static final String PROJECT_NAME_OPTION = "new-project-name";
@@ -136,11 +136,6 @@ public class Main {
             .longOpt(INSPECT_OPTION)
             .desc("Start the Chrome inspector when --run is used.")
             .build();
-    var dumpGraphs =
-        cliOptionBuilder()
-            .longOpt(DUMP_GRAPHS_OPTION)
-            .desc("Dumps IGV graphs when --run is used.")
-            .build();
     var docs =
         cliOptionBuilder()
             .longOpt(DOCS_OPTION)
@@ -461,7 +456,6 @@ public class Main {
         .addOption(jvm)
         .addOption(run)
         .addOption(inspect)
-        .addOption(dumpGraphs)
         .addOption(docs)
         .addOption(preinstall)
         .addOption(newOpt)
@@ -506,11 +500,7 @@ public class Main {
     return options;
   }
 
-  /**
+  /** Prints the help message to the standard output. */
-   * Prints the help message to the standard output.
-   *
-   * @param options object representing the CLI syntax
-   */
   private static void printHelp() {
     new HelpFormatter().printHelp(LanguageInfo.ID, CLI_OPTIONS);
   }
@@ -660,13 +650,12 @@ public class Main {
    *     ignored.
    * @param enableStaticAnalysis whether or not static type checking should be enabled
    * @param inspect shall inspect option be enabled
-   * @param dump shall graphs be sent to the IGV
    * @param executionEnvironment name of the execution environment to use during execution or {@code
    *     null}
    */
   private void handleRun(
       String path,
-      java.util.List<String> additionalArgs,
+      List<String> additionalArgs,
       String projectPath,
       Level logLevel,
       boolean logMasking,
@@ -676,7 +665,6 @@ public class Main {
       boolean enableStaticAnalysis,
       boolean enableDebugServer,
       boolean inspect,
-      boolean dump,
       String executionEnvironment,
       int warningsLimit)
       throws IOException {
@@ -688,10 +676,6 @@ public class Main {
     var file = fileAndProject._2();
     var projectRoot = fileAndProject._3();
     var options = new HashMap<String, String>();
-    if (dump) {
-      options.put("engine.TraceCompilation", "true");
-      options.put("engine.MultiTier", "false");
-    }
 
     var factory =
         ContextFactory.create()
@@ -1104,7 +1088,6 @@ public class Main {
           line.hasOption(ENABLE_STATIC_ANALYSIS_OPTION),
           line.hasOption(REPL_OPTION),
           line.hasOption(INSPECT_OPTION),
-          line.hasOption(DUMP_GRAPHS_OPTION),
           line.getOptionValue(EXECUTION_ENVIRONMENT_OPTION),
           scala.Option.apply(line.getOptionValue(WARNINGS_LIMIT))
               .map(Integer::parseInt)

engine/runtime-compiler/src/main/java/org/enso/compiler/dump/IRDumper.java

@@ -56,6 +56,9 @@ public class IRDumper {
   /** Whether to include some pass data in the GraphViz file. */
   private static final boolean INCLUDE_PASS_DATA = true;
 
+  public static final String DEFAULT_DUMP_DIR = "ir-dumps";
+  public static final String SYSTEM_PROP = "enso.compiler.dumpIr";
+
   private final OutputStream out;
   private final Set<GraphVizNode> nodes = new HashSet<>();
   private final Set<GraphVizEdge> edges = new HashSet<>();
@@ -219,6 +222,16 @@ public class IRDumper {
                 .addLabelLine("name: " + builtinAnnotation.name());
         addNode(bldr.build());
       }
+      case org.enso.compiler.core.ir.Type.Ascription ascription -> {
+        var ascriptionNode = GraphVizNode.Builder.fromIr(ascription).build();
+        addNode(ascriptionNode);
+        var typed = ascription.typed();
+        createIRGraph(typed);
+        createEdge(ascription, typed, "typed");
+        var signature = ascription.signature();
+        createIRGraph(signature);
+        createEdge(ascription, signature, "signature");
+      }
       default -> throw unimpl(definitionIr);
     }
   }
@@ -565,6 +578,10 @@ public class IRDumper {
                         "  - ModuleMethod(" + method.name() + ")");
                     case BindingsMap.PolyglotSymbol polySym -> bldr.addLabelLine(
                         "  - PolyglotSymbol(" + polySym.name() + ")");
+                    case BindingsMap.ExtensionMethod extensionMethod -> bldr.addLabelLine(
+                        "  - ExtensionMethod(" + extensionMethod.name() + ")");
+                    case BindingsMap.ConversionMethod conversionMethod -> bldr.addLabelLine(
+                        "  - ConversionMethod(" + conversionMethod.name() + ")");
                     default -> throw unimpl(entity);
                   }
                 }

engine/runtime-compiler/src/main/java/org/enso/compiler/dump/IRDumperPass.java

@@ -0,0 +1,75 @@
+package org.enso.compiler.dump;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.UUID;
+import org.enso.compiler.context.InlineContext;
+import org.enso.compiler.context.ModuleContext;
+import org.enso.compiler.core.IR;
+import org.enso.compiler.core.ir.Expression;
+import org.enso.compiler.core.ir.Module;
+import org.enso.compiler.pass.IRPass;
+import scala.collection.immutable.Seq;
+
+/** A pass that just dumps IR to the local {@code ir-dumps} directory. See {@link IRDumper}. */
+public class IRDumperPass implements IRPass {
+  public static final IRDumperPass INSTANCE = new IRDumperPass();
+  private UUID uuid;
+
+  private IRDumperPass() {}
+
+  @Override
+  public UUID key() {
+    return uuid;
+  }
+
+  @Override
+  public void org$enso$compiler$pass$IRPass$_setter_$key_$eq(UUID v) {
+    this.uuid = v;
+  }
+
+  @Override
+  public Seq<IRPass> precursorPasses() {
+    return nil();
+  }
+
+  @Override
+  public Seq<IRPass> invalidatedPasses() {
+    return nil();
+  }
+
+  @Override
+  public Module runModule(Module ir, ModuleContext moduleContext) {
+    var irDumpsDir = Path.of(IRDumper.DEFAULT_DUMP_DIR);
+    if (!irDumpsDir.toFile().exists()) {
+      try {
+        Files.createDirectory(irDumpsDir);
+      } catch (IOException e) {
+        throw new IllegalStateException(e);
+      }
+    }
+    var modName = moduleContext.getName().toString();
+    var irPath = irDumpsDir.resolve(modName + ".dot");
+    var irDumper = IRDumper.fromPath(irPath);
+    irDumper.dump(ir);
+    System.out.println("IR dumped to " + irPath);
+    return ir;
+  }
+
+  @Override
+  public Expression runExpression(Expression ir, InlineContext inlineContext) {
+    return ir;
+  }
+
+  @Override
+  public <T extends IR> T updateMetadataInDuplicate(T sourceIr, T copyOfIr) {
+    return IRPass.super.updateMetadataInDuplicate(sourceIr, copyOfIr);
+  }
+
+  @SuppressWarnings("unchecked")
+  private static scala.collection.immutable.List<IRPass> nil() {
+    Object obj = scala.collection.immutable.Nil$.MODULE$;
+    return (scala.collection.immutable.List<IRPass>) obj;
+  }
+}

engine/runtime-compiler/src/main/scala/org/enso/compiler/Passes.scala

@@ -1,6 +1,7 @@
 package org.enso.compiler
 
 import org.enso.compiler.data.CompilerConfig
+import org.enso.compiler.dump.IRDumperPass
 import org.enso.compiler.pass.PassConfiguration._
 import org.enso.compiler.pass.analyse._
 import org.enso.compiler.pass.analyse.types.TypeInference
@@ -50,6 +51,10 @@ class Passes(config: CompilerConfig) {
               PrivateModuleAnalysis.INSTANCE,
               PrivateConstructorAnalysis.INSTANCE
             )
+          } else List()) ++ (if (config.dumpIrs) {
+                               List(
+                                 IRDumperPass.INSTANCE
+                               )
                              } else List())
     ++ List(
       ShadowedPatternFields,

engine/runtime-compiler/src/main/scala/org/enso/compiler/data/CompilerConfig.scala

@@ -9,6 +9,7 @@ import java.io.PrintStream
   * @param warningsEnabled whether or not warnings are enabled
   * @param privateCheckEnabled whether or not private keyword is enabled
   * @param staticTypeInferenceEnabled whether or not type inference is enabled
+  * @param dumpIrs whether or not to dump IRs. See [[org.enso.compiler.dump.IRDumper]].
   * @param isStrictErrors if true, presence of any Error in IR will result in an exception
   * @oaram isLintingDisabled if true, compilation should not run any linting passes
   * @param outputRedirect redirection of the output of warnings and errors of compiler
@@ -18,6 +19,7 @@ case class CompilerConfig(
   warningsEnabled: Boolean            = true,
   privateCheckEnabled: Boolean        = true,
   staticTypeInferenceEnabled: Boolean = false,
+  dumpIrs: Boolean                    = false,
   isStrictErrors: Boolean             = false,
   isLintingDisabled: Boolean          = false,
   outputRedirect: Option[PrintStream] = None

engine/runtime-integration-tests/src/test/java/org/enso/compiler/TypeInferenceTest.java

@@ -1157,7 +1157,8 @@ public class TypeInferenceTest extends CompilerTest {
 
     Module rawModule = parse(src.getCharacters());
 
-    var compilerConfig = new CompilerConfig(false, true, true, true, true, false, Option.empty());
+    var compilerConfig =
+        new CompilerConfig(false, true, true, true, false, true, false, Option.empty());
     var passes = new Passes(compilerConfig);
     @SuppressWarnings("unchecked")
     var passConfig =

engine/runtime-integration-tests/src/test/java/org/enso/compiler/dump/test/IRDumpTest.java

@@ -0,0 +1,44 @@
+package org.enso.compiler.dump.test;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.is;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.file.Path;
+import org.enso.compiler.dump.IRDumper;
+import org.enso.test.utils.ContextUtils;
+import org.enso.test.utils.ProjectUtils;
+import org.junit.Test;
+
+public class IRDumpTest {
+  @Test
+  public void testIrDump() {
+    var irDumpsDir = Path.of(IRDumper.DEFAULT_DUMP_DIR);
+    var out = new ByteArrayOutputStream();
+    System.setProperty(IRDumper.SYSTEM_PROP, "true");
+    try (var ctx = ContextUtils.defaultContextBuilder().out(out).build()) {
+      // Dumping is done in the compiler, so it is enough just to compile the module
+      var moduleIr =
+          ContextUtils.compileModule(ctx, """
+          main = 42
+          """, "MyMainModule");
+      assertThat(
+          "ir-dumps directory was generated in current working directory",
+          irDumpsDir.toFile().exists(),
+          is(true));
+      var mainModDump = irDumpsDir.resolve("MyMainModule.dot");
+      assertThat(
+          "MyMainModule.dot file was generated in ir-dumps directory",
+          mainModDump.toFile().exists(),
+          is(true));
+    } finally {
+      System.setProperty(IRDumper.SYSTEM_PROP, "false");
+      try {
+        ProjectUtils.deleteRecursively(irDumpsDir);
+      } catch (IOException e) {
+        // Ignore. The ir-dumps directory should be deleted eventually.
+      }
+    }
+  }
+}

engine/runtime/src/main/java/org/enso/interpreter/EnsoLanguage.java

@@ -237,6 +237,7 @@ public final class EnsoLanguage extends TruffleLanguage<EnsoContext> {
               false,
               true,
               false,
+              false,
               true,
               false,
               scala.Option.apply(new PrintStream(outputRedirect)));

engine/runtime/src/main/java/org/enso/interpreter/runtime/EnsoContext.java

@@ -42,6 +42,7 @@ import org.enso.common.LanguageInfo;
 import org.enso.common.RuntimeOptions;
 import org.enso.compiler.Compiler;
 import org.enso.compiler.data.CompilerConfig;
+import org.enso.compiler.dump.IRDumper;
 import org.enso.distribution.DistributionManager;
 import org.enso.distribution.locking.LockManager;
 import org.enso.editions.LibraryName;
@@ -144,12 +145,14 @@ public final class EnsoContext {
     this.assertionsEnabled = shouldAssertionsBeEnabled();
     this.shouldWaitForPendingSerializationJobs =
         getOption(RuntimeOptions.WAIT_FOR_PENDING_SERIALIZATION_JOBS_KEY);
+    var dumpIrs = Boolean.parseBoolean(System.getProperty(IRDumper.SYSTEM_PROP));
     this.compilerConfig =
         new CompilerConfig(
             isParallelismEnabled,
             true,
             !isPrivateCheckDisabled,
             isStaticTypeAnalysisEnabled,
+            dumpIrs,
             getOption(RuntimeOptions.STRICT_ERRORS_KEY),
             getOption(RuntimeOptions.DISABLE_LINTING_KEY),
             scala.Option.empty());

tools/enso4igv/IGV.md

@@ -56,12 +56,16 @@ to _finish_ the installation.
 
 Build an instance of the Enso runtime engine (see
 [Running Enso](../../docs/CONTRIBUTING.md#running-enso)) using and then launch
-it with special `--dump-graphs` option:
+it the following system properties:
 
 ```bash
-enso$ sbt runEngineDistribution --dump-graphs --run yourprogram.enso
+enso$ env JAVA_OPTS='-Dgraal.Dump=Truffle:2 -Dgraal.PrintGraph=File' ./built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso --run yourprogram.enso
 ```
 
+See
+[Graal system props docs](https://github.com/oracle/graal/blob/master/compiler/docs/Debugging.md#jvmci-and-compiler-specific-options)
+for the description of the `graal` system properties.
+
 When executed on [GraalVM 22.3.1](http://graalvm.org) these options instruct the
 _Graal/Truffle compiler_ to dump files into `graal_dumps/_sometimestamp_`
 directory. Generating these files takes a while - make sure `yourprogram.enso`
@@ -73,10 +77,10 @@ speed_.
 As an example you can download
 [sieve.enso](https://github.com/jtulach/sieve/blob/5b32450da35415322e683bb9769aa45f0d71f1df/enso/sieve.enso)
 which computes hundred thousand of prime numbers repeatedly and measures time of
-each round. Download the file and launch Enso with `--dump-graphs` argument:
+each round. Download the file and launch Enso with:
 
 ```bash
-enso$ ./built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso --dump-graphs --run sieve.enso
+enso$ env JAVA_OPTS='-Dgraal.Dump=Truffle:2 -Dgraal.PrintGraph=File' ./built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso --run yourprogram.enso
 ```
 
 Bunch of files in `graal_dumps/*` subdirectory is going to be generated:

tools/enso4igv/src/main/java/org/enso/tools/enso4igv/EnsoActionProvider.java

@@ -81,11 +81,14 @@ public final class EnsoActionProvider implements ActionProvider {
 
                 var b = ProcessBuilder.getLocal();
                 b.setExecutable(file.getPath());
-                b.setArguments(prepareArguments(script, isGraalVM));
+                b.setArguments(prepareArguments(script));
                 b.setWorkingDirectory(script.getParent());
                 b.setRedirectErrorStream(true);
 
                 var env = b.getEnvironment();
+                if (isGraalVM && isIGVConnected()) {
+                    env.setVariable("JAVA_OPTS", "-Dgraal.Dump=Truffle:2");
+                }
                 var path = env.getVariable("PATH");
                 if (path != null && java != null) {
                     var javaBinDir = FileUtil.toFile(java.getParent());
@@ -133,14 +136,14 @@ public final class EnsoActionProvider implements ActionProvider {
         });
     }
 
-    private static List<String> prepareArguments(File script, boolean isGraalVM) {
+    private static boolean isIGVConnected() {
+        return Modules.getDefault().findCodeNameBase("org.graalvm.visualizer.connection") != null;
+    }
+
+    private static List<String> prepareArguments(File script) {
         var list = new ArrayList<String>();
         list.add("--run");
         list.add(script.getPath());
-        var isIGV = Modules.getDefault().findCodeNameBase("org.graalvm.visualizer.connection") != null;
-        if (isGraalVM && isIGV) {
-            list.add("--dump-graphs");
-        }
         return list;
     }