Add org.enso.compiler.dumpIr system prop (#10740)

Working on compiler IR is a daunting task. I have therefore added a new system property `enso.compiler.dumpIr` that will help with that. It dumps the encountered IRs to `ir-dumps` directory in the [GraphViz](www.graphviz.org) format. More info in updated docs.

Note that all the functionality to dump IRs to `dot` files was already implemented. This PR just adds the command line option and updates docs.

# Important Notes
- `--dump-graphs` cmd line option is removed as per [Jaroslav's request](https://github.com/enso-org/enso/pull/10740#pullrequestreview-2216676140).
- To dump graphs, use `-Dgraal.Dump=Truffle:2` system property passed via `JAVA_OPTS` env var.

If you run `env JAVA_OPTS='-Denso.compiler.dumpIr=true' enso --run tmp.enso` where `tmp.enso` is, e.g.:
```
from Standard.Base import all
main = 42
```
You will then have something like:
```
$ ls ir-dumps
Standard.Base.Data.Filter_Condition.dot     Standard.Base.Data.Time.dot              Standard.Base.System.Advanced.dot       Standard.Base.Warning.dot
Standard.Base.Data.Locale.dot               Standard.Base.Enso_Cloud.Enso_File.dot   Standard.Base.System.File.Advanced.dot  tmp.dot
Standard.Base.Data.Numeric.dot              Standard.Base.Errors.dot                 Standard.Base.System.File.dot
Standard.Base.Data.Numeric.Internal.dot     Standard.Base.Network.HTTP.Internal.dot  Standard.Base.System.File.Generic.dot
Standard.Base.Data.Text.Regex.Internal.dot  Standard.Base.Runtime.dot                Standard.Base.System.Internal.dot
```
You can then visualize any of these with `dot -Tsvg -O ir-dumps/tmp.dot`.

An example how that could look like is
![image.svg](https://github.com/user-attachments/assets/26ab8415-72cf-46da-bc63-f475e9fa628e)
This commit is contained in:
Pavel Marek 2024-08-06 14:00:27 +02:00 committed by GitHub
parent f97dd0506a
commit f0de43a970
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
16 changed files with 204 additions and 56 deletions

View File

@ -1,12 +1,5 @@
COMP_PATH=$(dirname "$0")/../component
EXTRA_OPTS="-Dgraal.PrintGraph=Network"
for opt in "$@"; do
if [ "$opt" = "--dump-graphs" ]; then
EXTRA_OPTS="$EXTRA_OPTS -Dgraal.Dump=Truffle:1"
fi
done
JAVA_OPTS="--add-opens=java.base/java.nio=ALL-UNNAMED $JAVA_OPTS"
exec java --module-path $COMP_PATH $EXTRA_OPTS $JAVA_OPTS -m org.enso.runtime/org.enso.EngineRunnerBootLoader "$@"
exec java --module-path $COMP_PATH $JAVA_OPTS -m org.enso.runtime/org.enso.EngineRunnerBootLoader "$@"
exit

View File

@ -1,11 +1,5 @@
@echo off
set comp-dir=%~dp0\..\component
set EXTRA_OPTS=-Dgraal.PrintGraph=Network
FOR %%A in (%*) DO (
if /I %%A==--dump-graphs (
set EXTRA_OPTS=%EXTRA_OPTS% -Dgraal.Dump=Truffle:1
)
)
set JAVA_OPTS=%JAVA_OPTS% --add-opens=java.base/java.nio=ALL-UNNAMED
java --module-path %comp-dir% -Dpolyglot.compiler.IterativePartialEscape=true %EXTRA_OPTS% %JAVA_OPTS% -m org.enso.runtime/org.enso.EngineRunnerBootLoader %*
java --module-path %comp-dir% -Dpolyglot.compiler.IterativePartialEscape=true %JAVA_OPTS% -m org.enso.runtime/org.enso.EngineRunnerBootLoader %*
exit /B %errorlevel%

View File

@ -200,17 +200,16 @@ design choices. Here's a list with some explanations:
7. **Microbenchmarks**: There are some microbenchmarks for tiny Enso programs
for basic language constructs. They are located in
[this directory](https://github.com/enso-org/enso/tree/develop/engine/runtime/src/bench).
They can be run through `sbt runtime/bench`. Each run will generate (or
append to) the `bench-report.xml` file. It will also fail the benchmark suite
if any benchmark is more than 20% slower than the fastest recorded run. Don't
use your computer when running these. It is also worth noting that these can
be run through the `withDebug` utility, which allows you to test truffle
compilations (and e.g. watch the graphs in IGV with
They can be run through `sbt runtime-benchmarks/bench`. Each run will
generate (or append to) the `bench-report.xml` file. See
[Benchmarks](infrastructure/benchmarks.md) for more information about the
benchmarking infrastructure.
[Enso Language Support](../tools/enso4igv/README.md)).
8. **Tests**: There are scalatests that comprehensively test all of the language
semantics and compiler passes. These are run with `sbt runtime/test`. For
newer functionalities, we prefer adding tests to the `Tests` project in the
standard library test. At this point, Enso is mature enough to self-test.
semantics and compiler passes. These are run with
`sbt runtime-integration-tests/test`. For newer functionalities, we prefer
adding tests to the `Tests` project in the standard library test. At this
point, Enso is mature enough to self-test.
### Language Server

View File

@ -50,5 +50,6 @@ broken up as follows:
- [**Searcher:**](./searcher.md) An explanation of how the searcher works.
- [**Instruments:**](./instruments.md) An explanation of how we compile Truffle
instrumentation.
- [**Compiler IR:**](./compiler-ir.md) An explanation of the Enso compiler IR.
- [**IR Caching:**](./ir-caching.md) An explanation of how we cache the compiler
IR to reduce startup times.

View File

@ -0,0 +1,23 @@
# Enso Compiler IR
Enso IR, currently implemented in Scala, with base class
`org.enso.compiler.core.IR`, is created from the output of the native
[parser](../parser/README.md). The IR is an immutable annotated AST subjected to
multiple passes. Every pass is a class implementing the
`org.enso.compiler.pass.IRPass` interface.
See [Runtime roadmap - static analysis](../runtime-roadmap.md#static-analysis)
for future goals.
## Visualization
The IR can be visualized using `-Denso.compiler.dumpIr` system property. This
will output a `.dot` file in [GraphViz](www.graphviz.org) format in the
`ir-dumps` directory for each IR in the program. The _dot_ file format is a
minimal textual format, that can be converted to a graphical representation
using the `dot` command from the GraphViz package. For example, on Ubuntu,
install `dot` with `sudo apt install graphviz`. Then, convert the `.dot` file to
a `.svg` image with `dot -Tsvg -o <output>.svg <input>.dot`. An example is:
![image.svg](https://github.com/user-attachments/assets/26ab8415-72cf-46da-bc63-f475e9fa628e)
See `org.enso.compiler.dump.IRDumper`.

View File

@ -12,6 +12,7 @@ import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.concurrent.TimeUnit;
import java.util.function.Function;
import java.util.stream.Collectors;
@ -60,7 +61,6 @@ public class Main {
private static final String JVM_OPTION = "jvm";
private static final String RUN_OPTION = "run";
private static final String INSPECT_OPTION = "inspect";
private static final String DUMP_GRAPHS_OPTION = "dump-graphs";
private static final String HELP_OPTION = "help";
private static final String NEW_OPTION = "new";
private static final String PROJECT_NAME_OPTION = "new-project-name";
@ -136,11 +136,6 @@ public class Main {
.longOpt(INSPECT_OPTION)
.desc("Start the Chrome inspector when --run is used.")
.build();
var dumpGraphs =
cliOptionBuilder()
.longOpt(DUMP_GRAPHS_OPTION)
.desc("Dumps IGV graphs when --run is used.")
.build();
var docs =
cliOptionBuilder()
.longOpt(DOCS_OPTION)
@ -461,7 +456,6 @@ public class Main {
.addOption(jvm)
.addOption(run)
.addOption(inspect)
.addOption(dumpGraphs)
.addOption(docs)
.addOption(preinstall)
.addOption(newOpt)
@ -506,11 +500,7 @@ public class Main {
return options;
}
/**
* Prints the help message to the standard output.
*
* @param options object representing the CLI syntax
*/
/** Prints the help message to the standard output. */
private static void printHelp() {
new HelpFormatter().printHelp(LanguageInfo.ID, CLI_OPTIONS);
}
@ -660,13 +650,12 @@ public class Main {
* ignored.
* @param enableStaticAnalysis whether or not static type checking should be enabled
* @param inspect shall inspect option be enabled
* @param dump shall graphs be sent to the IGV
* @param executionEnvironment name of the execution environment to use during execution or {@code
* null}
*/
private void handleRun(
String path,
java.util.List<String> additionalArgs,
List<String> additionalArgs,
String projectPath,
Level logLevel,
boolean logMasking,
@ -676,7 +665,6 @@ public class Main {
boolean enableStaticAnalysis,
boolean enableDebugServer,
boolean inspect,
boolean dump,
String executionEnvironment,
int warningsLimit)
throws IOException {
@ -688,10 +676,6 @@ public class Main {
var file = fileAndProject._2();
var projectRoot = fileAndProject._3();
var options = new HashMap<String, String>();
if (dump) {
options.put("engine.TraceCompilation", "true");
options.put("engine.MultiTier", "false");
}
var factory =
ContextFactory.create()
@ -1104,7 +1088,6 @@ public class Main {
line.hasOption(ENABLE_STATIC_ANALYSIS_OPTION),
line.hasOption(REPL_OPTION),
line.hasOption(INSPECT_OPTION),
line.hasOption(DUMP_GRAPHS_OPTION),
line.getOptionValue(EXECUTION_ENVIRONMENT_OPTION),
scala.Option.apply(line.getOptionValue(WARNINGS_LIMIT))
.map(Integer::parseInt)

View File

@ -56,6 +56,9 @@ public class IRDumper {
/** Whether to include some pass data in the GraphViz file. */
private static final boolean INCLUDE_PASS_DATA = true;
public static final String DEFAULT_DUMP_DIR = "ir-dumps";
public static final String SYSTEM_PROP = "enso.compiler.dumpIr";
private final OutputStream out;
private final Set<GraphVizNode> nodes = new HashSet<>();
private final Set<GraphVizEdge> edges = new HashSet<>();
@ -219,6 +222,16 @@ public class IRDumper {
.addLabelLine("name: " + builtinAnnotation.name());
addNode(bldr.build());
}
case org.enso.compiler.core.ir.Type.Ascription ascription -> {
var ascriptionNode = GraphVizNode.Builder.fromIr(ascription).build();
addNode(ascriptionNode);
var typed = ascription.typed();
createIRGraph(typed);
createEdge(ascription, typed, "typed");
var signature = ascription.signature();
createIRGraph(signature);
createEdge(ascription, signature, "signature");
}
default -> throw unimpl(definitionIr);
}
}
@ -565,6 +578,10 @@ public class IRDumper {
" - ModuleMethod(" + method.name() + ")");
case BindingsMap.PolyglotSymbol polySym -> bldr.addLabelLine(
" - PolyglotSymbol(" + polySym.name() + ")");
case BindingsMap.ExtensionMethod extensionMethod -> bldr.addLabelLine(
" - ExtensionMethod(" + extensionMethod.name() + ")");
case BindingsMap.ConversionMethod conversionMethod -> bldr.addLabelLine(
" - ConversionMethod(" + conversionMethod.name() + ")");
default -> throw unimpl(entity);
}
}

View File

@ -0,0 +1,75 @@
package org.enso.compiler.dump;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.UUID;
import org.enso.compiler.context.InlineContext;
import org.enso.compiler.context.ModuleContext;
import org.enso.compiler.core.IR;
import org.enso.compiler.core.ir.Expression;
import org.enso.compiler.core.ir.Module;
import org.enso.compiler.pass.IRPass;
import scala.collection.immutable.Seq;
/** A pass that just dumps IR to the local {@code ir-dumps} directory. See {@link IRDumper}. */
public class IRDumperPass implements IRPass {
public static final IRDumperPass INSTANCE = new IRDumperPass();
private UUID uuid;
private IRDumperPass() {}
@Override
public UUID key() {
return uuid;
}
@Override
public void org$enso$compiler$pass$IRPass$_setter_$key_$eq(UUID v) {
this.uuid = v;
}
@Override
public Seq<IRPass> precursorPasses() {
return nil();
}
@Override
public Seq<IRPass> invalidatedPasses() {
return nil();
}
@Override
public Module runModule(Module ir, ModuleContext moduleContext) {
var irDumpsDir = Path.of(IRDumper.DEFAULT_DUMP_DIR);
if (!irDumpsDir.toFile().exists()) {
try {
Files.createDirectory(irDumpsDir);
} catch (IOException e) {
throw new IllegalStateException(e);
}
}
var modName = moduleContext.getName().toString();
var irPath = irDumpsDir.resolve(modName + ".dot");
var irDumper = IRDumper.fromPath(irPath);
irDumper.dump(ir);
System.out.println("IR dumped to " + irPath);
return ir;
}
@Override
public Expression runExpression(Expression ir, InlineContext inlineContext) {
return ir;
}
@Override
public <T extends IR> T updateMetadataInDuplicate(T sourceIr, T copyOfIr) {
return IRPass.super.updateMetadataInDuplicate(sourceIr, copyOfIr);
}
@SuppressWarnings("unchecked")
private static scala.collection.immutable.List<IRPass> nil() {
Object obj = scala.collection.immutable.Nil$.MODULE$;
return (scala.collection.immutable.List<IRPass>) obj;
}
}

View File

@ -1,6 +1,7 @@
package org.enso.compiler
import org.enso.compiler.data.CompilerConfig
import org.enso.compiler.dump.IRDumperPass
import org.enso.compiler.pass.PassConfiguration._
import org.enso.compiler.pass.analyse._
import org.enso.compiler.pass.analyse.types.TypeInference
@ -50,7 +51,11 @@ class Passes(config: CompilerConfig) {
PrivateModuleAnalysis.INSTANCE,
PrivateConstructorAnalysis.INSTANCE
)
} else List())
} else List()) ++ (if (config.dumpIrs) {
List(
IRDumperPass.INSTANCE
)
} else List())
++ List(
ShadowedPatternFields,
UnreachableMatchBranches,

View File

@ -9,6 +9,7 @@ import java.io.PrintStream
* @param warningsEnabled whether or not warnings are enabled
* @param privateCheckEnabled whether or not private keyword is enabled
* @param staticTypeInferenceEnabled whether or not type inference is enabled
* @param dumpIrs whether or not to dump IRs. See [[org.enso.compiler.dump.IRDumper]].
* @param isStrictErrors if true, presence of any Error in IR will result in an exception
* @oaram isLintingDisabled if true, compilation should not run any linting passes
* @param outputRedirect redirection of the output of warnings and errors of compiler
@ -18,6 +19,7 @@ case class CompilerConfig(
warningsEnabled: Boolean = true,
privateCheckEnabled: Boolean = true,
staticTypeInferenceEnabled: Boolean = false,
dumpIrs: Boolean = false,
isStrictErrors: Boolean = false,
isLintingDisabled: Boolean = false,
outputRedirect: Option[PrintStream] = None

View File

@ -1157,7 +1157,8 @@ public class TypeInferenceTest extends CompilerTest {
Module rawModule = parse(src.getCharacters());
var compilerConfig = new CompilerConfig(false, true, true, true, true, false, Option.empty());
var compilerConfig =
new CompilerConfig(false, true, true, true, false, true, false, Option.empty());
var passes = new Passes(compilerConfig);
@SuppressWarnings("unchecked")
var passConfig =

View File

@ -0,0 +1,44 @@
package org.enso.compiler.dump.test;
import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.is;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.nio.file.Path;
import org.enso.compiler.dump.IRDumper;
import org.enso.test.utils.ContextUtils;
import org.enso.test.utils.ProjectUtils;
import org.junit.Test;
public class IRDumpTest {
@Test
public void testIrDump() {
var irDumpsDir = Path.of(IRDumper.DEFAULT_DUMP_DIR);
var out = new ByteArrayOutputStream();
System.setProperty(IRDumper.SYSTEM_PROP, "true");
try (var ctx = ContextUtils.defaultContextBuilder().out(out).build()) {
// Dumping is done in the compiler, so it is enough just to compile the module
var moduleIr =
ContextUtils.compileModule(ctx, """
main = 42
""", "MyMainModule");
assertThat(
"ir-dumps directory was generated in current working directory",
irDumpsDir.toFile().exists(),
is(true));
var mainModDump = irDumpsDir.resolve("MyMainModule.dot");
assertThat(
"MyMainModule.dot file was generated in ir-dumps directory",
mainModDump.toFile().exists(),
is(true));
} finally {
System.setProperty(IRDumper.SYSTEM_PROP, "false");
try {
ProjectUtils.deleteRecursively(irDumpsDir);
} catch (IOException e) {
// Ignore. The ir-dumps directory should be deleted eventually.
}
}
}
}

View File

@ -237,6 +237,7 @@ public final class EnsoLanguage extends TruffleLanguage<EnsoContext> {
false,
true,
false,
false,
true,
false,
scala.Option.apply(new PrintStream(outputRedirect)));

View File

@ -42,6 +42,7 @@ import org.enso.common.LanguageInfo;
import org.enso.common.RuntimeOptions;
import org.enso.compiler.Compiler;
import org.enso.compiler.data.CompilerConfig;
import org.enso.compiler.dump.IRDumper;
import org.enso.distribution.DistributionManager;
import org.enso.distribution.locking.LockManager;
import org.enso.editions.LibraryName;
@ -144,12 +145,14 @@ public final class EnsoContext {
this.assertionsEnabled = shouldAssertionsBeEnabled();
this.shouldWaitForPendingSerializationJobs =
getOption(RuntimeOptions.WAIT_FOR_PENDING_SERIALIZATION_JOBS_KEY);
var dumpIrs = Boolean.parseBoolean(System.getProperty(IRDumper.SYSTEM_PROP));
this.compilerConfig =
new CompilerConfig(
isParallelismEnabled,
true,
!isPrivateCheckDisabled,
isStaticTypeAnalysisEnabled,
dumpIrs,
getOption(RuntimeOptions.STRICT_ERRORS_KEY),
getOption(RuntimeOptions.DISABLE_LINTING_KEY),
scala.Option.empty());

View File

@ -56,12 +56,16 @@ to _finish_ the installation.
Build an instance of the Enso runtime engine (see
[Running Enso](../../docs/CONTRIBUTING.md#running-enso)) using and then launch
it with special `--dump-graphs` option:
it the following system properties:
```bash
enso$ sbt runEngineDistribution --dump-graphs --run yourprogram.enso
enso$ env JAVA_OPTS='-Dgraal.Dump=Truffle:2 -Dgraal.PrintGraph=File' ./built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso --run yourprogram.enso
```
See
[Graal system props docs](https://github.com/oracle/graal/blob/master/compiler/docs/Debugging.md#jvmci-and-compiler-specific-options)
for the description of the `graal` system properties.
When executed on [GraalVM 22.3.1](http://graalvm.org) these options instruct the
_Graal/Truffle compiler_ to dump files into `graal_dumps/_sometimestamp_`
directory. Generating these files takes a while - make sure `yourprogram.enso`
@ -73,10 +77,10 @@ speed_.
As an example you can download
[sieve.enso](https://github.com/jtulach/sieve/blob/5b32450da35415322e683bb9769aa45f0d71f1df/enso/sieve.enso)
which computes hundred thousand of prime numbers repeatedly and measures time of
each round. Download the file and launch Enso with `--dump-graphs` argument:
each round. Download the file and launch Enso with:
```bash
enso$ ./built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso --dump-graphs --run sieve.enso
enso$ env JAVA_OPTS='-Dgraal.Dump=Truffle:2 -Dgraal.PrintGraph=File' ./built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso --run yourprogram.enso
```
Bunch of files in `graal_dumps/*` subdirectory is going to be generated:

View File

@ -81,11 +81,14 @@ public final class EnsoActionProvider implements ActionProvider {
var b = ProcessBuilder.getLocal();
b.setExecutable(file.getPath());
b.setArguments(prepareArguments(script, isGraalVM));
b.setArguments(prepareArguments(script));
b.setWorkingDirectory(script.getParent());
b.setRedirectErrorStream(true);
var env = b.getEnvironment();
if (isGraalVM && isIGVConnected()) {
env.setVariable("JAVA_OPTS", "-Dgraal.Dump=Truffle:2");
}
var path = env.getVariable("PATH");
if (path != null && java != null) {
var javaBinDir = FileUtil.toFile(java.getParent());
@ -133,14 +136,14 @@ public final class EnsoActionProvider implements ActionProvider {
});
}
private static List<String> prepareArguments(File script, boolean isGraalVM) {
private static boolean isIGVConnected() {
return Modules.getDefault().findCodeNameBase("org.graalvm.visualizer.connection") != null;
}
private static List<String> prepareArguments(File script) {
var list = new ArrayList<String>();
list.add("--run");
list.add(script.getPath());
var isIGV = Modules.getDefault().findCodeNameBase("org.graalvm.visualizer.connection") != null;
if (isGraalVM && isIGV) {
list.add("--dump-graphs");
}
return list;
}