Always log to console and file (#7825)

* Always log verbose to a file

The change adds an option by default to always log to a file with
verbose log level.
The implementation is a bit tricky because in the most common use-case
we have to always log in verbose mode to a socket and only later apply
the desired log levels. Previously socket appender would respect the
desired log level already before forwarding the log.

If by default we log to a file, verbose mode is simply ignored and does
not override user settings.

To test run `project-manager` with `ENSO_LOGSERVER_APPENDER=console` env
variable. That will output to the console with the default `INFO` level
and `TRACE` log level for the file.

* add docs

* changelog

* Address some PR requests

1. Log INFO level to CONSOLE by default
2. Change runner's default log level from ERROR to WARN

Took a while to figure out why the correct log level wasn't being passed
to the language server, therefore ignoring the (desired) verbose logs
from the log file.

* linter

* 3rd party uses log4j for logging

Getting rid of the warning by adding a log4j over slf4j bridge:
```
ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
```

* legal review update

* Make sure tests use test resources

Having `application.conf` in `src/main/resources` and `test/resources`
does not guarantee that in Tests we will pick up the latter. Instead, by
default it seems to do some kind of merge of different configurations,
which is far from desired.

* Ensure native launcher test log to console only

Logging to console and (temporary) files is problematic for Windows.
The CI also revealed a problem with the native configuration because it
was not possible to modify the launcher via env variables as everything
was initialized during build time.

* Adapt to method changes

* Potentially deal with Windows failures

CHANGELOG.md

@@ -962,6 +962,7 @@
 - [Support runtime checks of intersection types][7769]
 - [Merge `Small_Integer` and `Big_Integer` types][7636]
 - [Inline type ascriptions][7796]
+- [Always persist `TRACE` level logs to a file][7825]
 - [Downloadable VSCode extension][7861]
 - [New `project/status` route for reporting LS state][7801]
 
@@ -1106,6 +1107,7 @@
 [7636]: https://github.com/enso-org/enso/pull/7636
 [7796]: https://github.com/enso-org/enso/pull/7796
 [7801]: https://github.com/enso-org/enso/pull/7801
+[7825]: https://github.com/enso-org/enso/pull/7825
 [7861]: https://github.com/enso-org/enso/pull/7861
 
 # Enso 2.0.0-alpha.18 (2021-10-12)

build.sbt

@@ -882,6 +882,7 @@ lazy val `project-manager` = (project in file("lib/scala/project-manager"))
       case _                  => MergeStrategy.first
     },
     (Test / test) := (Test / test).dependsOn(`engine-runner` / assembly).value,
+    Test / javaOptions += s"-Dconfig.file=${sourceDirectory.value}/test/resources/application.conf",
     rebuildNativeImage := NativeImage
       .buildNativeImage(
         "project-manager",
@@ -1156,6 +1157,7 @@ lazy val `language-server` = (project in file("engine/language-server"))
           .map(_.data)
           .mkString(File.pathSeparator)
       Seq(
+        s"-Dconfig.file=${sourceDirectory.value}/test/resources/application.conf",
         s"-Dtruffle.class.path.append=$runtimeClasspath",
         s"-Duser.dir=${file(".").getCanonicalPath}"
       )
@@ -1758,7 +1760,9 @@ lazy val launcher = project
     (Test / testOnly) := (Test / testOnly)
       .dependsOn(buildNativeImage)
       .dependsOn(LauncherShimsForTest.prepare())
-      .evaluated
+      .evaluated,
+    Test / fork := true,
+    Test / javaOptions += s"-Dconfig.file=${sourceDirectory.value}/test/resources/application.conf"
   )
   .dependsOn(cli)
   .dependsOn(`runtime-version-manager`)
@@ -2194,12 +2198,13 @@ lazy val `std-table` = project
       (Antlr4 / sourceManaged).value / "main" / "antlr4"
     },
     libraryDependencies ++= Seq(
-      "org.graalvm.sdk"     % "graal-sdk"               % graalMavenPackagesVersion % "provided",
+      "org.graalvm.sdk"          % "graal-sdk"               % graalMavenPackagesVersion % "provided",
-      "org.netbeans.api"    % "org-openide-util-lookup" % netbeansApiVersion        % "provided",
+      "org.netbeans.api"         % "org-openide-util-lookup" % netbeansApiVersion        % "provided",
-      "com.univocity"       % "univocity-parsers"       % univocityParsersVersion,
+      "com.univocity"            % "univocity-parsers"       % univocityParsersVersion,
-      "org.apache.poi"      % "poi-ooxml"               % poiOoxmlVersion,
+      "org.apache.poi"           % "poi-ooxml"               % poiOoxmlVersion,
-      "org.apache.xmlbeans" % "xmlbeans"                % xmlbeansVersion,
+      "org.apache.xmlbeans"      % "xmlbeans"                % xmlbeansVersion,
-      "org.antlr"           % "antlr4-runtime"          % antlrVersion
+      "org.antlr"                % "antlr4-runtime"          % antlrVersion,
+      "org.apache.logging.log4j" % "log4j-to-slf4j"          % "2.18.0" // org.apache.poi uses log4j
     ),
     Compile / packageBin := Def.task {
       val result = (Compile / packageBin).value

distribution/lib/Standard/Table/0.0.0-dev/THIRD-PARTY/NOTICE

@@ -51,6 +51,11 @@ The license information can be found along with the copyright notices.
 Copyright notices related to this dependency can be found in the directory `org.apache.logging.log4j.log4j-api-2.18.0`.
 
 
+'log4j-to-slf4j', licensed under the Apache License, Version 2.0, is distributed with the Table.
+The license file can be found at `licenses/APACHE2.0`.
+Copyright notices related to this dependency can be found in the directory `org.apache.logging.log4j.log4j-to-slf4j-2.18.0`.
+
+
 'poi', licensed under the Apache License, Version 2.0, is distributed with the Table.
 The license information can be found along with the copyright notices.
 Copyright notices related to this dependency can be found in the directory `org.apache.poi.poi-5.2.3`.
@@ -70,3 +75,8 @@ Copyright notices related to this dependency can be found in the directory `org.
 The license file can be found at `licenses/APACHE2.0`.
 Copyright notices related to this dependency can be found in the directory `org.apache.xmlbeans.xmlbeans-5.1.1`.
 
+
+'slf4j-api', licensed under the MIT License, is distributed with the Table.
+The license file can be found at `licenses/MIT`.
+Copyright notices related to this dependency can be found in the directory `org.slf4j.slf4j-api-1.7.36`.
+

distribution/lib/Standard/Table/0.0.0-dev/THIRD-PARTY/licenses/MIT

@@ -0,0 +1,6 @@
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+

distribution/lib/Standard/Table/0.0.0-dev/THIRD-PARTY/org.apache.logging.log4j.log4j-to-slf4j-2.18.0/NOTICE

@@ -0,0 +1,8 @@
+
+Apache Log4j to SLF4J Adapter
+Copyright 1999-2022 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (http://www.apache.org/).
+
+

distribution/lib/Standard/Table/0.0.0-dev/THIRD-PARTY/org.apache.logging.log4j.log4j-to-slf4j-2.18.0/NOTICES

@@ -0,0 +1 @@
+this work for additional information regarding copyright ownership.

distribution/lib/Standard/Table/0.0.0-dev/THIRD-PARTY/org.slf4j.slf4j-api-1.7.36/NOTICES

@@ -0,0 +1 @@
+Copyright (c) 2004-2011 QOS.ch

docs/infrastructure/logging.md

@@ -29,6 +29,7 @@ involving the centralized logging service.
   - [Setting Up Logging](#setting-up-logging)
   - [Log Masking](#log-masking)
   - [Logging in Tests](#logging-in-tests)
+  - [Logging to file](#logging-to-file)
 
 <!-- /MarkdownTOC -->
 
@@ -346,3 +347,15 @@ information even if the object implements custom interface for masked logging.
 The Logging Service provides a helper function `TestLogger.gatherLogs` that will
 execute the closure and collect all logs reported in the specified class. That
 way it can verify that all logs are being reported within the provided code.
+
+### Logging to file
+
+By default Enso will attempt to persist (verbose) logs into a designated log
+file. This means that even though a user might be shown `WARNING` level logs in
+the console, Logs with up to `TRACE` level will be dumped into the log file. A
+user can disable this parallel logging to a file by setting the environment
+variable:
+
+```
+ENSO_LOG_TO_FILE=false project-manager ...
+```

engine/language-server/src/main/resources/application.conf

@@ -43,4 +43,6 @@ logging-service {
   ]
   default-appender = socket
   default-appender = ${?ENSO_APPENDER_DEFAULT}
+  always-log-to-file = false
+  always-log-to-file = ${?ENSO_LOG_TO_FILE}
 }

engine/launcher/src/main/resources/META-INF/native-image/org/enso/launcher/native-image.properties

@@ -0,0 +1 @@
+Args=--initialize-at-run-time=com.typesafe.config.impl.ConfigImpl$EnvVariablesHolder,com.typesafe.config.impl.ConfigImpl$SystemPropertiesHolder

engine/launcher/src/main/resources/application.conf

@@ -30,6 +30,8 @@ logging-service {
         pattern = "[%level{lowercase=true}] [%d{yyyy-MM-dd'T'HH:mm:ssXXX}] [%logger] %msg%n%nopex"
       }
   ]
-  default-appender = file
+  default-appender = console
   default-appender = ${?ENSO_APPENDER_DEFAULT}
+  always-log-to-file = true
+  always-log-to-file = ${?ENSO_LOG_TO_FILE}
 }

engine/launcher/src/test/scala/org/enso/launcher/NativeLauncherSpec.scala

@@ -6,7 +6,10 @@ import io.circe.parser
 class NativeLauncherSpec extends NativeTest {
   "native launcher" should {
     "display its version" in {
-      val run = runLauncher(Seq("version", "--json", "--only-launcher"))
+      val run = runLauncher(
+        Seq("version", "--json", "--only-launcher"),
+        extraJVMProps = Map("ENSO_LOG_TO_FILE" -> "false")
+      )
       run should returnSuccess
 
       val version = parser.parse(run.stdout).getOrElse {

engine/launcher/src/test/scala/org/enso/launcher/NativeTest.scala

@@ -51,10 +51,12 @@ trait NativeTest
     * @param args arguments to forward to the launcher
     * @param extraEnv environment variables to override for the launched
     *                 program, do not use these to override PATH
+    * @param extraJVMProps JVM properties to append to the launcher command
     */
   def runLauncher(
     args: Seq[String],
-    extraEnv: Map[String, String] = Map.empty
+    extraEnv: Map[String, String]      = Map.empty,
+    extraJVMProps: Map[String, String] = Map.empty
   ): RunResult = {
     if (extraEnv.contains("PATH")) {
       throw new IllegalArgumentException(
@@ -64,7 +66,8 @@ trait NativeTest
 
     runCommand(
       Seq(baseLauncherLocation.toAbsolutePath.toString) ++ args,
-      extraEnv.toSeq
+      extraEnv.toSeq,
+      extraJVMProps.toSeq
     )
   }
 
@@ -75,11 +78,13 @@ trait NativeTest
     * @param args arguments to forward to the launcher
     * @param extraEnv environment variables to override for the launched
     *                 program, do not use these to override PATH
+    * @param extraJVMProps JVM properties to append to the launcher command
     */
   def runLauncherAt(
     pathToLauncher: Path,
     args: Seq[String],
-    extraEnv: Map[String, String] = Map.empty
+    extraEnv: Map[String, String]      = Map.empty,
+    extraJVMProps: Map[String, String] = Map.empty
   ): RunResult = {
     if (extraEnv.contains("PATH")) {
       throw new IllegalArgumentException(
@@ -89,16 +94,20 @@ trait NativeTest
 
     runCommand(
       Seq(pathToLauncher.toAbsolutePath.toString) ++ args,
-      extraEnv.toSeq
+      extraEnv.toSeq,
+      extraJVMProps.toSeq
     )
   }
 
+  /** Returns a relative path to the root directory of the project. */
+  def rootDirectory: Path = Path.of("../../")
+
   /** Returns the expected location of the launcher binary compiled by the
     * Native Image. This binary can be copied into various places to test its
     * functionality.
     */
   def baseLauncherLocation: Path =
-    Path.of(".").resolve(OS.executableName("enso"))
+    rootDirectory.resolve(OS.executableName("enso"))
 
   /** Creates a copy of the tested launcher binary at the specified location.
     *
@@ -124,14 +133,17 @@ trait NativeTest
     * @param args arguments to forward to the launcher
     * @param pathOverride the system PATH that should be set for the launched
     *                     program
+    * @param extraJVMProps JVM properties to append to the launcher command
     */
   def runLauncherWithPath(
     args: Seq[String],
-    pathOverride: String
+    pathOverride: String,
+    extraJVMProps: Map[String, String] = Map.empty
   ): RunResult = {
     runCommand(
       Seq(baseLauncherLocation.toAbsolutePath.toString) ++ args,
-      Seq(NativeTest.PATH -> pathOverride)
+      Seq(NativeTest.PATH -> pathOverride),
+      extraJVMProps.toSeq
     )
   }
 }

engine/launcher/src/test/scala/org/enso/launcher/PluginManagerSpec.scala

@@ -13,6 +13,8 @@ class PluginManagerSpec
     with OptionValues
     with WithTemporaryDirectory {
 
+  val extraJVMProps = Map("ENSO_LOG_TO_FILE" -> "false")
+
   def makePluginCode(name: String): Seq[String] =
     Seq(s"""echo Plugin $name.""")
 
@@ -45,7 +47,7 @@ class PluginManagerSpec
       writePlugin(path, "plugin2")
       writePlugin(path, "plugin3", prefixed = false)
 
-      val run = runLauncherWithPath(Seq("help"), path.toString)
+      val run = runLauncherWithPath(Seq("help"), path.toString, extraJVMProps)
       run should returnSuccess
       run.stdout should include("Plugin plugin1.")
       run.stdout should include("Plugin plugin2.")
@@ -56,7 +58,8 @@ class PluginManagerSpec
       val path = getTestDirectory.toAbsolutePath
       writePlugin(path, "plugin1")
 
-      val run = runLauncherWithPath(Seq("plugin1"), path.toString)
+      val run =
+        runLauncherWithPath(Seq("plugin1"), path.toString, extraJVMProps)
       run should returnSuccess
       run.stdout.trim shouldEqual "Plugin plugin1."
     }
@@ -64,7 +67,8 @@ class PluginManagerSpec
     "suggest similar plugin name on typo" in {
       val path = getTestDirectory.toAbsolutePath
       writePlugin(path, "plugin1")
-      val run = runLauncherWithPath(Seq("plugin2"), path.toString)
+      val run =
+        runLauncherWithPath(Seq("plugin2"), path.toString, extraJVMProps)
       run.exitCode should not equal 0
       run.stdout should include("plugin1")
     }

engine/launcher/src/test/scala/org/enso/launcher/installation/UninstallerSpec.scala

@@ -11,6 +11,8 @@ import org.enso.testkit.WithTemporaryDirectory
 class UninstallerSpec extends NativeTest with WithTemporaryDirectory {
   def installedRoot: Path = getTestDirectory / "installed"
 
+  private val extraJVMProps = Map("ENSO_LOG_TO_FILE" -> "false")
+
   /** Prepares an installed distribution for the purposes of testing
     * uninstallation.
     *
@@ -59,7 +61,8 @@ class UninstallerSpec extends NativeTest with WithTemporaryDirectory {
       runLauncherAt(
         launcher,
         Seq("--auto-confirm", "uninstall", "distribution"),
-        env
+        env,
+        extraJVMProps
       ) should returnSuccess
 
       assert(Files.notExists(installedRoot), "Should remove the data root.")
@@ -77,7 +80,8 @@ class UninstallerSpec extends NativeTest with WithTemporaryDirectory {
       runLauncherAt(
         launcher,
         Seq("--auto-confirm", "uninstall", "distribution"),
-        env
+        env,
+        extraJVMProps
       ) should returnSuccess
 
       assert(Files.notExists(installedRoot), "Should remove the data root.")
@@ -93,7 +97,8 @@ class UninstallerSpec extends NativeTest with WithTemporaryDirectory {
       runLauncherAt(
         launcher,
         Seq("--auto-confirm", "uninstall", "distribution"),
-        env
+        env,
+        extraJVMProps
       ) should returnSuccess
 
       assert(

engine/launcher/src/test/scala/org/enso/launcher/upgrade/UpgradeSpec.scala

@@ -138,11 +138,13 @@ class UpgradeSpec
     *
     * @param args arguments for the launcher
     * @param extraEnv environment variable overrides
+    * @param extraJVMProps JVM properties to append to the launcher command
     * @return wrapped process
     */
   def startLauncher(
     args: Seq[String],
-    extraEnv: Map[String, String] = Map.empty
+    extraEnv: Map[String, String]      = Map.empty,
+    extraJVMProps: Map[String, String] = Map.empty
   ): WrappedProcess = {
     val testArgs = Seq(
       "--internal-emulate-repository",
@@ -154,7 +156,8 @@ class UpgradeSpec
       extraEnv.updated("ENSO_LAUNCHER_LOCATION", realLauncherLocation.toString)
     start(
       Seq(launcherPath.toAbsolutePath.toString) ++ testArgs ++ args,
-      env.toSeq
+      env.toSeq,
+      extraJVMProps.toSeq
     )
   }
 

engine/runner/src/main/scala/org/enso/runner/Main.scala

@@ -1024,7 +1024,7 @@ object Main {
 
   /** Default log level to use if the LOG_LEVEL option is not provided.
     */
-  val defaultLogLevel: Level = Level.ERROR
+  val defaultLogLevel: Level = Level.WARN
 
   /** Main entry point for the CLI program.
     *

engine/runtime/src/main/scala/org/enso/compiler/SerializationManager.scala

@@ -546,7 +546,7 @@ final class SerializationManager(compiler: Compiler) {
       pool.shutdownNow()
       Thread.sleep(100)
       compiler.context.logSerializationManager(
-        debugLogLevel,
+        Level.WARNING,
         "Serialization manager has been shut down."
       )
     }

lib/scala/logging-config/src/main/java/org/enso/logger/config/Appender.java

@@ -66,10 +66,6 @@ public sealed abstract class Appender permits FileAppender, SocketAppender, Sent
     return setup(logLevel, loggerSetup);
   }
 
-  public boolean setupForURI(Level logLevel, String hostname, int port, LoggerSetup loggerSetup) {
-    return setup(logLevel, loggerSetup);
-  }
-
   public static final String defaultPattern =
       "[%level] [%d{yyyy-MM-dd'T'HH:mm:ssXXX}] [%logger] %msg%n";
   protected static final String patternKey = "pattern";

lib/scala/logging-config/src/main/java/org/enso/logger/config/BaseConfig.java

@@ -0,0 +1,25 @@
+package org.enso.logger.config;
+
+import java.util.Map;
+
+/** Base config corresponding to the main logger section in the application config. */
+public interface BaseConfig {
+
+  /** Returns the default appender. */
+  Appender getAppender();
+
+  /** Returns a map of appenders defined in the logger section of the config. */
+  Map<String, Appender> getAppenders();
+
+  /**
+   * Returns true, if logging infrastructure should always log in verbose mode, irrespective of the
+   * log target.
+   */
+  boolean logToFile();
+
+  /**
+   * Returns a list of custom loggers and their levels that need to be taken into account when
+   * logging events.
+   */
+  LoggersLevels getLoggers();
+}

lib/scala/logging-config/src/main/java/org/enso/logger/config/ConsoleAppender.java

@@ -1,6 +1,7 @@
 package org.enso.logger.config;
 
 import com.typesafe.config.Config;
+import java.nio.file.Path;
 import org.enso.logger.LoggerSetup;
 import org.slf4j.event.Level;
 
@@ -24,6 +25,15 @@ public final class ConsoleAppender extends Appender {
     return appenderSetup.setupConsoleAppender(logLevel);
   }
 
+  @Override
+  public boolean setupForPath(
+      Level logLevel, Path logRoot, String logPrefix, LoggerSetup loggerSetup) {
+    if (loggerSetup.getConfig().logToFile()) {
+      loggerSetup.setupFileAppender(Level.TRACE, logRoot, logPrefix);
+    }
+    return loggerSetup.setupConsoleAppender(logLevel);
+  }
+
   public String getPattern() {
     return pattern;
   }

lib/scala/logging-config/src/main/java/org/enso/logger/config/LoggingServer.java

@@ -13,22 +13,42 @@ import java.util.Map;
  * @param appender appender's configuration describing how to transform received log events
  * @param start if true, will be started by the service defining the configuration
  */
-public record LoggingServer(int port, Map<String, Appender> appenders, String appender, Boolean start) {
+public record LoggingServer(int port, Map<String, Appender> appenders, String appender, boolean start, boolean logToFile) implements BaseConfig {
+
+    public static final String startKey = "start";
+
+    public static final String portKey = "port";
+
 
     public static LoggingServer parse(Config config) throws MissingConfigurationField {
-        int port = config.getInt("port");
+        int port = config.getInt(portKey);
 
         Map<String, Appender> appendersMap = new HashMap<>();
-        if (config.hasPath("appenders")) {
+        if (config.hasPath(LoggingServiceConfig.appendersKey)) {
-            List<? extends Config> configs = config.getConfigList("appenders");
+            List<? extends Config> configs = config.getConfigList(LoggingServiceConfig.appendersKey);
             for (Config c : configs) {
                 Appender a = Appender.parse(c);
                 appendersMap.put(a.getName(), a);
             }
         }
-        String defaultAppender = config.getString("default-appender");
+        String defaultAppender = config.getString(LoggingServiceConfig.defaultAppenderKey);
-        boolean start = config.getBoolean("start");
+        boolean start = config.hasPath(startKey) ? config.getBoolean(startKey) : false;
-        return new LoggingServer(port, appendersMap, defaultAppender, start);
+        boolean logToFile = config.hasPath(LoggingServiceConfig.alwaysLogToFileKey) ? config.getBoolean(LoggingServiceConfig.alwaysLogToFileKey) : false;
+        return new LoggingServer(port, appendersMap, defaultAppender, start, logToFile);
     }
 
+    @Override
+    public Appender getAppender() {
+        return appenders.get(appender);
+    }
+
+    @Override
+    public Map<String, Appender> getAppenders() {
+        return appenders;
+    }
+
+    @Override
+    public LoggersLevels getLoggers() {
+        return null;
+    }
 }

lib/scala/logging-config/src/main/java/org/enso/logger/config/LoggingServiceConfig.java

@@ -12,18 +12,20 @@ import java.util.Optional;
  * Parsed and verified representation of `logging-service` section of `application.conf`. Defines
  * custom log levels, logging appenders and, optionally, logging server configuration.
  */
-public class LoggingServiceConfig {
+public class LoggingServiceConfig implements BaseConfig {
   public static final String configurationRoot = "logging-service";
   public static final String serverKey = "server";
   public static final String loggersKey = "logger";
   public static final String appendersKey = "appenders";
   public static final String defaultAppenderKey = "default-appender";
   public static final String logLevelKey = "log-level";
+  public static final String alwaysLogToFileKey = "always-log-to-file";
 
   private final LoggersLevels loggers;
   private final Map<String, Appender> appenders;
 
   private final String defaultAppenderName;
+  private final boolean alwaysLogToFile;
   private final Optional<String> logLevel;
   private final LoggingServer server;
 
@@ -32,10 +34,12 @@ public class LoggingServiceConfig {
       Optional<String> logLevel,
       Map<String, Appender> appenders,
       String defaultAppender,
+      boolean alwaysLogToFile,
       LoggingServer server) {
     this.loggers = loggers;
     this.appenders = appenders;
     this.defaultAppenderName = defaultAppender;
+    this.alwaysLogToFile = alwaysLogToFile;
     this.logLevel = logLevel;
     this.server = server;
   }
@@ -64,29 +68,42 @@ public class LoggingServiceConfig {
     } else {
       loggers = LoggersLevels.parse();
     }
+    boolean logToFile =
+        root.hasPath(alwaysLogToFileKey) ? root.getBoolean(alwaysLogToFileKey) : false;
     return new LoggingServiceConfig(
         loggers,
         getStringOpt(logLevelKey, root),
         appendersMap,
         root.getString(defaultAppenderKey),
+        logToFile,
         server);
   }
 
-  public static LoggingServiceConfig withSingleAppender(Appender appender) {
+  public static LoggingServiceConfig withSingleAppender(BaseConfig config) {
-    Map<String, Appender> map = new HashMap<>();
+    Map<String, Appender> map = config.getAppenders();
-    map.put(appender.getName(), appender);
     return new LoggingServiceConfig(
-        LoggersLevels.parse(), Optional.empty(), map, appender.getName(), null);
+        LoggersLevels.parse(),
+        Optional.empty(),
+        map,
+        config.getAppender().getName(),
+        config.logToFile(),
+        null);
   }
 
   public LoggersLevels getLoggers() {
     return loggers;
   }
 
+  @Override
   public Appender getAppender() {
     return appenders.get(defaultAppenderName);
   }
 
+  @Override
+  public Map<String, Appender> getAppenders() {
+    return appenders;
+  }
+
   public SocketAppender getSocketAppender() {
     return (SocketAppender) appenders.getOrDefault(SocketAppender.appenderName, null);
   }
@@ -133,7 +150,14 @@ public class LoggingServiceConfig {
         + (defaultAppenderName == null ? "unknown" : defaultAppenderName)
         + ", logLevel: "
         + logLevel.orElseGet(() -> "default")
+        + ", log-to-file: "
+        + logToFile()
         + ", server: "
         + server;
   }
+
+  @Override
+  public boolean logToFile() {
+    return alwaysLogToFile;
+  }
 }

lib/scala/logging-config/src/main/java/org/enso/logger/config/SentryAppender.java

@@ -50,6 +50,9 @@ public final class SentryAppender extends Appender {
   @Override
   public boolean setupForPath(
       Level logLevel, Path logRoot, String logPrefix, LoggerSetup loggerSetup) {
+    if (loggerSetup.getConfig().logToFile()) {
+      loggerSetup.setupFileAppender(Level.TRACE, logRoot, logPrefix);
+    }
     return loggerSetup.setupSentryAppender(logLevel, logRoot);
   }
 

lib/scala/logging-config/src/main/java/org/enso/logger/config/SocketAppender.java

@@ -57,11 +57,6 @@ public final class SocketAppender extends Appender {
     return loggerSetup.setupSocketAppender(logLevel, host, port);
   }
 
-  @Override
-  public boolean setupForURI(Level logLevel, String host, int port, LoggerSetup loggerSetup) {
-    return loggerSetup.setupSocketAppender(logLevel, host, port);
-  }
-
   private static final String hostKey = "hostname";
   private static final String portKey = "port";
   private static final String reconnectionDelayKey = "reconnection-delay";

lib/scala/logging-service-logback/src/main/java/org/enso/logger/LogbackSetup.java

@@ -2,6 +2,7 @@ package org.enso.logger;
 
 import ch.qos.logback.classic.LoggerContext;
 import ch.qos.logback.classic.Logger;
+import ch.qos.logback.classic.filter.ThresholdFilter;
 import ch.qos.logback.classic.net.SocketAppender;
 import ch.qos.logback.classic.spi.ILoggingEvent;
 import ch.qos.logback.classic.encoder.PatternLayoutEncoder;
@@ -43,10 +44,10 @@ public final class LogbackSetup extends LoggerSetup {
     /**
      * Create a logger setup for a provided context and a single appender configuration
      * @param context context that will be initialized by this setup
-     * @param appender appender configuration to use during initialization
+     * @param config configuration to use during initialization
      */
-    public static LogbackSetup forContext(LoggerContext context, Appender appender) {
+    public static LogbackSetup forContext(LoggerContext context, BaseConfig config) {
-        return new LogbackSetup(LoggingServiceConfig.withSingleAppender(appender), context);
+        return new LogbackSetup(LoggingServiceConfig.withSingleAppender(config), context);
     }
 
 
@@ -97,7 +98,15 @@ public final class LogbackSetup extends LoggerSetup {
             Level logLevel,
             String hostname,
             int port) {
-        LoggerAndContext env = contextInit(logLevel, config);
+        Level targetLogLevel;
+        // Modify log level if we were asked to always log to a file.
+        // The receiver needs to get all logs (up to `trace`) so as to be able to log all verbose messages.
+        if (config.logToFile()) {
+            targetLogLevel = Level.TRACE;
+        } else {
+            targetLogLevel = logLevel;
+        }
+        LoggerAndContext env = contextInit(targetLogLevel, config, !config.logToFile());
 
         org.enso.logger.config.SocketAppender appenderConfig = config.getSocketAppender();
 
@@ -109,8 +118,8 @@ public final class LogbackSetup extends LoggerSetup {
         if (appenderConfig != null)
             socketAppender.setReconnectionDelay(Duration.buildByMilliseconds(appenderConfig.getReconnectionDelay()));
 
-        env.finalizeAppender(socketAppender);
 
+        env.finalizeAppender(socketAppender, config.logToFile());
         return true;
     }
 
@@ -121,8 +130,7 @@ public final class LogbackSetup extends LoggerSetup {
             Path logRoot,
             String logPrefix) {
         try {
-            LoggerAndContext env = contextInit(logLevel, config);
+            LoggerAndContext env = contextInit(logLevel, config, true);
-
             org.enso.logger.config.FileAppender appenderConfig = config.getFileAppender();
             if (appenderConfig == null) {
                 throw new MissingConfigurationField(org.enso.logger.config.FileAppender.appenderName);
@@ -175,7 +183,7 @@ public final class LogbackSetup extends LoggerSetup {
             fileAppender.setEncoder(encoder);
 
 
-            env.finalizeAppender(fileAppender);
+            env.finalizeAppender(fileAppender, false);
         } catch (Throwable e) {
             e.printStackTrace();
             return false;
@@ -185,7 +193,7 @@ public final class LogbackSetup extends LoggerSetup {
 
     @Override
     public boolean setupConsoleAppender(Level logLevel) {
-        LoggerAndContext env = contextInit(logLevel, config);
+        LoggerAndContext env = contextInit(logLevel, config, !getConfig().logToFile());
         org.enso.logger.config.ConsoleAppender appenderConfig = config.getConsoleAppender();
         final PatternLayoutEncoder encoder = new PatternLayoutEncoder();
         try {
@@ -200,7 +208,7 @@ public final class LogbackSetup extends LoggerSetup {
         consoleAppender.setName("enso-console");
         consoleAppender.setEncoder(encoder);
 
-        env.finalizeAppender(consoleAppender);
+        env.finalizeAppender(consoleAppender, false);
         return true;
     }
 
@@ -209,7 +217,7 @@ public final class LogbackSetup extends LoggerSetup {
         // TODO: handle proxy
         // TODO: shutdown timeout configuration
         try {
-            LoggerAndContext env = contextInit(logLevel, config);
+            LoggerAndContext env = contextInit(logLevel, config, !config.logToFile());
 
             org.enso.logger.config.SentryAppender appenderConfig = config.getSentryAppender();
             if (appenderConfig == null) {
@@ -234,7 +242,7 @@ public final class LogbackSetup extends LoggerSetup {
             opts.setDsn(appenderConfig.getDsn());
             appender.setOptions(opts);
 
-            env.finalizeAppender(appender);
+            env.finalizeAppender(appender, config.logToFile());
         } catch (Throwable e) {
             e.printStackTrace();
             return false;
@@ -244,12 +252,12 @@ public final class LogbackSetup extends LoggerSetup {
 
     @Override
     public boolean setupNoOpAppender() {
-        LoggerAndContext env = contextInit(Level.ERROR, null);
+        LoggerAndContext env = contextInit(Level.ERROR, null, true);
 
         NOPAppender<ILoggingEvent> appender = new NOPAppender<>();
         appender.setName("enso-noop");
 
-        env.finalizeAppender(appender);
+        env.finalizeAppender(appender, false);
         return true;
     }
 
@@ -259,9 +267,11 @@ public final class LogbackSetup extends LoggerSetup {
         context.stop();
     }
 
-    private LoggerAndContext contextInit(Level level, LoggingServiceConfig config) {
+    private LoggerAndContext contextInit(Level level, LoggingServiceConfig config, boolean shouldResetContext) {
-        context.reset();
+        if (shouldResetContext) {
-        context.setName("enso-custom");
+            context.reset();
+            context.setName("enso-custom");
+        }
         Logger rootLogger = context.getLogger(Logger.ROOT_LOGGER_NAME);
 
         Filter<ILoggingEvent> filter;
@@ -280,8 +290,17 @@ public final class LogbackSetup extends LoggerSetup {
             encoder.setContext(ctx);
             encoder.start();
         }
-        void finalizeAppender(ch.qos.logback.core.Appender<ILoggingEvent> appender) {
+        void finalizeAppender(ch.qos.logback.core.Appender<ILoggingEvent> appender, boolean isLogToFile) {
-            logger.setLevel(ch.qos.logback.classic.Level.convertAnSLF4JLevel(level));
+            ThresholdFilter threshold = new ThresholdFilter();
+            threshold.setLevel(ch.qos.logback.classic.Level.convertAnSLF4JLevel(level).toString());
+            appender.addFilter(threshold);
+            threshold.setContext(ctx);
+            threshold.start();
+
+            // Root's log level is set to TRACE, meaning we want to log all events.
+            // Log level is controlled by `ThresholdFilter` instead, allowing is to specify different
+            // log levels for different outputs.
+            logger.setLevel(ch.qos.logback.classic.Level.TRACE);
             if (filter != null) {
                 appender.addFilter(filter);
                 filter.setContext(ctx);

lib/scala/logging-service-logback/src/main/java/org/enso/logging/LoggingServer.java

@@ -6,7 +6,7 @@ import java.net.URI;
 import java.net.URISyntaxException;
 import java.nio.file.Path;
 import org.enso.logger.LogbackSetup;
-import org.enso.logger.config.Appender;
+import org.enso.logger.config.BaseConfig;
 import org.slf4j.event.Level;
 
 class LoggingServer extends LoggingService<URI> {
@@ -19,13 +19,13 @@ class LoggingServer extends LoggingService<URI> {
     this.logServer = null;
   }
 
-  public URI start(Level level, Path path, String prefix, Appender appender) {
+  public URI start(Level level, Path path, String prefix, BaseConfig config) {
     var lc = new LoggerContext();
-    var setup = LogbackSetup.forContext(lc, appender);
 
-    logServer = new SimpleSocketServer(lc, port);
-    logServer.start();
     try {
+      var setup = LogbackSetup.forContext(lc, config);
+      logServer = new SimpleSocketServer(lc, port);
+      logServer.start();
       setup.setup(level, path, prefix, setup.getConfig());
       return new URI(null, null, "localhost", port, null, null, null);
     } catch (URISyntaxException e) {

lib/scala/logging-service/src/main/java/org/enso/logging/LoggingService.java

@@ -1,7 +1,7 @@
 package org.enso.logging;
 
 import java.nio.file.Path;
-import org.enso.logger.config.Appender;
+import org.enso.logger.config.BaseConfig;
 import org.slf4j.event.Level;
 
 /**
@@ -16,12 +16,12 @@ public abstract class LoggingService<T> {
    * events.
    *
    * @param level the maximal log level handled by this service
-   * @param path
+   * @param logRoot the root directory where logs are located
-   * @param prefix
+   * @param logPrefix the prefix used in the name of the log file
-   * @param appender
+   * @param config config for the server log target
    * @return
    */
-  public abstract T start(Level level, Path path, String prefix, Appender appender);
+  public abstract T start(Level level, Path logRoot, String logPrefix, BaseConfig config);
 
   /** Shuts down the service. */
   public abstract void teardown();

lib/scala/logging-service/src/main/java/org/enso/logging/LoggingServiceManager.java

@@ -2,7 +2,6 @@ package org.enso.logging;
 
 import java.net.URI;
 import java.nio.file.Path;
-import org.enso.logger.config.Appender;
 import org.enso.logger.config.LoggingServer;
 import org.slf4j.event.Level;
 import scala.concurrent.ExecutionContext;
@@ -27,13 +26,12 @@ public class LoggingServiceManager {
       throw new LoggingServiceAlreadySetup();
     } else {
       if (config.appenders().containsKey(config.appender())) {
-        currentLevel = logLevel;
+        currentLevel = config.logToFile() ? Level.TRACE : logLevel;
         return Future.apply(
             () -> {
               var server = LoggingServiceFactory.get().localServerFor(port);
               loggingService = server;
-              Appender appender = config.appenders().get(config.appender());
+              return server.start(logLevel, logPath, logFileSuffix, config);
-              return server.start(logLevel, logPath, logFileSuffix, appender);
             },
             ec);
       } else {

lib/scala/logging-service/src/main/java/org/enso/logging/LoggingSetupHelper.java

@@ -93,7 +93,7 @@ public abstract class LoggingSetupHelper {
               ec);
     } else {
       // Setup logger according to config
-      if (loggerSetup.setup(logLevel)) {
+      if (loggerSetup.setup(logLevel, logPath(), logFileSuffix(), loggerSetup.getConfig())) {
         loggingServiceEndpointPromise.success(Option.empty());
       }
     }

lib/scala/project-manager/src/main/resources/application.conf

@@ -21,6 +21,7 @@ logging-service {
     akka.http = warn
     akka.stream = error
     akka.routing = error
+    ch.qos.logback.classic.net.SimpleSocketServer = error
   }
   appenders = [
       {
@@ -31,19 +32,23 @@ logging-service {
         port = ${?ENSO_LOGSERVER_PORT}
       },
       {
-        name = "file",
+        name = "file"
       },
       {
         name = "console"
       }
   ]
-  default-appender = ${?ENSO_APPENDER_DEFAULT}
   default-appender = socket
+  default-appender = ${?ENSO_APPENDER_DEFAULT}
+  always-log-to-file = true
+  always-log-to-file = ${?ENSO_LOG_TO_FILE}
   server {
     start = true
     start = ${?ENSO_LOGSERVER_START}
     port = 6000
     port = ${?ENSO_LOGSERVER_PORT}
+    always-log-to-file = true
+    always-log-to-file = ${?ENSO_LOG_TO_FILE}
     appenders = [ # file/console/socket/sentry
         {
           name = "file"
@@ -62,7 +67,7 @@ logging-service {
           name = "console"
         }
     ]
-    default-appender = file
+    default-appender = console
     default-appender = ${?ENSO_LOGSERVER_APPENDER}
   }
 }

lib/scala/runtime-version-manager-test/src/main/scala/org/enso/runtimeversionmanager/test/NativeTestHelper.scala

@@ -15,19 +15,19 @@ trait NativeTestHelper {
 
   /** Starts the provided `command`.
     *
-    * `extraEnv` may be provided to extend the environment. Care must be taken
+    * @param command       executable and its arguments
-    * on Windows where environment variables are (mostly) case-insensitive.
+    * @param extraEnv      extra environment properties added to the environment. Care must be taken
-    *
+    *                      on Windows where environment variables are (mostly) case-insensitive.
-    * If `waitForDescendants` is set, tries to wait for descendants of the
+    * @param extraJVMProps extra JVM properties to be appended to the command
-    * launched process to finish too. Especially important on Windows where
-    * child processes may run after the launcher parent has been terminated.
     */
   def start(
     command: Seq[String],
-    extraEnv: Seq[(String, String)]
+    extraEnv: Seq[(String, String)],
+    extraJVMProps: Seq[(String, String)]
   ): WrappedProcess = {
-    val builder = new JProcessBuilder(command: _*)
+    val fullCommand = command ++ extraJVMProps.map(v => s"-D${v._1}=${v._2}")
-    val newKeys = extraEnv.map(_._1.toLowerCase)
+    val builder     = new JProcessBuilder(fullCommand: _*)
+    val newKeys     = extraEnv.map(_._1.toLowerCase)
     if (newKeys.distinct.size < newKeys.size) {
       throw new IllegalArgumentException(
         "The extra environment keys have to be unique"
@@ -56,13 +56,20 @@ trait NativeTestHelper {
 
   /** Runs the provided `command`.
     *
-    * `extraEnv` may be provided to extend the environment. Care must be taken
+    * @param command executable and its arguments
-    * on Windows where environment variables are (mostly) case-insensitive.
+    * @param extraEnv extra environment properties added to the environment. Care must be taken
+    *                 on Windows where environment variables are (mostly) case-insensitive.
+    * @param extraJVMProps extra JVM properties to be appended to the command
+    * @param waitForDescendants if true, tries to wait for descendants of the launched process to finish too.
+    *                           Especially important on Windows where child processes may run after the launcher
+    *                           parent has been terminated.
     */
   def runCommand(
     command: Seq[String],
     extraEnv: Seq[(String, String)],
+    extraJVMProps: Seq[(String, String)],
     waitForDescendants: Boolean = true
-  ): RunResult = start(command, extraEnv).join(waitForDescendants)
+  ): RunResult =
+    start(command, extraEnv, extraJVMProps).join(waitForDescendants)
 
 }

lib/scala/runtime-version-manager-test/src/test/scala/org/enso/distribution/locking/ThreadSafeFileLockManagerTest.scala

@@ -145,7 +145,8 @@ class ThreadSafeFileLockManagerTest
 
       val otherProcess = start(
         Seq("java", "-jar", "locking-test-helper.jar", lockFilePath.toString),
-        Seq()
+        Seq.empty,
+        Seq.empty
       )
 
       try {

tools/legal-review/Table/org.apache.logging.log4j.log4j-to-slf4j-2.18.0/copyright-ignore

@@ -0,0 +1 @@
+<bottom><![CDATA[<p align="center">Copyright &#169; {inceptionYear}-{currentYear} {organizationName}. All Rights Reserved.<br />

tools/legal-review/Table/org.apache.logging.log4j.log4j-to-slf4j-2.18.0/copyright-keep

@@ -0,0 +1 @@
+this work for additional information regarding copyright ownership.

tools/legal-review/Table/org.apache.logging.log4j.log4j-to-slf4j-2.18.0/files-ignore

@@ -0,0 +1 @@
+META-INF/LICENSE

tools/legal-review/Table/org.apache.logging.log4j.log4j-to-slf4j-2.18.0/files-keep

@@ -0,0 +1 @@
+META-INF/NOTICE

tools/legal-review/Table/org.slf4j.slf4j-api-1.7.36/copyright-keep

@@ -0,0 +1 @@
+Copyright (c) 2004-2011 QOS.ch

tools/legal-review/Table/report-state

@@ -1,3 +1,3 @@
-843708DDAA13743FBA3E74EE4A81567A1CB3F631F093C23899150D46975868FD
+AE474B24FC7C88ACA56C70EC19DCD5F224178089AA2910DB117EE7D914D6C7FF
-DB34B2F25C499C2E108C31C84D8AD9D824C8A9DD03484EEE245C95309A2A672E
+A4E29BBEAAEE4B4A5593D949D658170B8591E0FADA2F469CDCBF640B307B74D4
 0