New year, new copyright, new expected unknown issues with various files
that won't be covered by the script and/or will be but shouldn't change.
I'll do the details on Jan 1, but would appreciate this being
preapproved so I can actually get it merged by then.
CHANGELOG_BEGIN
CHANGELOG_END
* Update rules_haskell
changelog_begin
changelog_end
* Use mostly deterministic distdir path
changelog_begin
changelog_end
Use a path for distdir in the `haskell_cabal_*` rules that is based on
the package name and deterministic unless there is a collision with an
already existing file or directory.
The distdir path enters flags that are passed to GHC and thereby enters
the 'flag hash' field of the generated interface files. If the distdir
path is indeterministic, then the interface file will also be
indeterministic.
* Pin com_google_protobuf to 3.17.3
The latest version of rules_haskell pins protoc to 3.19.1. However, this
generates code that also requires a more recent version of the Java
proto libraries. This change pins the protoc version at the same version
as the protobuf-java library, see https://github.com/protocolbuffers/protobuf/issues/9236.
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Try to upgrade protobuf docs plugin
changelog_begin
changelog_end
* Fix extension number 65020 is already registered
Building `//ledger-api/grpc-definitions:ledger-api-docs` [failed
with](https://github.com/digital-asset/daml/issues/11761#issuecomment-978947565)
```
panic: proto: extension number 65020 is already registered on message google.protobuf.FieldOptions
```
Go dependencies are now pulled in via Gazelle. By default Gazelle will
generate new proto rules for any `.proto` files encountered in third
party Go dependencies. However, many of these already have pregenerated
`.pb.go` files generate with the appropriate configuration.
The problem can be avoided by configuring Gazelle to not generate new
proto rules, but instead use pre-existing `.pb.go` files.
For reference the field number is set in
[go-proto-validators](32a686adf8/validator.proto (L19))
which is an indirect dependency through protoc-gen-doc.
In this case we need to update protoc-gen-validate to v0.6.2 to include
4f41f10dde
which fixes unknown label errors.
* ./fmt
* Expose gRPC status.proto for Haskell bindings
* Update Gazelle to support embedsrcs on Windows
`protoc-gen-doc` relies on `go:embed` file embedding
2dde01902b/resources.go (L8).
Gazelle supports `embedsrcs`, however, it did not generate the attribute
correctly on Windows due to the different directory separator. This is
fixed in https://github.com/bazelbuild/bazel-gazelle/pull/1101.
* Add gazelle to compatibility workspace
It's loaded into `@daml`'s top-level `BUILD` file and ends up being a
dependency of the compatibility workspace as well.
* shift go_googleapis import
* Delete dead code
protobuf is imported transitively.
* Document how to add Go dependencies
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Drop LF < 1.14 from supported damlc output versions
fixes#11319
We keep test coverage by depending on the most recent snapshot which
still has 1.14 support.
changelog_begin
- [Daml Compiler] Damlc can only produce Daml-LF 1.14 or
newer. Passing aynthing older to `--target` is an error. If you
need to produce older versions, use an older SDK.
changelog_end
* Switch around legacy_compiler_lf_versions
changelog_begin
changelog_end
* drop since-lf
changelog_begin
changelog_end
* Drop Scala 2.12 support
This only includes the CI/build system infrastructure
changes. Dropping compatibility layers from our code for 2.12 can be
done separately.
This is fine even in the context of backport builds since we already
disable the Scala 2.12 job for those anyway.
fixes#11315
changelog_begin
changelog_end
* Update bazel-java-deps.bzl
Co-authored-by: pbatko-da <pawel.batko@digitalasset.com>
Co-authored-by: pbatko-da <pawel.batko@digitalasset.com>
* Update rules_scala
Two patches have since been upstreamed and can be removed.
changelog_begin
changelog_end
* JMH deps were moved into a toolchain
https://github.com/bazelbuild/rules_scala/pull/1106
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
We need to make some changes to have any chance of ARM support working
and I’d rather make those on top of the latest version.
Also deletes an unused patch.
changelog_begin
changelog_end
* Fix rules_scala Windows launcher
* Define nixpkgs_java_configure
To generate a java_runtime based on a Nix provided JDK similar to how it
is done with the Python and CC toolchain.
changelog_begin
changelog_end
* Enable Nix provided Java runtime on Linux and MacOS
* Define dadew_java_configure
To import a JDK and JAVA_HOME from dadew similar to
nixpkgs_java_configure.
* Enable dadew Java runtime on Windows
* Remove now unused java_home_runtime
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
If we build the dependency lists in the `_create_scala_repl` function,
we don't add the correct `scalac` options; specifically, we don't add
the Silencer option to ignore unused imports from
`scala.collection .compat`.
By passing directly to `_wrap_rule`, we compute the `scalac` options
correctly.
CHANGELOG_BEGIN
CHANGELOG_END
We need _wrap_rule to make sure plugins like wartremover are also
correctly applied here.
We need jline because otherwise the repl starts but then fails because
jline isn’t in scope.
changelog_begin
changelog_end
* Use nix-provided gRPC on Unix-like OSs
This fixes the da-ghcid command on macOS and Linux.
Unfortunately on macos at least on CI it still seems to segfault when
trying to run damlc in ghci. However, at least typechecking works
so this is clearly progress.
changelog_begin
changelog_end
Adding support for accepting server's private key as an encrypted file (since storing unencrypted private key in a file system might be a risk).
Encrypted private key is assumed to be encrypted using AES or similar algorithm. The details necessary to decrypt it are be obtained from a secrets server over HTTP as JSON document. The URL to secret's server is supplied through the new `--secrets-url` CLI parameter.
One can supply private in either plaintext (old behavior) or ciphertext: if a private key's file ends with .enc suffix it is assumed to be ciphertext. Otherwise it is assumed to be plain text.
CHANGELOG_BEGIN
- [DPP-418] [Participant] Add support for supplying server's private key as an encrypted file and then decrypting it with the help of a secrets server.
CHANGELOG_END
* Generate short to long name mapping in aspect
Maps shortened test names in da_scala_test_suite on Windows to their
long name on Linux and MacOS.
Names are shortened on Windows to avoid exceeding MAX_PATH.
* Script to generate scala test name mapping
* Generate scala-test-suite-name-map.json on Windows
changelog_begin
changelog_end
* Generate UTF-8 with Unix line endings
Otherwise the file will be formatted using UTF-16 with CRLF line
endings, which confuses `jq` on Linux.
* Apply Scala test name remapping before ES upload
* Pipe bazel output into intermediate file
Bazel writes the output of --experimental_show_artifacts to stderr
instead of stdout. In Powershell this means that these outputs are not
plain strings, but instead error objects. Simply redirecting these to
stdout and piping them into further processing will lead to
indeterministically missing items or indeterministically introduced
additional newlines which may break paths.
To work around this we extract the error message from error objects,
introduce appropriate newlines, and write the output to a temporary file
before further processing.
This solution is taken and adapted from
https://stackoverflow.com/a/48671797/841562
* Add copyright header
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Use `extra` in the port file runner, rather than `temporary`.
* ledger-api-test-tool-on-canton: Use the port check runner.
Much simpler than the port file runner for our purposes.
* Replace `runner` with `runner_with_port_file`.
Rather than expecting a particular set of command-line-arguments, we use
templating.
CHANGELOG_BEGIN
CHANGELOG_END
* Rename the `runner_with_port_check` target to the default.
* Use the port file and dynamic port generation in client/server tests.
This creates a runner named `runner_with_port_file` which knows how to
interpolate two variables, `%PORT_FILE%` and `%PORT%`. This allows us to
use the `port-file` argument to the kvutils runner rather than
hard-coding a port for conformance tests.
For now, we only use this for generating the kvutils reference ledger
export.
CHANGELOG_BEGIN
CHANGELOG_END
* Simplify the runner_with_port_file considerably.
It doesn't need to check if the port is open; we trust that the process
will do it.
This also makes sure the port file will be cleaned up, and reduces the
number of dependencies by making use of more functions in `extra`.
* Simplify port file generation in the new client-server runner.
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Simplify the runner_with_port_file further.
This doesn't need to work if the server doesn't take a port file.
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Update rules_haskell & rules_nixpkgs
changelog_begin
changelog_end
* remove unused patch
* Remove the strict-source-names patch
This patch was largely motivated by issues with directory outputs on
Windows occasionally producing empty outputs that would poison the
remote cache. However, we have continued seeing this type of issue
despite this patch and have since resorted to a regular fixup job that
cleans such empty items out of the cache.
With that the strongest motivation with this change is no longer valid
and it doesn't seem worth it to maintain it any longer.
changelog_begin
changelog_end
* Remove windows-remove-fake-libs patch
This patch was upstreamed.
* Update windows-extra-libraries patch
The GHC bindist patch is no longer required as it applied to GHC version
8.10.3 but we are using 8.10.4 instead.
* Remove cc-wrapper-windows patch
This was upstreamed.
* Remove protobuf-source patch
This was upstreamed
* Keep patches in sync between daml and compatibility
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* contextualized-logging: Automatically convert logging values to strings.
For now, this has almost the same behavior, but it allows us to
customize the output in the future.
The main change is that the log format has gone from:
context: {a=b, x=1, foo=bar}
to:
context: {a: "b", x: "1", foo: "bar"}
* contextualized-logging: Move `writeTo` inside `LoggingValue`.
* contextualized-logging: Allow for more than just strings.
`null`, numbers, and sequences are now correctly logged.
The log format has gone from:
context: {a: "b", x: "1", foo: "bar", parties: "[alice, bob]"}
to:
context: {a: "b", x: 1, foo: "bar", parties: ["alice", "bob"]}
CHANGELOG_BEGIN
- The log output of Daml components has changed so that the structured
part is closer to JSON. This allows us to distinguish and parse
numbers and lists. If you are parsing this log output, you may need to
change your parser.
The log output has changed from:
.. code-block::
context: {a=b, x=1, foo=bar, parties=[alice, bob]}
to:
.. code-block::
context: {a: "b", x: 1, foo: "bar", parties: ["alice", "bob"]}
CHANGELOG_END
* contextualized-logging: Extract the string serializer.
* Use non-string logging where possible.
* contextualized-logging: Split logging values from serialization.
So that callers don't have to know about Jackson.
* contextualized-logging: `SeqView` is `Iterable`. Don't need both.
* contextualized-logging: Make `ToStringToLoggingValue` a `val`.
Co-Authored-By: Stephen Compall <stephen.compall@daml.com>
* contextualized-logging: Add a transient dependency for 2.12 only.
This required more infrastructure than I thought it would.
* kvutils: Make it explicit that we're logging the hashes of archives.
The implicit was found to be a little confusing.
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* Upgrade protobuf-java and scalapb.
CHANGELOG_BEGIN
CHANGELOG_END
* Bazel: Generate ScalaPB classes without `unknownFields`.
This is very annoying because it means that extractors need an extra
`_` for no good reason. It also breaks Circe's automatic derivation.
Including unknown fields is the default behavior in Protocol Buffers for
Java now, and we'll probably have to reckon with it at some point, but
let's kick that can down the road a little.
* Upgrade protobuf-java to 3.17.1.
This is identical to 3.17.0, but let's track it anyway.
* Bazel: Move ScalaPB versions into their own file.
They don't need to go into the generated one.
* client_server_build - user defined outputs
Let the user define a list of output files on client_server_build.
changelog_begin
changelog_end
* daml-script runner expose main(config)
changelog_begin
changelog_end
* Add an example Daml ledger export to the docs
* Build Daml ledger export
changelog_begin
changelog_end
* sh_inline_test
Support toolchain arguments to sh_inline_test. Usefuly for make variable
expansion, e.g. for the POSIX toolchain.
* Add a golden test for Daml ledger export
* Test args files as well
* Use sed from POSIX toolchain
* Add normalization comment to top-level comment
* Ignore trailing CR on Windows
The JSON formatting of the args file uses Windows line endings on
Windows. Therefore, the args.json output technically differs from the
expected output. We're happy to ignore the difference in CRLF here.
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* client_server runner - use temp dir for port file
The sandbox will not overwrite an already existing port file but instead
fail, or worse, silently ignore the error and leave the port-file empty.
changelog_begin
changelog_end
* sandbox: Fail if writing the port file fails
So far this was being silently ignored, leaving a pre-existing port-file
untouched.
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* check whether collection.compat is unused when compiling for Scala 2.12
- Instead of always suppressing warnings for collection.compat._,
we should only do it for Scala 2.13
- We can also reduce boilerplate by automatically adding this
option when both silencer_plugin and collection-compat are
present
CHANGELOG_BEGIN
CHANGELOG_END
* remove unused import
* remove another unused import
* remove even more unused imports
* missed compat dependency
* more missed compat dependencies
* missed compat dependency
* use scala_deps in scaladoc_jar
- #8423 inlined the major version expansion, but this seems to
have been prior to proper support by scaladoc_jar
* restore custom handling of participant-integration-api
- fixing scaladoc_jar isn't worth it for a single case, as with
deps vs scala_deps
* Add a REPL for each Scala target, for debugging.
For each Scala library target, this adds a `*_repl` target that can be
built and run to provide a Scala REPL with access to the library and all
its dependencies.
To use:
```
$ bazel build //ledger/ledger-on-memory:ledger-on-memory_repl
INFO: Invocation ID: f1c4ec07-68e7-4bc2-a182-40f6ea035110
INFO: Analyzed target //ledger/ledger-on-memory:ledger-on-memory_repl (32 packages loaded, 714 targets configured).
INFO: Found 1 target...
Target //ledger/ledger-on-memory:ledger-on-memory_repl up-to-date:
bazel-bin/ledger/ledger-on-memory/ledger-on-memory_repl.jar
bazel-bin/ledger/ledger-on-memory/ledger-on-memory_repl
INFO: Build completed successfully, 7 total actions
$ ./bazel-bin/ledger/ledger-on-memory/ledger-on-memory_repl
Welcome to Scala 2.12.13 (OpenJDK 64-Bit Server VM, Java 1.8.0_272).
Type in expressions for evaluation. Or try :help.
scala> import scala.concurrent.ExecutionContext.Implicits.global
scala> import com.google.protobuf.ByteString
scala> import com.daml.ledger.on.memory._
scala> import com.daml.ledger.participant.state.kvutils.OffsetBuilder
scala> import com.daml.ledger.participant.state.kvutils.Raw
scala> import com.daml.ledger.participant.state.kvutils.api.LedgerRecord
scala> val state = InMemoryState.empty
state: com.daml.ledger.on.memory.InMemoryState = com.daml.ledger.on.memory.InMemoryState@ff21443
scala> state.write { (log, state) => Future { log += LedgerRecord(OffsetBuilder.fromLong(1), Raw.LogEntryId(ByteString.copyFromUtf8("A")), Raw.Envelope(ByteString.EMPTY)) } }
```
The REPL target has the tag "manual", so will only be built on demand.
CHANGELOG_BEGIN
CHANGELOG_END
* Turns out you can run a REPL with `bazel run`.
* participant-integration-api: Build Oracle tests, but don't run them.
CHANGELOG_BEGIN
CHANGELOG_END
* triggers: Switch to an environment variable for enabling Oracle tests.
* http-json: Switch to an environment variable for enabling Oracle tests.
* Disable running Oracle tests by default, not building them.
* triggers/service: Remove unused test dependencies.
* Switch from `@silent` to `@nowarn`.
This annotation is native to Scala 2.12.13+ and 2.13.2+. It replaces
most usages of `@silent`.
I had to get creative about a couple of use cases that didn't work.
Specifically:
1. Suppressing deprecation warnings works, but Scala 2.12 erroneously
complains that the `@nowarn` is unnecessary. I had to suppress
this warning too with `-Ywarn-unused:-nowarn`.
2. I can't seem to suppress the warning, "The outer reference in this
type test cannot be checked at run time." Instead, I have
refactored the code to remove the warning.
We still need to use the silencer plugin to suppress some warnings about
unused imports (because of compatibility between Scala 2.12 and 2.13),
but this means we no longer need the library, and therefore it is not a
transitive dependency that downstream consumers need to worry about.
CHANGELOG_BEGIN
CHANGELOG_END
* Add some comments around `@nowarn` support.
* language-support/scala: Fix a warning suppression.
* Revert to the default warnings.
Compatibility was complaining.
* compatibility: Use the same Scala version as the root.
It's confusing to get out of sync. `compatibility` was using Scala
2.12.12 while the root was on 2.12.13.
This brings them in sync to 2.12.13.
CHANGELOG_BEGIN
CHANGELOG_END
* Factor out Scala artifacts into scala_version.bzl.
* Bazel: Explicitly specify artifact hashes for Scala 2.13.
Not strictly necessary, as we _happen_ to depend on the same point
release as the Bazel Scala plugin, but this is incidental and we don't
want to bank on it.
* Upgrade Scala 2.12 to v2.12.13.
This is being pulled in anyway because of Maven/Gradle/etc's fun
"favor the most recent" resolution mechanism. The version of Akka we
depend upon transitively depends on Scala 2.12.13, so any downstream
consumers will see that as the Scala version required.
Bringing the Daml repo in line means no more confusion.
CHANGELOG_BEGIN
- [Scala Bindings] If you are using Daml on Scala 2.12, it now depends
on Scala v2.12.13 (from v2.12.12).
CHANGELOG_END
* Scala 2.12.13 is the default version in our pinned version of nixpkgs.
* Upgrade Scala 2.13's Wartremover version.
* Rename `scala_version_rule` to `scala_version_configure`.
* Add a test case to ensure the Scala versions are the same everywhere.
* Add tests for the Scala JAR versions in maven_install_*.json
* gatling-utils: Change the sort order of the expected CSV in tests.
I don't know why this changed, but it seems to be stable.
* compatibility: `scala_version` -> `scala_version_configure`.
* Bazel: Disable the Scala version tests on Windows.
* compatibility: Upgrade Wartremover to Scala 2.12.13.
052f69cde9
broke the structure of our protobufs by changing the import prefix
from "." to an empty string which ends up flattening the whole
file. This PR changes the default for tars to match the old
behavior. We can’t just change the default at the function level since
the prefix is used outside of pkg_tar and in those places an empty
string is required :(
changelog_begin
changelog_end
* Merge Maven uploads for different Scala versions
It turns out Maven will abort an existing staging operation if you
create a new one. This means our jobs race against each other. We
could try to fix that by either sequencing the jobs in a clever
way (annoying and can break things like rerunning if only parts
failed), or by creating more profiles (unclear if you can even have
two profiles for the same group id, even if you do, it’s annoying to
merge).
So in this PR I (grudgingly) merged both uploads into the Haskell
script. This isn’t all bad:
1. It moves some logic from bash embedded in yaml string literals into
Haskell code.
2. It duplicates some versions but it removes duplication in other
places so overall not too much worse.
3. It does however, make things slower. We don’t run this stuff in
parallel. That said, the release step is relatively small (< 5min) and
it only runs on Linux.
We could add CLI arguments to make the Scala versions configurable for
local development. Given that this is blocking releases, I wanted to
get something in that works first and then see what we need in that regard.
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
Some of our targets, e.g., //ledger/ledger-api-health have no
dependencies. On Scala 2.13, an empty classpath option produces a
bunch of confusing but afaict harmless warnings, on 2.12 it doesn’t
but there is still no point in passing an empty classpath so we can
omit it on both.
changelog_begin
changelog_end
* Replaced diff with diffx
* Explicitly added magnolia and mercator dependencies to fix automatical type class derivation
* CHANGELOG_BEGIN
CHANGELOG_END
* Removed unnecessary Diff type class instance for Seq[T]
* Removed ai.x.diff leftovers
* Added an explanatory comment for magnolia and mercator dependencies
* Formatted changes
* Added optional scaladoc parameter to Bazel's da_scala_library_suite()
* Formatted changes
This fixes Scaladoc and our pom file generation.
It also clears up the confusing error around gatling and removes a
redundant dependency on sbt (no idea why we had that in the first
place) both of which resulted in Scala 2.12 dependencies in our 2.13
lockfile which is obviously bad.
With this, we should now be ready to publish Scala 2.13 artifacts once
the ledger API test tool PR lands.
changelog_begin
changelog_end
* Draw the rest of the Scala 2.13 owl
Not quite but pretty close and this switches us over from inclusions
to exclusions which makes it much easier to track.
Ledger API test tool should be fixed by #8821. Non-repudiation needs a
tiny bit of work since unwrapArray doesn’t work the same on 2.13 but
shouldn’t be hard to fix.
changelog_begin
changelog_end
* Fix ScriptService tests
Those tests were all dumb. They asserted on a fixed order while the
function to sort the things was broken so we ended up with the random
Map order which is unsurprisingly not the same.
This is easily fixed by fixing the sort function.
There is also a second issue with query not sorting.
changelog_begin
changelog_end
* Turns out if you fix one test the next one breaks
And clearly nobody ever tested this or give this a second thought.
changelog_begin
changelog_end
We can’t upgrade to the latest version 1.35.0 since it depends on a
newer version of Bazel but we can at least go to 1.34.1. We also have
to patch absl and grpc itself to fix mingw support and some undeclared
inclusions on Windows. Both patches are taken from upstream PRs (the
mingw one has not been merged yet).
There is also a small patch to grpc-haskell-core. I’ve started working
on upstreaming that but the nix build system is a mess and I haven’t
quite managed to get it working there.
changelog_begin
changelog_end
Disabling it per target works nicely for compilation but it gets
annoying in ghci since the warnings are still triggered. We could
disable it everywhere but I think the warning is generally useful. I
tried patching proto3-suite to use DerivingStrategies but that doesn’t
work because haskell-src is dead and doesn’t support it. So for now
adding it to the per-file list seems like the best option.
changelog_begin
changelog_end
* Clean broken entries from the Bazel cache
This is hopefully a somewhat reasonable workaround for the "output not
created" errors that keep annoying us.
For now, this is just part of the hourly cronjob but we could move it
somewhere else if desired.
changelog_begin
changelog_end
* Fix GCS credentials
changelog_begin
changelog_end
Fixes#8573.
Fixes#8574.
We set the LOCA_ARCHIVE environment variable when a locales archive
exists and the variable is unset.
CHANGELOG_BEGIN
CHANGELOG_END
Fixes#8573.
Fixes#8574.
We ship the `locale-archive` packaged with `nix` along binaries and set
the LOCALE_ARCHIVE environment variable. This makes sure the shipped
binaries behave the same as in our dev-env.
CHANGELOG_BEGIN
CHANGELOG_END
This PR updates scalafmt and enables trailingCommas =
multiple. Unfortunately, scalafmt broke the version field which means
we cannot fully preserve the rest of the config. I’ve made some
attempts to stay reasonably close to the original config but couldn’t
find an exact equivalent in a lot of cases. I don’t feel strongly
about any of the settings so happy to change them to something else.
As announced, this will be merged on Saturday to avoid too many conflicts.
changelog_begin
changelog_end
The one thing that is still missing is making the generated Scala code
from the codegen compatible with Scala 2.13 so the examples are
excluded for now.
changelog_begin
changelog_end
* Port the rest //daml-lf/... to Scala 2.13
Draw the rest of the owl
changelog_begin
changelog_end
* Update daml-lf/encoder/src/main/scala/com/digitalasset/daml/lf/archive/testing/DamlLfEncoder.scala
Co-authored-by: Remy <remy.haemmerle@daml.com>
Co-authored-by: Remy <remy.haemmerle@daml.com>
This only compiles `damlc-bootstrap` as a Haskell binary and `damlc` is
a simple symlink with additional runfiles.
We now need to define `damlc@ghci` manually since it is no longer
defined automatically. The manual definition was tested with
```
$ bazel run //:damlc@ghci --define ghci_data=True
> :main ide -d
```
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
As we strive for more inclusiveness, we are becoming less comfortable
with historically-charged terms being used in our everyday work.
This is targeted for merge on Dec 26, _after_ the necessary
corresponding changes at both the GitHub and Azure Pipelines levels.
CHANGELOG_BEGIN
- DAML Connect development is now conducted from the `main` branch,
rather than the `master` one. If you had any dependency on the
digital-asset/daml repository, you will need to update this parameter.
CHANGELOG_END
* Port //daml-lf/data to Scala 2.13
changelog_begin
changelog_end
* factor common ImmArraySeq code to version-agnostic file
- ImmArraySeq itself is agnostic; the 2.12 and 2.13 versions contain
implementation mixins/superclasses for parts that must be specific. The 2.13
version will collapse into the agnostic version when 2.12 support is no longer
desired
* factor common InsertOrdMap code to version-agnostic file
- InsertOrdMap itself is agnostic; the 2.12 and 2.13 versions contain
implementation mixins/superclasses for parts that must be specific. The 2.13
version will collapse into the agnostic version when 2.12 support is no longer
desired
* factor common InsertOrdSet code to version-agnostic file
- InsertOrdSet itself is agnostic; the 2.12 and 2.13 versions contain
implementation mixins/superclasses for parts that must be specific. The 2.13
version will collapse into the agnostic version when 2.12 support is no longer
desired
* factor Map removal
* Move ImmArraySeq back into ImmArray
changelog_begin
changelog_end
* Type assertion instead of symbol
changelog_begin
changelog_end
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* Build //libs-scala/... on 2.13
One test is unfortunately disabled at the moment since I utterly
failed to figure out why I get a ClassNotFoundException on 2.13.
changelog_begin
changelog_end
* Copyright headers
changelog_begin
changelog_end
* I can’t bazel today
changelog_begin
changelog_end
* Apply suggestions from code review
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Update libs-scala/resources/src/main/2.13/com/daml/resources/UnitCanBuildFrom.scala
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* No split on view
changelog_begin
changelog_end
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* Add a Scala 2.13 build pipeline
This adds initial support for multiple Scala versions controlled via
the DAML_SCALA_VERSION env var and a CI job to make sure we don’t
regress. For now we only test //libs-scala/ports/... which seemed like
the easiest starting point I could find. We can incrementally expand
that over time.
changelog_begin
changelog_end
* Document pinning
changelog_begin
changelog_end
* Address review comments
changelog_begin
changelog_end
This is necessary to at least attempt an upgrade to 2.13 and
generally, I want to keep our rulesets up2date. rules-scala forces the
version of scalatest so we have to bump that at the same time.
This requires changes to basically all Scala test suites since the
import structure has changed and a bunch of things (primarily
scalacheck support) got split out.
Apologies for the giant PR, I don’t see a way to keep it smaller.
changelog_begin
changelog_end
Previously, each Protobuf source JAR was considered independent, and did
not require dependencies. This meant that the kvutils Protobuf sources
were published without the associated DAML-LF sources, making them
useless.
This change adds the source JARs as runtime dependencies to ensure that
when publishing a Protobuf source JAR, the release process will fail if
the dependencies are not also published.
CHANGELOG_BEGIN
- [Ledger API] We now publish Protobuf sources in JARs for DAML-LF
types to Maven Central, with the group "com.daml". The artifact names
follow the format "<name>_proto_jar".
CHANGELOG_END
* Bazel: Reduce the visibility of some Protobuf targets.
* Bazel: Make Protobuf JAR visibility parameterizable.
Default to private, and explicitly make it public where it's needed.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-api: Use `proto_jars`.
CHANGELOG_BEGIN
- [Ledger API] The Scala JARs containing the gRPC definitions no longer
contain the *.proto files used to generate the ScalaPB-based classes.
CHANGELOG_END
* Create a source JAR for *.proto files in `proto_jars`.
* ledger-api: Publish the protobuf sources as "ledger-api-proto".
CHANGELOG_BEGIN
- [Ledger API] The *.proto files containing the gRPC definitions are now
provided by a new Maven Central artifact, with the group "com.daml"
and the artifact name "ledger-api-proto".
CHANGELOG_END
* release: We don't need the "main-jar" option.
* Bazel: Proto JARs will always have a Maven artifact suffix.
* Bazel: Simplify Protobuf source file TAR and JAR targets.
* Bazel: Extract out Protobuf functions.
Turns out the `package_dir` is not what we're looking for. Omitting it
fixes the path.
_Before:_
```
1766 2010-01-01 00:00 protos-0.0.0/com/digitalasset/daml_lf_1_8/com/digitalasset/daml_lf_1_8/daml_lf.proto
```
_After:_
```
1766 2010-01-01 00:00 protos-0.0.0/com/digitalasset/daml_lf_1_8/daml_lf.proto
```
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Use ScalaPB to generate a Scala JAR for daml_kvutils.proto.
* Bazel: Delete the unused `da_java_binary` rule, and inline `_wrap_rule`.
* Bazel: Factor out Java/Scala protobuf class generation into a helper.
CHANGELOG_BEGIN
CHANGELOG_END
* daml-lf/archive: Use `proto_jars`.
* Bazel: Remove the visibility modifier from `proto_jars`.
It's too confusing. Just make everything public.
* daml-lf/archive: Push protobuf source tarballs into `proto_jars`.
* Bazel: Add comments to the various parts of `proto_jars`.
* daml-assistant: Do unpleasant things with `location` in Bazel.
These libraries have to appear on the linker command line after grpc and
gpr when linking against these. rules_haskell does currently not offer a
way to add extra libraries to the package configration file that are not
themselves Bazel targets. In case of libstdc++ and libws2_32 these are
system libraries that are not explicitly tracked by Bazel. As a
workaround we add these libraries to base's package configuration file.
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
daml-sdk-head: add optional sha information
This PR add an option, `--sha`, to `daml-sdk-head` so that it produces a
more accurate version number including the current git sha.
The main consequence is that calling `daml-sdk-head --sha` take
significantly longer than calling `daml-sdk-head`, because it needs to
recompile everything that depends on the version number.
> ## Wait, but why??
I started this work in support of an internal project that needed to
test against unreleased, and possibly unmerged, daml versions. However,
after further discussion I believe there is a better option for their
use-case. I've decided to still open this PR because the work was done
and there is no downside to it. It may still be useful if one wanted to
be able to maintain more than one non-released local version of daml.
CHANGELOG_BEGIN
CHANGELOG_END
* add silent_annotations option to da scala bazel functions
* use silent_annotations for several scala targets
* use silencer_plugin instead when the lib isn't used
* use silent_annotations for several more scala targets
* use silencer_lib for strange indirect requirement for running tests
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* silent_annotations support for scaladoc
Somehow I didn’t realize that this section was actually important and
removed due to the sdk mention and because it seemed a bit
useless. This has now resulted in Maven rejecting our
packages. Apologies for the mess.
changelog_begin
changelog_end
* Update rules_haskell
Removes the warning about Bazel 3.3.1 being too recent.
* Remove unused rules_haskell patch
changelog_begin
changelog_end
* fmt
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* add -Ywarn-unused to all scalac options
* remove some unused arguments
* remove some unused definitions
* remove some unused variable names
* suppress some unused variable names
* changeExtension doesn't use baseName
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* work around no plugins in scenario interpreter perf tests
* remove many more unused things
* remove more unused things, restore some used things
* remove more unused things, restore a couple signature mistakes
* missed import
* unused argument
* remove more unused loggingContexts
* some unused code in triggers
* some unused code in sandbox and kvutils
* some unused code in repl-service and daml-script
* some unused code in bindings-rxjava tests
* some unused code in triggers runner
* more comments on silent usages
- suggested by @cocreature; thanks
* fix missing reference in TestCommands
* more unused in triggers
* more unused in sandbox
* more unused in daml-script
* more unused in ledger-client tests
* more unused in triggers
* more unused in kvutils
* more unused in daml-script
* more unused in sandbox
* remove unused in ledger-api-test-tool
* suppress final special case for codegen unused warnings
.../com/daml/sample/mymain/ContractIdNT.scala:24: warning: parameter value ev 0 in method ContractIdNT Value is never used
implicit def `ContractIdNT Value`[a_a1dk](implicit `ev 0`: ` lfdomainapi`.Value[a_a1dk]): ` lfdomainapi`.Value[_root_.com.daml.sample.MyMain.ContractIdNT[a_a1dk]] = {
^
.../com/daml/sample/mymain/ContractIdNT.scala:41: warning: parameter value eva_a1dk in method ContractIdNT LfEncodable is never used
implicit def `ContractIdNT LfEncodable`[a_a1dk](implicit eva_a1dk: ` lfdomainapi`.encoding.LfEncodable[a_a1dk]): ` lfdomainapi`.encoding.LfEncodable[_root_.com.daml.sample.MyMain.ContractIdNT[a_a1dk]] = {
^
* one more unused in daml-script
* special scaladoc rules may need silencer, too
* unused in compatibility/sandbox-migration
* more commas, a different way to `find`
- suggested by @remyhaemmerle-da; thanks
* Factor out tar/gzip reproducibility flags
* use mktgz in package-app
* Bazel managed tar/gzip
* Remove quiet = True
As stated in the comment this is no longer required with Bazel >= 3.0.
* Build package-app as a sh_binary
This way Bazel will manage the runtime dependencies tar, gzip, mktgz,
and patchelf.
package-app.sh changes directory so it needs to make sure that all paths
are absolute and that the runfiles tree/manifest location is forwarded
to programs called by package-app.sh.
* Avoid file path too long errors
* Fix readlink -f on MacOS
* Document abspath
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Use Bazel builtin pkg_tar rule
* Use @rules_pkg//:pkg.bzl%pkg_tar
The pkg_tar rule builtin to Bazel has been deprecated.
See https://docs.bazel.build/versions/master/be/pkg.html
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* add -Xlint:doc-detached
- reverts 1feae964e3 from #6798
* attach several scaladocs where they'll actually be included
* no changelog
* attach several more scaladocs where they'll actually be included
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
GHC hates its users and defaults to optimizing out assertions. We
fixed that in Buck at some point but clearly that got lost when
migrating to Bazel.
Turns out enabling assertions catches bugs. This insight was brought to
you from the people that also brought you “Turns out writing tests
catches bugs”.
fixes#5624
changelog_begin
changelog_end
This does not change the version of rules_apple, it only pins the http
archive instead of fetching via git tag.
To avoid Bazel warnings of the following form since Bazel 3.3.1
```
DEBUG: Rule 'build_bazel_rules_apple' indicated that a canonical reproducible form can be obtained by modifying arguments commit = "ff6a37b24fcbbd525a5bf61692a12c810d0ee3c1", shallow_since = "1559833568 -0700" and dropping ["tag"]
DEBUG: Repository build_bazel_rules_apple instantiated at:
no stack (--record_rule_instantiation_callstack not enabled)
Repository rule git_repository defined at:
/home/aj/.cache/bazel/_bazel_aj/f66bee630c6a2cd906f92a0f5cdf8769/external/bazel_tools/tools/build_defs/repo/git.bzl:195:33: in <toplevel>
```
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Upgrade nixpkgs revision
* Remove unused minio
It used to be used as a gateway to push the Nix cache to GCS, but has
since been replaced by nix-store-gcs-proxy.
* Update Bazel on Windows
changelog_begin
changelog_end
* Fix hlint warnings
The nixpkgs update implied an hlint update which enabled new warnings.
* Fix "Error applying patch"
Since Bazel 2.2.0 the order of generating `WORKSPACE` and `BUILD` files
and applying patches has been reversed. The allows users to define
patches to these files that will not be immediately overwritten.
However, it also means that patches on another repository's original
`WORKSPACE` file will likely become invalid.
* a948eb7255
* https://github.com/bazelbuild/bazel/issues/10681
Hint: If you're generating a patch with `git` then you can use the
following command to exclude the `WORKSPACE` file.
```
git diff ':(exclude)WORKSPACE'
```
* Update rules_nixpkgs
* nixpkgs location expansion escaping
* Drop --noincompatible_windows_native_test_wrapper
* client_server_test using sh_inline_test
client_server_test used to produce an executable shell script in form of
a text file output. However, since the removal of
`--noincompatible_windows_native_test_wrapper` this no longer works on
Windows since `.sh` files are not directly executable on Windows.
This change fixes the issue by producing the script file in a dedicated
rule and then wrapping it in a `sh_test` rule which also works on
Windows.
* daml_test using sh_inline_test
* daml_doc_test using sh_inline_test
* _daml_validate_test using sh_inline_test
* damlc_compile_test using sh_inline_test
* client_server_test find .exe on Windows
* Bump Windows cache for Bazel update
Remove `clean --expunge` after merge.
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* set many extra scalac -Xlint options for all Scala projects
CHANGELOG_BEGIN
CHANGELOG_END
* move NoCopy to its own file
package.scala:18: warning: it is not recommended to define classes/objects inside of package objects.
If possible, define trait NoCopy in package data instead.
trait NoCopy {
^
* move more traits, classes, and objects to proper packages
- note that `package` is itself a scoping construct, so if your reason
is the apparent aesthetic of placing a bunch of things in one `package
object`, that is easily remedied by deleting the `object` keyword
* fix some type-parameter-shadow warnings
- I'm generally in favor of sensible name-shadowing, following the
"deliberately hide variables that should not be accessed here" school
of thought. But I think type name shadowing isn't quite as valuable
and more likely to confuse than general variable shadowing, so have
experimentally linted it out.
Example warning:
EventsTableFlatEventsRangeQueries.scala:11: warning: type parameter
Offset defined in trait EventsTableFlatEventsRangeQueries shadows class
Offset defined in package v1. You may want to rename your type
parameter, or possibly remove it.
private[events] sealed trait EventsTableFlatEventsRangeQueries[Offset] {
^
* fix more package-object-classes warnings
* fix an inaccessible warning
ContractsService.scala:197: warning: method searchDb in class ContractsService references private class ContractsFetch.
Classes which cannot access ContractsFetch may be unable to override searchDb.
def searchDb(dao: dbbackend.ContractDao, fetch: ContractsFetch)(
^
* enable -Xlint:infer-any
- continuing the saga of #6116, #6132
* enable -explaintypes for more detailed type errors
* missed header for NoCopy; probably should have left it in the package file
* misspelling in comment
* revert -Xlint:doc-detached
- there are a lot of these fixes, and they are noisy, so shifting to a
separate PR
- thanks to @leo-da for pointing out
* Update rules_haskell hie-bios support
* Decouple Haskell ghcide and DAML ghcide
Creates a separate `stack_snapshot` to pull in `ghcide` for the Haskell
IDE use case independent of the `ghcide` for DAML. This allows to update
these two `ghcide` instances independently. As DAML uses `ghcide` the
library updates can be involved if the API experienced breaking changes.
At the same time we may wish to update `ghcide` for Haskell earlier to
make use of new features and stay compatible with rules_haskell's ghcide
support.
* Fix Haddock warnings reported by ghcide
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Buildifier now comes with a handy attachment to catch single `\`
characters inside strings and replace them with `\\` if the escape
sequence is invalid. Skylark/Python will do this at runtime anyway; this
just makes it clearer what the actual behavior is.
I needed to change `\` characters at the end of lines to `\\` manually
in order to stop Buildifier from simply concatenating the lines
together. Everything else was automatic.
CHANGELOG_BEGIN
CHANGELOG_END
* Update rules_haskell
* Pin stack_snapshot repositories
* Document stack_snapshot_json
CHANGELOG_BEGIN
CHANGELOG_END
* Don't pin stack_snapshot on Windows
The lock file is generated on Unix and includes unix specific
dependencies, e.g. `unix`. Most developers don't have easy access to a
Windows machine, so regenerating the lock file for Windows would be
inconvenient.
* upgrade stack 2.1.3 --> 2.3.1 on Windows
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* upgrade to wartremover 2.4.9
* simplify wart list and list JavaConversions as disabled
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* delete long-obsolete, contradictory comment
* also upgrade wartremover in compatibility (leaving aside maven_install.json)
* update compatibility maven_install.json to match
* add -Xsource:2.13, -Ypartial-unification to common_scalacopts
* add now-referenced scalaz-core where needed
* work around bad type signatures in scalatest Aggregating, Containing
* unused Any suppression
* work around bad partial-unification wrought by type alias
* remove unused Conversions import
- not required in 4f68cfc480 either, so unsure how it's survived this long
* work around Future.traverse; remove unused show import
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* remove unused bounds
* remove -Ypartial-unification and -Xsource:2.13 where they were explicitly passed
* longer comment on what the options do
- suggested by @stefanobaghino-da; thanks
* forget Future.traverse, just use scalaz, it knows how to do this
* disable Any wart
* first pass removal of Any suppressions for false positives
* second pass removal of Any suppressions for false positives
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* third pass removal of Any suppressions for false positives
* fourth pass removal of Any suppressions for false positives
* reformat newly single-suppressions into single lines
- suggested by @SamirTalwar-DA; thanks
The workspace for the vendored node wrapper script `@nodejs_linux_amd64`
did previously not record a dependency on the nixpkgs provided node
workspace. This patch enforces that dependency by introducing a dummy
read of the vendored node binary.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* update rules_nixpkgs
* Use hermetic nixpkgs cc toolchain
CHANGELOG_BEGIN
CHANGELOG_END
* Work around Bazel's cc toolchain autodetection
* Use --crosstool_top for hermetic cc toolchain
When using --incompatible_enable_cc_toolchain_resolution instead
cc actions still depend on
`external/local_config_cc/builtin_include_directory_paths`
as well as
`external/nixpkgs_cc_toolchain_config/builtin_include_directory_paths`.
* override local_config_cc
* remove unused attribute
* Fix posix toolchain on Windows
* nixpkgs cc toolchain not on Windows
* Fix nixpkgs cc toolchain on MacOS
* nixpkgs cc toolchain uses bin/cc
* Use darwin.binutils on MacOS
* Remove clang(++) and gcc (g++) symlinks
The toolchain only considers `bin/cc` and having the other symlinks
around could lead to confusion
* Use hermetic toolchain in compatibility workspace
* Avoid empty linker flags
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
This should fix the following issue that we see constantly on CI:
```
ld: warning: object file _ was built for newer OSX version (10.15) than being linked (10.14)
```
The issue was that the CC toolchain was not fully used in
haskell_cabal_package. --with-gcc (which is really --with-cc) only
applies when Cabal is calling the C compiler. However, in most cases
it is actually GHC itself which calls the C compiler. To make sure
that the right compiler is used in those cases, we have to pass
`-pgmc` and friends to GHC. This matches what rules_haskell does for
non cabal targets.
changelog_begin
changelog_end
* Make compat tests work on windows
This required some changes to the daml_sdk rule since the read-only
installation by the assistant breaks Bazel completely. We could only
apply those changes on Windows but I think I prefer the consistency
across platforms here over trying to stay close to how the SDK is
installed on user machines given that the SDK installation is not
something we’ve had issues with.
I’ve excluded the postgresql tests for now. I don’t expect them to be
particularly hard to fix but I’ve already spent almost 2 days on this
and having some tests run on Windows seems like a clear improvement
over running no tests on Windows :)
changelog_begin
changelog_end
* Remove todo
changelog_begin
changelog_end
This PR extends the existing Linux compatibility tests to run on macOS
too. Fixes#5692.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
This should hopefully fix the issues we have been seeing on CI. While
I’m not super keen on including non-upstreamable patches it seems
better than having CI be flaky.
changelog_begin
changelog_end
* Fix redirects, java-bindings javadoc, and live-preview.sh
- javadoc_library now supports sources from filegroups as well
- //language-support/java:javadoc now generates javadoc for ledger-api, java bindings, rxjava bindings
- live-preview.sh refers to the correct javadoc target //language-support/java:javadoc
- removed leading / from redirects.map
* Only generate daml-lf javadocs if not on windows
CHANGELOG_BEGIN
CHANGELOG_END
Packages com.digitalasset.daml and com.daml have been unified under com.daml
Ledger API and DAML-LF DEV protos have also been moved from `com/digitalasset`
to `com/daml` on the file system.
Protos for already released DAML LF versions (1.6, 1.7, 1.8) stay in the
package `com.digitalasset`.
CHANGELOG_BEGIN
[SDK] All Java and Scala packages starting with
``com.digitalasset.daml`` and ``com.digitalasset`` are now consolidated
under ``com.daml``. Simply changing imports should be enough to
migrate your code.
CHANGELOG_END
* bazel_tools: Set `unused_dependency_checker_mode` in one place.
* bazel_tools: Set the default max heap size for Scala processes to 2GB.
And the default initial max heap size to 512MB.
CHANGELOG_BEGIN
CHANGELOG_END
* bazel_tools: Set the `scalac` heap size to 2GB and stack size to 2MB.
* bazel_tools: Delete `da_scala_macro_library`, as it's unused.
* bazel_tools: Revert the description of `da_scala_library_suite`.
Misread it.
* Use com.daml as groupId for all artifacts
CHANGELOG_BEGIN
[SDK] Changed the groupId for Maven artifacts to ``com.daml``.
CHANGELOG_END
* Add 2 additional maven related checks to the release binary
1. Check that all maven upload artifacts use com.daml as the groupId
2. Check that all maven upload artifacts have a unique artifactId
* Address @cocreature's comments in https://github.com/digital-asset/daml/pull/5272#pullrequestreview-385026181
* Remove unused da_doc_package
The only use-site was `//compiler/daml-licenses:daml-licenses`, which
itself was unused.
* Remove unused notices-gen
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Update rules_haskell
The workaround for linking against `Cffi` in the REPL has been
upstreamed in a more generalized form.
CHANGELOG_BEGIN
CHANGELOG_END
* ghcide: Use rules_haskell's hie-bios support
* Document `ghcide` Bazel integration
* Rename files to match module names
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
This removes the sample/reference implementation of kvutils
InMemoryKVParticipantState.
This used to be the only implementation of kvutils, but now with the
simplified kvutils api we have ledger-on-memory and ledger-on-sql.
InMemoryKVParticipantState was also used for the ledger dump utility,
which now uses ledger-on-memory.
* Runner now supports a multi participant configuration
This change removes the "extra participants" config and goes for consistent
participant setup with --participant.
* Run all conformance tests in the repository in verbose mode.
This means we'll print stack traces on error, which should make it
easier to figure out what's going on with flaky tests on CI.
This doesn't change the default for other users of the
ledger-api-test-tool; we just add the flag for:
- ledger-api-test-tool-on-canton
- ledger-on-memory
- ledger-on-sql
- sandbox
Fixes#4225.
CHANGELOG_BEGIN
CHANGELOG_END
* Bump rules_haskell
Still checking if that helps with GHC 8.8 but we should upgrade this
either way.
changelog_begin
changelog_end
* disable grpc patch
* shut up buildifier
* delete unused ghci grpc patch
* Fix Cffi library not found issues
* Update deps.bzl
Co-Authored-By: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
Context
=======
After multiple discussions about our current release schedule and
process, we've come to the conclusion that we need to be able to make a
distinction between technical snapshots and marketing releases. In other
words, we need to be able to create a bundle for early adopters to test
without making it an officially-supported version, and without
necessarily implying everyone should go through the trouble of
upgrading. The underlying goal is to have less frequent but more stable
"official" releases.
This PR is a proposal for a new release process designed under the
following constraints:
- Reuse as much as possible of the existing infrastructure, to minimize
effort but also chances of disruptions.
- Have the ability to create "snapshot"/"nightly"/... releases that are
not meant for general public consumption, but can still be used by savvy
users without jumping through too many extra hoops (ideally just
swapping in a slightly-weirder version string).
- Have the ability to promote an existing snapshot release to "official"
release status, with as few changes as possible in-between, so we can be
confident that the official release is what we tested as a prerelease.
- Have as much of the release pipeline shared between the two types of
releases, to avoid discovering non-transient problems while trying to
promote a snapshot to an official release.
- Triggerring a release should still be done through a PR, so we can
keep the same approval process for SOC2 auditability.
The gist of this proposal is to replace the current `VERSION` file with
a `LATEST` file, which would have the following format:
```
ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7
```
This file would be maintained with a script to reduce manual labor in
producing the version string. Other than that, the process will be
largely the same, with releases triggered by changes to this `LATEST`
and the release notes files.
Version numbers
===============
Because one of the goals is to reduce the velocity of our published
version numbers, we need a different version scheme for our snapshot
releases. Fortunately, most version schemes have some support for that;
unfortunately, the SDK sits at the intersection of three different
version schemes that have made incompatible choices. Without going into
too much detail:
- Semantic versioning (which we chose as the version format for the SDK
version number) allows for "prerelease" version numbers as well as
"metadata"; an example of a complete version string would be
`1.2.3-nightly.201+server12.43`. The "main" part of the version string
always has to have 3 numbers separated by dots; the "prerelease"
(after the `-` but before the `+`) and the "metadata" (after the `+`)
parts are optional and, if present, must consist of one or more segments
separated by dots, where a segment can be either a number or an
alphanumeric string. In terms of ordering, metadata is irrelevant and
any version with a prerelease string is before the corresponding "main"
version string alone. Amongst prereleases, segments are compared in
order with purely numeric ones compared as numbers and mixed ones
compared lexicographically. So 1.2.3 is more recent than 1.2.3-1,
which is itself less recent than 1.2.3-2.
- Maven version strings are any number of segments separated by a `.`, a
`-`, or a transition between a number and a letter. Version strings
are compared element-wise, with numeric segments being compared as
numbers. Alphabetic segments are treated specially if they happen to be
one of a handful of magic words (such as "alpha", "beta" or "snapshot"
for example) which count as "qualifiers"; a version string with a
qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`,
which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a
special ordering amongst qualifiers. Other alphabetic segments are
compared alphabetically and count as being "after" their prefix
(`1.2.3-really-final-this-time` counts as being released after `1.2.3`).
- GHC package numbers are comprised of any number of numeric segments
separated by `.`, plus an optional (though deprecated) alphanumeric
"version tag" separated by a `-`. I could not find any official
documentation on ordering for the version tag; numeric segments are
compared as numbers.
- npm uses semantic versioning so that is covered already.
After much more investigation than I'd care to admit, I have come up
with the following compromise as the least-bad solution. First,
obviously, the version string for stable/marketing versions is going to
be "standard" semver, i.e. major.minor.patch, all numbers, which works,
and sorts as expected, for all three schemes. For snapshot releases, we
shall use the following (semver) format:
```
0.13.53-alpha.20200214.3025.ef5d32b7
```
where the components are, respectively:
- `0.13.53`: the expected version string of the next "stable" release.
- `alpha`: a marker that hopefully scares people enough.
- `20200214`: the date of the release commit, which _MUST_ be on
master.
- `3025`: the number of commits in master up to the release commit
(included). Because we have a linear, append-only master branch, this
uniquely identifies the commit.
- `ef5d32b7ù : the first 8 characters of the release commit sha. This is
not strictly speaking necessary, but makes it a lot more convenient to
identify the commit.
The main downsides of this format are:
1. It is not a valid format for GHC packages. We do not publish GHC
packages from the SDK (so far we have instead opted to release our
Haskell code as separate packages entirely), so this should not be an
issue. However, our SDK version currently leaks to `ghc-pkg` as the
version string for the stdlib (and prim) packages. This PR addresses
that by tweaking the compiler to remove the offending bits, so `ghc-pkg`
would see the above version number as `0.13.53.20200214.3025`, which
should be enough to uniquely identify it. Note that, as far as I could
find out, this number would never be exposed to users.
2. It is rather long, which I think is good from a human perspective as
it makes it more scary. However, I have been told that this may be
long enough to cause issues on Windows by pushing us past the max path
size limitation of that "OS". I suggest we try it and see what
happens.
The upsides are:
- It clearly indicates it is an unstable release (`alpha`).
- It clearly indicates how old it is, by including the date.
- To humans, it is immediately obvious which version is "later" even if
they have the same date, allowing us to release same-day patches if
needed. (Note: that is, commits that were made on the same day; the
release date itself is irrelevant here.)
- It contains the git sha so the commit built for that release is
immediately obvious.
- It sorts correctly under all schemes (modulo the modification for
GHC).
Alternatives I considered:
- Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would
be accepted by all schemes, but will not sort as expected under semantic
versioning (though Maven will be fine). I have no idea how it will sort
under GHC.
- Not having any non-numeric component, e.g. `0.13.53.20200214.3025`.
This is not valid semantic versioning and is therefore rejected by
npm.
- Not having detailed info: just go with `0.13.53-snapshot`. This is
what is generally done in the Java world, but we then lose track of what
version is actually in use and I'm concerned about bug reports. This
would also not let us publish to the main Maven repo (at least not more
than once), as artifacts there are supposed to be immutable.
- No having a qualifier: `0.13.53-3025` would be acceptable to all three
version formats. However, it would not clearly indicate to humans that
it is not meant as a stable version, and would sort differently under
semantic versioning (which counts it as a prerelease, i.e. before
`0.13.53`) than under maven (which counts it as a patch, so after
`0.13.53`).
- Just counting releases: `0.13.53-alpha.1`, where we just count the
number of prereleases in-between `0.13.52` and the next. This is
currently the fallback plan if Windows path length causes issues. It
would be less convenient to map releases to commits, but it could still
be done via querying the history of the `LATEST` file.
Release notes
=============
> Note: We have decided not to have release notes for snapshot releases.
Release notes are a bit tricky. Because we want the ability to make
snapshot releases, then later on promote them to stable releases, it
follows that we want to build commits from the past. However, if we
decide post-hoc that a commit is actually a good candidate for a
release, there is no way that commit can have the appropriate release
notes: it cannot know what version number it's getting, and, moreover,
we now track changes in commit messages. And I do not think anyone wants
to go back to the release notes file being a merge bottleneck.
But release notes need to be published to the releases blog upon
releasing a stable version, and the docs website needs to be updated and
include them.
The only sensible solution here is to pick up the release notes as of
the commit that triggers the release. As the docs cron runs
asynchronously, this means walking down the git history to find the
relevant commit.
> Note: We could probably do away with the asynchronicity at this point.
> It was originally included to cover for the possibility of a release
> failing. If we are releasing commits from the past after they have been
> tested, this should not be an issue anymore. If the docs generation were
> part of the synchronous release step, it would have direct access to the
> correct release notes without having to walk down the git history.
>
> However, I think it is more prudent to keep this change as a future step,
> after we're confident the new release scheme does indeed produce much more
> reliable "stable" releases.
New release process
===================
Just like releases are currently controlled mostly by detecting
changes to the `VERSION` file, the new process will be controlled by
detecting changes to the `LATEST` file. The format of that file will
include both the version string and the corresponding SHA.
Upon detecting a change to the `LATEST` file, CI will run the entire
release process, just like it does now with the VERSION file. The main
differences are:
1. Before running the release step, CI will checkout the commit
specified in the LATEST file. This requires separating the release
step from the build step, which in my opinion is cleaner anyway.
2. The `//:VERSION` Bazel target is replaced by a repository rule
that gets the version to build from an environment variable, with a
default of `0.0.0` to remain consistent with the current `daml-head`
behaviour.
Some of the manual steps will need to be skipped for a snapshot release.
See amended `release/RELEASE.md` in this commit for details.
The main caveat of this approach is that the official release will be a
different binary from the corresponding snapshot. It will have been
built from the same source, but with a different version string. This is
somewhat mitigated by Bazel caching, meaning any build step that does
not depend on the version string should use the cache and produce
identical results. I do not think this can be avoided when our artifact
includes its own version number.
I must note, though, that while going through the changes required after
removing the `VERSION` file, I have been quite surprised at the sheer number of
things that actually depend on the SDK version number. I believe we should
look into reducing that over time.
CHANGELOG_BEGIN
CHANGELOG_END
* Avoid opening a server to the world when finding a free port.
This is very annoying on macOS because we get a focus-stealing popup for
a split second, asking for permission to allow the server through the
firewall. The popup pretty much always disappears before it can even be
read, when the server is closed.
This is almost certainly not an attack vector, because:
- we only do this in tests,
- the server is open for only a few milliseconds,
- nothing is served,
- and finding the port is tricky, because it's effectively random.
Nevertheless, it's very annoying.
CHANGELOG_BEGIN
CHANGELOG_END
* Extract a Bazel package for finding free ports.
We seem to do it in 4 different places, which I think is enough to
remove the duplication.
As mentioned in the title, this is still very experimental and needs
more work before we want to advertise it. However, the code is in a
somewhat reasonable shape, there are tests and I think even in the
current state it is already useful. Also this PR is already getting
very large so I don’t want to hold off much longer before merging this.
It is included in the SDK but hidden from `damlc --help` and `daml
--help` until the most pressing issues are addressed (primarily around
making sure that it doesn’t just shut down if you have a type error
and better error messages in general).
changelog_begin
changelog_end
This is a bit ugly but after spending some time digging into the
issues in rules_haskell around data-files, this seems like the most
sensible option especially given that we also want to ship them in the
SDK which woud require additional work even if we do fix it in
rules_haskell.
fixes#4457
changelog_begin
changelog_end
* Disable http2 with Nix to work around segfaults
CHANGELOG_BEGIN
CHANGELOG_END
* Disable http2 in dev-env calls to nix-build as well
Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
