* Use Bazel builtin pkg_tar rule
* Use @rules_pkg//:pkg.bzl%pkg_tar
The pkg_tar rule builtin to Bazel has been deprecated.
See https://docs.bazel.build/versions/master/be/pkg.html
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* add -Xlint:doc-detached
- reverts 1feae964e3 from #6798
* attach several scaladocs where they'll actually be included
* no changelog
* attach several more scaladocs where they'll actually be included
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
GHC hates its users and defaults to optimizing out assertions. We
fixed that in Buck at some point but clearly that got lost when
migrating to Bazel.
Turns out enabling assertions catches bugs. This insight was brought to
you from the people that also brought you “Turns out writing tests
catches bugs”.
fixes#5624
changelog_begin
changelog_end
This does not change the version of rules_apple, it only pins the http
archive instead of fetching via git tag.
To avoid Bazel warnings of the following form since Bazel 3.3.1
```
DEBUG: Rule 'build_bazel_rules_apple' indicated that a canonical reproducible form can be obtained by modifying arguments commit = "ff6a37b24fcbbd525a5bf61692a12c810d0ee3c1", shallow_since = "1559833568 -0700" and dropping ["tag"]
DEBUG: Repository build_bazel_rules_apple instantiated at:
no stack (--record_rule_instantiation_callstack not enabled)
Repository rule git_repository defined at:
/home/aj/.cache/bazel/_bazel_aj/f66bee630c6a2cd906f92a0f5cdf8769/external/bazel_tools/tools/build_defs/repo/git.bzl:195:33: in <toplevel>
```
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Upgrade nixpkgs revision
* Remove unused minio
It used to be used as a gateway to push the Nix cache to GCS, but has
since been replaced by nix-store-gcs-proxy.
* Update Bazel on Windows
changelog_begin
changelog_end
* Fix hlint warnings
The nixpkgs update implied an hlint update which enabled new warnings.
* Fix "Error applying patch"
Since Bazel 2.2.0 the order of generating `WORKSPACE` and `BUILD` files
and applying patches has been reversed. The allows users to define
patches to these files that will not be immediately overwritten.
However, it also means that patches on another repository's original
`WORKSPACE` file will likely become invalid.
* a948eb7255
* https://github.com/bazelbuild/bazel/issues/10681
Hint: If you're generating a patch with `git` then you can use the
following command to exclude the `WORKSPACE` file.
```
git diff ':(exclude)WORKSPACE'
```
* Update rules_nixpkgs
* nixpkgs location expansion escaping
* Drop --noincompatible_windows_native_test_wrapper
* client_server_test using sh_inline_test
client_server_test used to produce an executable shell script in form of
a text file output. However, since the removal of
`--noincompatible_windows_native_test_wrapper` this no longer works on
Windows since `.sh` files are not directly executable on Windows.
This change fixes the issue by producing the script file in a dedicated
rule and then wrapping it in a `sh_test` rule which also works on
Windows.
* daml_test using sh_inline_test
* daml_doc_test using sh_inline_test
* _daml_validate_test using sh_inline_test
* damlc_compile_test using sh_inline_test
* client_server_test find .exe on Windows
* Bump Windows cache for Bazel update
Remove `clean --expunge` after merge.
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* set many extra scalac -Xlint options for all Scala projects
CHANGELOG_BEGIN
CHANGELOG_END
* move NoCopy to its own file
package.scala:18: warning: it is not recommended to define classes/objects inside of package objects.
If possible, define trait NoCopy in package data instead.
trait NoCopy {
^
* move more traits, classes, and objects to proper packages
- note that `package` is itself a scoping construct, so if your reason
is the apparent aesthetic of placing a bunch of things in one `package
object`, that is easily remedied by deleting the `object` keyword
* fix some type-parameter-shadow warnings
- I'm generally in favor of sensible name-shadowing, following the
"deliberately hide variables that should not be accessed here" school
of thought. But I think type name shadowing isn't quite as valuable
and more likely to confuse than general variable shadowing, so have
experimentally linted it out.
Example warning:
EventsTableFlatEventsRangeQueries.scala:11: warning: type parameter
Offset defined in trait EventsTableFlatEventsRangeQueries shadows class
Offset defined in package v1. You may want to rename your type
parameter, or possibly remove it.
private[events] sealed trait EventsTableFlatEventsRangeQueries[Offset] {
^
* fix more package-object-classes warnings
* fix an inaccessible warning
ContractsService.scala:197: warning: method searchDb in class ContractsService references private class ContractsFetch.
Classes which cannot access ContractsFetch may be unable to override searchDb.
def searchDb(dao: dbbackend.ContractDao, fetch: ContractsFetch)(
^
* enable -Xlint:infer-any
- continuing the saga of #6116, #6132
* enable -explaintypes for more detailed type errors
* missed header for NoCopy; probably should have left it in the package file
* misspelling in comment
* revert -Xlint:doc-detached
- there are a lot of these fixes, and they are noisy, so shifting to a
separate PR
- thanks to @leo-da for pointing out
* Update rules_haskell hie-bios support
* Decouple Haskell ghcide and DAML ghcide
Creates a separate `stack_snapshot` to pull in `ghcide` for the Haskell
IDE use case independent of the `ghcide` for DAML. This allows to update
these two `ghcide` instances independently. As DAML uses `ghcide` the
library updates can be involved if the API experienced breaking changes.
At the same time we may wish to update `ghcide` for Haskell earlier to
make use of new features and stay compatible with rules_haskell's ghcide
support.
* Fix Haddock warnings reported by ghcide
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Buildifier now comes with a handy attachment to catch single `\`
characters inside strings and replace them with `\\` if the escape
sequence is invalid. Skylark/Python will do this at runtime anyway; this
just makes it clearer what the actual behavior is.
I needed to change `\` characters at the end of lines to `\\` manually
in order to stop Buildifier from simply concatenating the lines
together. Everything else was automatic.
CHANGELOG_BEGIN
CHANGELOG_END
* Update rules_haskell
* Pin stack_snapshot repositories
* Document stack_snapshot_json
CHANGELOG_BEGIN
CHANGELOG_END
* Don't pin stack_snapshot on Windows
The lock file is generated on Unix and includes unix specific
dependencies, e.g. `unix`. Most developers don't have easy access to a
Windows machine, so regenerating the lock file for Windows would be
inconvenient.
* upgrade stack 2.1.3 --> 2.3.1 on Windows
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* upgrade to wartremover 2.4.9
* simplify wart list and list JavaConversions as disabled
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* delete long-obsolete, contradictory comment
* also upgrade wartremover in compatibility (leaving aside maven_install.json)
* update compatibility maven_install.json to match
* add -Xsource:2.13, -Ypartial-unification to common_scalacopts
* add now-referenced scalaz-core where needed
* work around bad type signatures in scalatest Aggregating, Containing
* unused Any suppression
* work around bad partial-unification wrought by type alias
* remove unused Conversions import
- not required in 4f68cfc480 either, so unsure how it's survived this long
* work around Future.traverse; remove unused show import
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* remove unused bounds
* remove -Ypartial-unification and -Xsource:2.13 where they were explicitly passed
* longer comment on what the options do
- suggested by @stefanobaghino-da; thanks
* forget Future.traverse, just use scalaz, it knows how to do this
* disable Any wart
* first pass removal of Any suppressions for false positives
* second pass removal of Any suppressions for false positives
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* third pass removal of Any suppressions for false positives
* fourth pass removal of Any suppressions for false positives
* reformat newly single-suppressions into single lines
- suggested by @SamirTalwar-DA; thanks
The workspace for the vendored node wrapper script `@nodejs_linux_amd64`
did previously not record a dependency on the nixpkgs provided node
workspace. This patch enforces that dependency by introducing a dummy
read of the vendored node binary.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* update rules_nixpkgs
* Use hermetic nixpkgs cc toolchain
CHANGELOG_BEGIN
CHANGELOG_END
* Work around Bazel's cc toolchain autodetection
* Use --crosstool_top for hermetic cc toolchain
When using --incompatible_enable_cc_toolchain_resolution instead
cc actions still depend on
`external/local_config_cc/builtin_include_directory_paths`
as well as
`external/nixpkgs_cc_toolchain_config/builtin_include_directory_paths`.
* override local_config_cc
* remove unused attribute
* Fix posix toolchain on Windows
* nixpkgs cc toolchain not on Windows
* Fix nixpkgs cc toolchain on MacOS
* nixpkgs cc toolchain uses bin/cc
* Use darwin.binutils on MacOS
* Remove clang(++) and gcc (g++) symlinks
The toolchain only considers `bin/cc` and having the other symlinks
around could lead to confusion
* Use hermetic toolchain in compatibility workspace
* Avoid empty linker flags
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
This should fix the following issue that we see constantly on CI:
```
ld: warning: object file _ was built for newer OSX version (10.15) than being linked (10.14)
```
The issue was that the CC toolchain was not fully used in
haskell_cabal_package. --with-gcc (which is really --with-cc) only
applies when Cabal is calling the C compiler. However, in most cases
it is actually GHC itself which calls the C compiler. To make sure
that the right compiler is used in those cases, we have to pass
`-pgmc` and friends to GHC. This matches what rules_haskell does for
non cabal targets.
changelog_begin
changelog_end
* Make compat tests work on windows
This required some changes to the daml_sdk rule since the read-only
installation by the assistant breaks Bazel completely. We could only
apply those changes on Windows but I think I prefer the consistency
across platforms here over trying to stay close to how the SDK is
installed on user machines given that the SDK installation is not
something we’ve had issues with.
I’ve excluded the postgresql tests for now. I don’t expect them to be
particularly hard to fix but I’ve already spent almost 2 days on this
and having some tests run on Windows seems like a clear improvement
over running no tests on Windows :)
changelog_begin
changelog_end
* Remove todo
changelog_begin
changelog_end
This PR extends the existing Linux compatibility tests to run on macOS
too. Fixes#5692.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
This should hopefully fix the issues we have been seeing on CI. While
I’m not super keen on including non-upstreamable patches it seems
better than having CI be flaky.
changelog_begin
changelog_end
* Fix redirects, java-bindings javadoc, and live-preview.sh
- javadoc_library now supports sources from filegroups as well
- //language-support/java:javadoc now generates javadoc for ledger-api, java bindings, rxjava bindings
- live-preview.sh refers to the correct javadoc target //language-support/java:javadoc
- removed leading / from redirects.map
* Only generate daml-lf javadocs if not on windows
CHANGELOG_BEGIN
CHANGELOG_END
Packages com.digitalasset.daml and com.daml have been unified under com.daml
Ledger API and DAML-LF DEV protos have also been moved from `com/digitalasset`
to `com/daml` on the file system.
Protos for already released DAML LF versions (1.6, 1.7, 1.8) stay in the
package `com.digitalasset`.
CHANGELOG_BEGIN
[SDK] All Java and Scala packages starting with
``com.digitalasset.daml`` and ``com.digitalasset`` are now consolidated
under ``com.daml``. Simply changing imports should be enough to
migrate your code.
CHANGELOG_END
* bazel_tools: Set `unused_dependency_checker_mode` in one place.
* bazel_tools: Set the default max heap size for Scala processes to 2GB.
And the default initial max heap size to 512MB.
CHANGELOG_BEGIN
CHANGELOG_END
* bazel_tools: Set the `scalac` heap size to 2GB and stack size to 2MB.
* bazel_tools: Delete `da_scala_macro_library`, as it's unused.
* bazel_tools: Revert the description of `da_scala_library_suite`.
Misread it.
* Use com.daml as groupId for all artifacts
CHANGELOG_BEGIN
[SDK] Changed the groupId for Maven artifacts to ``com.daml``.
CHANGELOG_END
* Add 2 additional maven related checks to the release binary
1. Check that all maven upload artifacts use com.daml as the groupId
2. Check that all maven upload artifacts have a unique artifactId
* Address @cocreature's comments in https://github.com/digital-asset/daml/pull/5272#pullrequestreview-385026181
* Remove unused da_doc_package
The only use-site was `//compiler/daml-licenses:daml-licenses`, which
itself was unused.
* Remove unused notices-gen
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Update rules_haskell
The workaround for linking against `Cffi` in the REPL has been
upstreamed in a more generalized form.
CHANGELOG_BEGIN
CHANGELOG_END
* ghcide: Use rules_haskell's hie-bios support
* Document `ghcide` Bazel integration
* Rename files to match module names
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
This removes the sample/reference implementation of kvutils
InMemoryKVParticipantState.
This used to be the only implementation of kvutils, but now with the
simplified kvutils api we have ledger-on-memory and ledger-on-sql.
InMemoryKVParticipantState was also used for the ledger dump utility,
which now uses ledger-on-memory.
* Runner now supports a multi participant configuration
This change removes the "extra participants" config and goes for consistent
participant setup with --participant.
* Run all conformance tests in the repository in verbose mode.
This means we'll print stack traces on error, which should make it
easier to figure out what's going on with flaky tests on CI.
This doesn't change the default for other users of the
ledger-api-test-tool; we just add the flag for:
- ledger-api-test-tool-on-canton
- ledger-on-memory
- ledger-on-sql
- sandbox
Fixes#4225.
CHANGELOG_BEGIN
CHANGELOG_END
* Bump rules_haskell
Still checking if that helps with GHC 8.8 but we should upgrade this
either way.
changelog_begin
changelog_end
* disable grpc patch
* shut up buildifier
* delete unused ghci grpc patch
* Fix Cffi library not found issues
* Update deps.bzl
Co-Authored-By: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
Context
=======
After multiple discussions about our current release schedule and
process, we've come to the conclusion that we need to be able to make a
distinction between technical snapshots and marketing releases. In other
words, we need to be able to create a bundle for early adopters to test
without making it an officially-supported version, and without
necessarily implying everyone should go through the trouble of
upgrading. The underlying goal is to have less frequent but more stable
"official" releases.
This PR is a proposal for a new release process designed under the
following constraints:
- Reuse as much as possible of the existing infrastructure, to minimize
effort but also chances of disruptions.
- Have the ability to create "snapshot"/"nightly"/... releases that are
not meant for general public consumption, but can still be used by savvy
users without jumping through too many extra hoops (ideally just
swapping in a slightly-weirder version string).
- Have the ability to promote an existing snapshot release to "official"
release status, with as few changes as possible in-between, so we can be
confident that the official release is what we tested as a prerelease.
- Have as much of the release pipeline shared between the two types of
releases, to avoid discovering non-transient problems while trying to
promote a snapshot to an official release.
- Triggerring a release should still be done through a PR, so we can
keep the same approval process for SOC2 auditability.
The gist of this proposal is to replace the current `VERSION` file with
a `LATEST` file, which would have the following format:
```
ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7
```
This file would be maintained with a script to reduce manual labor in
producing the version string. Other than that, the process will be
largely the same, with releases triggered by changes to this `LATEST`
and the release notes files.
Version numbers
===============
Because one of the goals is to reduce the velocity of our published
version numbers, we need a different version scheme for our snapshot
releases. Fortunately, most version schemes have some support for that;
unfortunately, the SDK sits at the intersection of three different
version schemes that have made incompatible choices. Without going into
too much detail:
- Semantic versioning (which we chose as the version format for the SDK
version number) allows for "prerelease" version numbers as well as
"metadata"; an example of a complete version string would be
`1.2.3-nightly.201+server12.43`. The "main" part of the version string
always has to have 3 numbers separated by dots; the "prerelease"
(after the `-` but before the `+`) and the "metadata" (after the `+`)
parts are optional and, if present, must consist of one or more segments
separated by dots, where a segment can be either a number or an
alphanumeric string. In terms of ordering, metadata is irrelevant and
any version with a prerelease string is before the corresponding "main"
version string alone. Amongst prereleases, segments are compared in
order with purely numeric ones compared as numbers and mixed ones
compared lexicographically. So 1.2.3 is more recent than 1.2.3-1,
which is itself less recent than 1.2.3-2.
- Maven version strings are any number of segments separated by a `.`, a
`-`, or a transition between a number and a letter. Version strings
are compared element-wise, with numeric segments being compared as
numbers. Alphabetic segments are treated specially if they happen to be
one of a handful of magic words (such as "alpha", "beta" or "snapshot"
for example) which count as "qualifiers"; a version string with a
qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`,
which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a
special ordering amongst qualifiers. Other alphabetic segments are
compared alphabetically and count as being "after" their prefix
(`1.2.3-really-final-this-time` counts as being released after `1.2.3`).
- GHC package numbers are comprised of any number of numeric segments
separated by `.`, plus an optional (though deprecated) alphanumeric
"version tag" separated by a `-`. I could not find any official
documentation on ordering for the version tag; numeric segments are
compared as numbers.
- npm uses semantic versioning so that is covered already.
After much more investigation than I'd care to admit, I have come up
with the following compromise as the least-bad solution. First,
obviously, the version string for stable/marketing versions is going to
be "standard" semver, i.e. major.minor.patch, all numbers, which works,
and sorts as expected, for all three schemes. For snapshot releases, we
shall use the following (semver) format:
```
0.13.53-alpha.20200214.3025.ef5d32b7
```
where the components are, respectively:
- `0.13.53`: the expected version string of the next "stable" release.
- `alpha`: a marker that hopefully scares people enough.
- `20200214`: the date of the release commit, which _MUST_ be on
master.
- `3025`: the number of commits in master up to the release commit
(included). Because we have a linear, append-only master branch, this
uniquely identifies the commit.
- `ef5d32b7ù : the first 8 characters of the release commit sha. This is
not strictly speaking necessary, but makes it a lot more convenient to
identify the commit.
The main downsides of this format are:
1. It is not a valid format for GHC packages. We do not publish GHC
packages from the SDK (so far we have instead opted to release our
Haskell code as separate packages entirely), so this should not be an
issue. However, our SDK version currently leaks to `ghc-pkg` as the
version string for the stdlib (and prim) packages. This PR addresses
that by tweaking the compiler to remove the offending bits, so `ghc-pkg`
would see the above version number as `0.13.53.20200214.3025`, which
should be enough to uniquely identify it. Note that, as far as I could
find out, this number would never be exposed to users.
2. It is rather long, which I think is good from a human perspective as
it makes it more scary. However, I have been told that this may be
long enough to cause issues on Windows by pushing us past the max path
size limitation of that "OS". I suggest we try it and see what
happens.
The upsides are:
- It clearly indicates it is an unstable release (`alpha`).
- It clearly indicates how old it is, by including the date.
- To humans, it is immediately obvious which version is "later" even if
they have the same date, allowing us to release same-day patches if
needed. (Note: that is, commits that were made on the same day; the
release date itself is irrelevant here.)
- It contains the git sha so the commit built for that release is
immediately obvious.
- It sorts correctly under all schemes (modulo the modification for
GHC).
Alternatives I considered:
- Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would
be accepted by all schemes, but will not sort as expected under semantic
versioning (though Maven will be fine). I have no idea how it will sort
under GHC.
- Not having any non-numeric component, e.g. `0.13.53.20200214.3025`.
This is not valid semantic versioning and is therefore rejected by
npm.
- Not having detailed info: just go with `0.13.53-snapshot`. This is
what is generally done in the Java world, but we then lose track of what
version is actually in use and I'm concerned about bug reports. This
would also not let us publish to the main Maven repo (at least not more
than once), as artifacts there are supposed to be immutable.
- No having a qualifier: `0.13.53-3025` would be acceptable to all three
version formats. However, it would not clearly indicate to humans that
it is not meant as a stable version, and would sort differently under
semantic versioning (which counts it as a prerelease, i.e. before
`0.13.53`) than under maven (which counts it as a patch, so after
`0.13.53`).
- Just counting releases: `0.13.53-alpha.1`, where we just count the
number of prereleases in-between `0.13.52` and the next. This is
currently the fallback plan if Windows path length causes issues. It
would be less convenient to map releases to commits, but it could still
be done via querying the history of the `LATEST` file.
Release notes
=============
> Note: We have decided not to have release notes for snapshot releases.
Release notes are a bit tricky. Because we want the ability to make
snapshot releases, then later on promote them to stable releases, it
follows that we want to build commits from the past. However, if we
decide post-hoc that a commit is actually a good candidate for a
release, there is no way that commit can have the appropriate release
notes: it cannot know what version number it's getting, and, moreover,
we now track changes in commit messages. And I do not think anyone wants
to go back to the release notes file being a merge bottleneck.
But release notes need to be published to the releases blog upon
releasing a stable version, and the docs website needs to be updated and
include them.
The only sensible solution here is to pick up the release notes as of
the commit that triggers the release. As the docs cron runs
asynchronously, this means walking down the git history to find the
relevant commit.
> Note: We could probably do away with the asynchronicity at this point.
> It was originally included to cover for the possibility of a release
> failing. If we are releasing commits from the past after they have been
> tested, this should not be an issue anymore. If the docs generation were
> part of the synchronous release step, it would have direct access to the
> correct release notes without having to walk down the git history.
>
> However, I think it is more prudent to keep this change as a future step,
> after we're confident the new release scheme does indeed produce much more
> reliable "stable" releases.
New release process
===================
Just like releases are currently controlled mostly by detecting
changes to the `VERSION` file, the new process will be controlled by
detecting changes to the `LATEST` file. The format of that file will
include both the version string and the corresponding SHA.
Upon detecting a change to the `LATEST` file, CI will run the entire
release process, just like it does now with the VERSION file. The main
differences are:
1. Before running the release step, CI will checkout the commit
specified in the LATEST file. This requires separating the release
step from the build step, which in my opinion is cleaner anyway.
2. The `//:VERSION` Bazel target is replaced by a repository rule
that gets the version to build from an environment variable, with a
default of `0.0.0` to remain consistent with the current `daml-head`
behaviour.
Some of the manual steps will need to be skipped for a snapshot release.
See amended `release/RELEASE.md` in this commit for details.
The main caveat of this approach is that the official release will be a
different binary from the corresponding snapshot. It will have been
built from the same source, but with a different version string. This is
somewhat mitigated by Bazel caching, meaning any build step that does
not depend on the version string should use the cache and produce
identical results. I do not think this can be avoided when our artifact
includes its own version number.
I must note, though, that while going through the changes required after
removing the `VERSION` file, I have been quite surprised at the sheer number of
things that actually depend on the SDK version number. I believe we should
look into reducing that over time.
CHANGELOG_BEGIN
CHANGELOG_END
* Avoid opening a server to the world when finding a free port.
This is very annoying on macOS because we get a focus-stealing popup for
a split second, asking for permission to allow the server through the
firewall. The popup pretty much always disappears before it can even be
read, when the server is closed.
This is almost certainly not an attack vector, because:
- we only do this in tests,
- the server is open for only a few milliseconds,
- nothing is served,
- and finding the port is tricky, because it's effectively random.
Nevertheless, it's very annoying.
CHANGELOG_BEGIN
CHANGELOG_END
* Extract a Bazel package for finding free ports.
We seem to do it in 4 different places, which I think is enough to
remove the duplication.
As mentioned in the title, this is still very experimental and needs
more work before we want to advertise it. However, the code is in a
somewhat reasonable shape, there are tests and I think even in the
current state it is already useful. Also this PR is already getting
very large so I don’t want to hold off much longer before merging this.
It is included in the SDK but hidden from `damlc --help` and `daml
--help` until the most pressing issues are addressed (primarily around
making sure that it doesn’t just shut down if you have a type error
and better error messages in general).
changelog_begin
changelog_end
This is a bit ugly but after spending some time digging into the
issues in rules_haskell around data-files, this seems like the most
sensible option especially given that we also want to ship them in the
SDK which woud require additional work even if we do fix it in
rules_haskell.
fixes#4457
changelog_begin
changelog_end
* Disable http2 with Nix to work around segfaults
CHANGELOG_BEGIN
CHANGELOG_END
* Disable http2 in dev-env calls to nix-build as well
Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
* Update rules_haskell
* Includes Bazel 2.0.0 support
* Some patches have been upstreamed
CHANGELOG_BEGIN
CHANGELOG_END
* External haskell_cabal_binary|library verbose = False
`haskell_cabal_binary|library` now supports setting `verbose = False`
which also avoids warnings coming from `Setup.hs`. As these are external
dependencies we are not going to address these warnings anyway. So, they
are just noise. This makes it unnecessary to pass `-w` or `-optF=-w`.
In case of build failure all errors and warnings will be displayed.
Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
The `sandbox-perf` build has been failing for a while with the following
errors:
```
INFO: From Generating benchmark code for //ledger/sandbox-perf:sandbox-perf_codegen:
JMH benchmark generation: JMH Benchmark generator failed
JMH benchmark generation: Benchmark classes should not be final.
[com.digitalasset.platform.sandbox.perf.LargeTransactionBench]
JMH benchmark generation: The instantiated @State class cannot be abstract.
[com.digitalasset.platform.sandbox.perf.PerfBenchState]
```
However, these errors are ignored; running the benchmarks runs the
`AcsBench` benchmark and ignores the fact `LargeTransactionBench` and
`SimpleBench` failed to compile.
I've fixed the errors by making sure that classes are not final, and
that `SimpleBench` uses a concrete state class.
This also introduces a cheap patch to the Scala JMH Bazel rules that
makes sure they fail if there are any errors.
I'm not really sure how to patch the Bazel rules properly, but someone
else might have an idea. :-)
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Disable all the TS stuff on Windows
changelog_begin
changelog_end
* disable jest explicitly
* more disabling
* :sadpanda:
* Replace @language_support_ts_deps on Windows
Provides dummy content so that `load` commands are still valid on
Windows without `yarn_install`.
* disable daml-ledger-fetch on windows
* shut up buildifier
Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
* language: put sdk versions into package.json
The typescript library versions of our support libraries are now given
by the sdk version.
CHANGELOG_BEGIN
CHANGELOG_END
* removed local field
* better placeholders
* consistent SDK_VERSION
* sed sdkversion in test script
* language: bazel rules for daml-json-types/daml-ledger-fetch
This moves the daml-json-types/daml-ledger-fetch libraries out of the
tests directory and builds them with bazel. We'll rename these libraries
in a follow up PR.
CHANGELOG_BEGIN
CHANGELOG_END
* Update deps.bzl
Co-Authored-By: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* updated package.json
* rename nodejs patch
* update yarn.lock
* update @bazel/bazel dependency
* wrong typescript version in toplevel package.json
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* Remove manual stack update
* Update rules_haskell
* rules_haskell_worker_dependencies after bazel-haskell-deps
* Update rules_haskell Windows patch
* make cabal haddock optional
* Don't generate Haddocks on stack_snapshot
Fails with ghc-lib and takes more time to build.
Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
* Change variant json encoding,
adding integration test
* Add DamlLfTypeLookup dependencies
* Add MetadataReader
* Add test WIP
* Add serialize test cases
* Add serialize test cases, WIP
* Test for variant encoding decoding
* Solving merge conflicts
* Updating roundtrip test
* Minor cleanup
* Addressing code review comments
Add JsonVariant custom matcher
* Update specification
* Update link
* Add test case, WIP
* Add proper template key resolution
* Got rid of choice record ID resolution, resolving choice type and key type
* Fixing logging
* Add Contract Key decoding tests
* cleanup
* cleanup
* Update JSON variant encoding tests
* Add more contract key JSON decoding tests
* Fix variant JSON encoding
* Change value predicate to support new variant encoding
* Change value predicate to support new variant encoding
* Add lookup by contract key test case
where contract key contains variant and record
Add `requiredResource` to bazel utils
CHANGELOG_BEGIN
- [JSON API - Experimental] Change variant JSON encoding. The new format is ``{ tag: data-constructor, value: argument }``.
For example, if we have: ``data Foo = Bar Int | Baz``, these are all valid JSON encodings for values of type Foo:
- ``{"tag": "Bar", "value": 42}``
- ``{"tag": "Baz", "value": {}}``
See #3622
- [JSON API - Experimental] Fix ``/contracts/lookup` find by contract key.
- [JSON API - Experimental] Fix ``/command/exercise`` to support any LF type as a choice argument.
See #3390
CHANGELOG_END
* minor cleanup
* Fix copy/paste
* Renaming
* Got rid of DAML LF identifier resolution
resolving DAML LF Type based on command type
* Address code review comments, thanks @S11001001
* Address code review comments, thanks @S11001001
Do not include any error handling here; this partial function should
only match the successful case, JsonVariant.
* Address code review comments, thanks @S11001001
comment
* Address code review comments, thanks @S11001001
using `JsonVariant` for variant encoding/decoding
* Address code review comments, thanks @S11001001
replace `find` and `map` chain with collectFirst
* Update docs/source/json-api/lf-value-specification.rst
Co-Authored-By: Stephen Compall <stephen.compall@daml.com>
Co-authored-by: Stephen Compall <scompall@nocandysw.com>
This still contains the main class so you can use it like you would
use the fat jar but publishing fat jars to maven central is apparently
bad practise and some peple have asked for the library.
This includes some slight tweaks to the scala_docs rule to make it
capable of coping with the generated source file and a hack in the
release script to avoid it complaining about the scenario proto
library not being published to Maven even though it is included in the
transitive deps.
* ledger-api-test-tool: Increase the duration when watching health.
This should hopefully stop CI from flaking out.
* reference-v2/sandbox: Avoid unnecessary companion object constructors.
I like indirection… when it does something.
* ledger: Propagate empty health checks throughout the services.
* reference: Remove duplication from the ReferenceServer object.
* ledger-api-common: Actually query a "reporter" in the health service.
* ledger-api-common: Report health per-component when required.
* ledger-api-health: Use a Map to represent components for health checks.
* sandbox: Fix warnings in SqlLedgerSpec.
* ledger-api-common: Throw GrpcHealthService errors inside the Future.
* ledger: Implement health checks against the PostgreSQL connection.
Without proper testing, because I am not great at this.
* sandbox: Remove duplication and fix warnings in PostgresAround.
* sandbox: Test the SQL Ledger's health reporting on failure.
* sandbox: Don't report as unhealthy until 3 connections fail.
* ledger-api-health: Remove unused parts of the API.
Bit of premature design there.
* sandbox: Rename the "ledger" health check to "write".
* participant-state: Add the ReportsHealth trait to ReadService.
* ledger-api-common: `Future.fromTry(Try(…))` -> `Future(…)`.
* ledger-api-common: Make it clearer that StubReporter closes over health.
* ledger-api-common: Explain the HealthService watch tests with comments.
* sandbox: Clean up SqlLedger a bit.
* sandbox: Don't try and stop PostgreSQL twice in PostgresAround.
* bazel_tools: Windows rlocation lookups need to be with forward slashes.
* release: Fix case of "true".
* ledger-api-common: Make `GrpcHealthService::matchResponse` return a Try.
* ledger-api-common: Make `GrpcHealthServiceSpec` async.
* sandbox: Make a couple of DB classes final.
* sandbox: Avoid importing `X._` in PostgresAround.
* sandbox: Add clues to the SqlLedgerSpec's multiple assertions.
* sandbox: If PostgreSQL doesn't come back up, keep retrying.
* sandbox: Remove duplication in SqlLedgerSpec.
* sandbox: In SqlLedgerSpec, actually wait for the health to change.
* sandbox: In PostgresAround, make stopping PostgreSQL idempotent.
* sandbox: Simplify the SqlLedgerSpec to make it work on CI.
It's worth a shot.
* ledger-api-common: Simplify the GrpcHealthServiceSpec a little.
And add a changelog.
CHANGELOG_BEGIN
- [Ledger API Server] Add a health check endpoint conforming to the
`GRPC Health Checking Protocol <https://github.com/grpc/grpc/blob/master/doc/health-checking.md>`_.
- [Ledger API Server] Add health checks for index database connectivity.
- [Participant State API] Add a mandatory ``currentHealth()`` method to
``IndexService``, ``ReadService`` and ``WriteService``.
CHANGELOG_END
* sandbox: Improve the Javadoc layout for DbDispatcher.
* sandbox: Capitalize constants in SqlExecutor.
* ledger-api-health: Convert HealthStatus to an abstract class.
* Get grpc from nix on unix
The one from Bazel seems to cause linking issues when trying to run
things in GHCi. I’ve spent some time trying to use rules_foreign_cc to
build gRPC using CMake but decided that for my own sanity it’s better
to not pursue that further.
* Address review comments
* Add missing module load
* Cleanup GHCI_SCRIPT
* use the correct file ending on macos
* Import is_linux
* Switch back to grpc-1.23
The newer version seems to cause issues in combination with the java libraries.
* Try to fix package_app on macos
* more debugging
* Maybe this is not necessary, we will never know
* linkers are the worst
* Remove debugging output again
* readd rpaths
* treat libdispatch specially
* remove hack
* more fooling around
* lalala
rules_haskell looks for stack in PATH. On Windows it is provided by
dadew (i.e. scoop). rules_haskell then symlinks (copies on Windows) the
stack binary. Unfortunately, this breaks with scoop as the shim file is
then not found.
* Add helper to produce an empty zip files.
This is used to generate empty sources and javadoc jars for
deploy jars later on.
* Create empty auxilliary jars.
da_java_binary:
- empty javadoc jar
- empty sources jar
da_java_proto_library:
- empty javadoc jar
- the sources jar is automatically generated by java_proto_library as a side effect
da_scala_binary:
- empty javadoc jar
- empty sources jar
* Support maven upload for jar-deploy and jar-proto
For jar-deploy targets we don't check for internal dependencies,
because these should already be contained in the (fat-)jar itself.
Additionally, the release program now uploads javadocs and sources
for jar-proto and jar-deploy as well to comply with maven central.
* Upload ledger-api-test-tool and kvutils + dependencies to maven central.
This is the diff running the output of the release without and with these changes.
A few artifacts now also get their javadoc and sources uploaded (mostly to bintray,
but now they are ready for a maven central upload).
ledger-api-test-tool has the scala version removed from the artifact as it is a
deploy jar and nobody should care which specific scala version is used.
Only in release/com/daml/ledger/participant-state-kvutils-java-proto/100.13.35: participant-state-kvutils-java-proto-100.13.35-javadoc.jar
Only in release/com/daml/ledger/participant-state-kvutils-java-proto/100.13.35: participant-state-kvutils-java-proto-100.13.35-sources.jar
Only in release/com/daml/ledger/testtool: ledger-api-test-tool
Only in release-before/com/daml/ledger/testtool: ledger-api-test-tool_2.12
Only in release/com/digitalasset/daml/lf/engine/trigger/runner_2.12/100.13.35: runner_2.12-100.13.35-javadoc.jar
Only in release/com/digitalasset/daml/lf/engine/trigger/runner_2.12/100.13.35: runner_2.12-100.13.35-sources.jar
Only in release/com/digitalasset/daml-lf-blindinginfo-java-proto/100.13.35: daml-lf-blindinginfo-java-proto-100.13.35-javadoc.jar
Only in release/com/digitalasset/daml-lf-blindinginfo-java-proto/100.13.35: daml-lf-blindinginfo-java-proto-100.13.35-sources.jar
Only in release/com/digitalasset/daml-lf-transaction-java-proto/100.13.35: daml-lf-transaction-java-proto-100.13.35-javadoc.jar
Only in release/com/digitalasset/daml-lf-transaction-java-proto/100.13.35: daml-lf-transaction-java-proto-100.13.35-sources.jar
Only in release/com/digitalasset/daml-lf-value-java-proto/100.13.35: daml-lf-value-java-proto-100.13.35-javadoc.jar
Only in release/com/digitalasset/daml-lf-value-java-proto/100.13.35: daml-lf-value-java-proto-100.13.35-sources.jar
Only in release/com/digitalasset/damlc/100.13.35: damlc-100.13.35-javadoc.jar
Only in release/com/digitalasset/damlc/100.13.35: damlc-100.13.35-sources.jar
Only in release/com/digitalasset/extractor/100.13.35: extractor-100.13.35-javadoc.jar
Only in release/com/digitalasset/extractor/100.13.35: extractor-100.13.35-sources.jar
Only in release/com/digitalasset/ledger-service/http-json-deploy/100.13.35: http-json-deploy-100.13.35-javadoc.jar
Only in release/com/digitalasset/ledger-service/http-json-deploy/100.13.35: http-json-deploy-100.13.35-sources.jar
Only in release/com/digitalasset/navigator/100.13.35: navigator-100.13.35-javadoc.jar
Only in release/com/digitalasset/navigator/100.13.35: navigator-100.13.35-sources.jar
* Fetch grpc and protobuf Haskell libraries from Hackage
All the changes that resulted in us fetching them from git, have been
included in the latest Hackage releases.
* Switch back to a custom build file for grpc-haskell-core
* Remove grpc-haskell-core from hazel packages
* Use cc_wrapper in ghcide test
The cc_wrapper is needed to find library dependencies.
* Update rules_haskell
* update rules_nixpkgs
Fixes issue with `nixpkgs_local_repository` and Nix `import`.
* Shorten RULES_HASKELL_EXEC_ROOT
Cabal package library paths are longer and overflow the command-line
length when calling `cc` on MacOS for template Haskell dependencies in
the IDE tests. Shortening the `RULES_HASKELL_EXEC_ROOT` prefix for each
`(LD_)LIBRARY_PATH` entry works around the issue.
* bazel: 0.28.1 --> 1.1.0
* bazel-watcher sha256
* Fix missing line in patch
* proto_source_root --> strip_import_prefix
See https://github.com/bazelbuild/bazel/issues/7153 for details.
* Update rules_nixpkgs
Required to avoid errors of the form
```
ERROR: An error occurred during the fetch of repository 'node_nix':
parameter 'sep' may not be specified by name, for call to method split(sep, maxsplit = None) of 'string'
```
and
```
ERROR: An error occurred during the fetch of repository 'node_nix':
Traceback (most recent call last):
File "/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 149
_execute_or_fail(repository_ctx, <3 more arguments>)
File "/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 318, in _execute_or_fail
fail(<1 more arguments>)
Cannot build Nix attribute 'nodejs'.
Command: [/Users/runner/.nix-profile/bin/nix-build, /private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/node_nix/nix/bazel.nix, "-A", "nodejs", "--out-link", "bazel-support/nix-out-link", "-I", "nixpkgs=/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/nixpkgs/nixpkgs"]
Return code: 1
Error output:
src/main/tools/process-tools.cc:173: "setitimer": Invalid argument
```
* Update rules_scala
* .proto has been removed, use [ProtoInfo] instead
See
https://docs.bazel.build/versions/1.1.0/be/protocol-buffer.html#proto_library
* python3_nix add nix_file attribute
To avoid the following error
```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/BUILD:66:1: //:nix_python3_runtime depends on @python3_nix//:bin/python in repository @python3_nix which failed to fetch. no such package '@python3_nix//': Traceback (most recent call last):
File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 149
_execute_or_fail(repository_ctx, <3 more arguments>)
File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 318, in _execute_or_fail
fail(<1 more arguments>)
Cannot build Nix attribute 'python3'.
Command: [/home/aj/.nix-profile/bin/nix-build, "-E", "import <nixpkgs> { config = {}; overlays = []; }", "-A", "python3", "--out-link", "bazel-support/nix-out-link", "-I", "nixpkgs=/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/nixpkgs/nixpkgs"]
Return code: 1
Error output:
error: anonymous function at /home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/nixpkgs/nixpkgs.nix:3:1 called with unexpected argument 'config', at (string):1:1
```
* rules_haskell unnamed string.split(_, maxsplit = _)
The keyword argument may no longer be named.
* string.replace(_, _, maxsplit = _) may not be named
* Move proto sources from deps to data
Fixes
```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/daml-lf/archive/BUILD.bazel:150:1: in deps attribute of scala_test rule //daml-lf/archive:daml_lf_archive_reader_tests_test_suite_src_test_scala_com_digitalasset_daml_lf_archive_DecodeV1Spec.scala: '//daml-lf/archive:daml_lf_1.6_archive_proto_srcs' does not have mandatory providers: 'JavaInfo'. Since this rule was created by the macro 'da_scala_test_suite', the error might have been caused by the macro implementation
```
* Define sha256 for haskell_ghc__paths
Bazel 1.1.0 fails on missing hashes.
* Disable --incompatible_windows_native_test_wrapper
* //compiler/daml-extension don't modify sources
Modifying sources in-place can cause issues on Windows, where build
actions are not sandboxed and changes on sources can affect other build
steps.
* bazel-genfiles --> bazel-bin
The bazel-genfiles symlink has been removed since Bazel 1.0.
See https://github.com/bazelbuild/bazel/issues/8651
* Mark dev_env_tool repository rule as configure
See
https://docs.bazel.build/versions/1.1.0/skylark/lib/globals.html#repository_rule
* Move data deps into data attribute
* Mark dev_env_tool as local = True
* Manually fetch @makensis_dev_env
* Remove git-revision from dependency graph
The navigator binary depended on the current git-revision. The issue is
that any target/test depending on the git-revision will have to be
rebuilt/rerun on every commit.
The navigator package itself was careful to avoid unnecessary
rebuilds/reruns. However, the SDK release tarball depends on the
navigator binary and thereby on the git-revision. The daml-assistant
integration tests, which depend on the SDK release tarball, therefore
had to be rerun on every commit.
This change removes the git-revision from the navigator alltogether. For
issue reporting the SDK version is still available.
* Remove unused git-revision and workspace_status
* Add client_server_build and integrity_test rules
And use them to implement ledger dump of the reference
server and to check it.
* Only build and test ledger dump on Linux. Only run tests relevant to dump.
* Make client_server_build quiet in happy path
* Reformat
* Remove unnecessary runfiles for client_server_build
* Update bazel-common to fix javadoc issues
Specifically, to fix the following error
```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/ledger-api/rs-grpc-bridge/BUILD.bazel:7:1: in javadoc_library rule //ledger-api/rs-grpc-bridge:rs-grpc-bridge_javadoc:
Traceback (most recent call last):
File "/home/aj/tweag.io/da/da-bazel-1.1/ledger-api/rs-grpc-bridge/BUILD.bazel", line 7
javadoc_library(name = 'rs-grpc-bridge_javadoc')
File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/com_github_google_bazel_common/tools/javadoc/javadoc.bzl", line 27, in _javadoc_library
dep.java.transitive_deps
object of type 'JavaSkylarkApiProvider' has no field 'transitive_deps'
```
* Define Maven deps using rules_jvm_external
* Pin artifacts
* Remove bazel-deps generated targets
* Remove bazel-deps
* Switch to rules_jvm_external targets
* update bazel documentation
* pom_file: There are no more bazel-deps targets
* BAZEL-JVM.md `maven_install` typo
* Fix bazel query deps(//...)
* Add rules_haskell cc_wrapper
Updates to latest rules_haskell master and adds the cc_wrapper PR as a
patch, see https://github.com/tweag/rules_haskell/pull/1039.
* Shorten include dirs in cc-wrapper
When using `haskell_cabal_library` GHC constructs unnecessarily long
include directories which can quickly overflow the maximum command-line
length. This patch avoids the issue by normalizing include paths.
* glob --> breadth_first_walk
We currently use a custom cabal file for ghc-lib that has libffi in
the extra-libraries section so Hazel adds the headers. Forcing GHC to
use the bundled libffi should hopefully remove the need for this hack
which simplifies things.
* Delete obsolete proto3-suite patch
My patch has been upstreamed so no need to keep it around in our repo.
* Upgrade proto3-wire
* Adapt to changes in proto3-suite
This makes sure that C dependencies like gRPC or zlib get compiled
with optimizations. I patched rules_haskell to use -O instead of -O2
since the latter slows down compilation while not making things
faster (according to my measurements).
* Bump stackage
This PR switches us over to the latest Stackage LTS 0.14.1 (we were on
0.13.x before, so this includes major bumps) and gets rid of some
obsolete overwrites.
* Go back to building grpc-haskell-core using c2hs
This should hopefully avoid issues like the CSize vs CULong issue we
had a while back and might fix some of the issues we have been seeing
on CI.
I’m marking the Haskell ledger bindings as non-flaky for now so we can
see if the issues reappear.
* Fix path
* Fix c2hs runfiles
* s/basedir/dirname/
* Fix varname
* Remove fixme \o/
* Mark hs ledger bindings flaky again
For now, this only works on Linux (that’s a GHC limitation as far as I
know) and you have to enable it by setting the GHC_DWARF env var to a
non-empty string.
* Upgrade haskell-lsp and lsp-test
There have been some fixes upstream that should hopefully mean that we
no longer need to mark the lsp-tests as flaky on Windows. I am having
trouble reproducing the flakiness locally, so let’s see what happens
on CI.
* Also bump stack.yaml
* Upgrade ghc-lib
* Patch bazel_tools : hazel-include-paths patch, no-isystem patch.
* Provide "haskell_c2hs" for package name to cabal_haskell_package
* Package name haskell_c2hs => c2hs.
* Switch to less hacky patch for include dirs
* DA.Bazel.Runfiles based on bazel-runfiles
* locateRunfilesMb -> locateRunfiles
* .exe extension on Windows
* Add docstring to locateRunfiles
* bazel-runfiles: Normalize on Windows
* replace ApiValue ADT with aliases to daml-lf/transaction Value ADT
* porting rest of navigator to LF Value ADT
* porting more of navigator to LF Value ADT
* last error, not first
* rename ApiValueImplicits file
* special conversion features for ImmArray and FrontStack
- just .to[ImmArray] or .to[FrontStack] any random collection
* finish porting most of navigator main code
* use numeric indices for record field name fallback when pretty-printing
* tuples are not serializable
* use numeric indices for label fallback in JSON verbose encoding
* make traverseEitherStrictly more likely to preserve the seq's class
* to shortcut for ImmArraySeq .to[ImmArraySeq]
* compiling, passing navigator backend tests
* test traverseEitherStrictly more, er, strictly
* pass scalacopts through to scaladoc
* deal with unused warning
* remove unneeded function
* simpler error reporting, more private functions in ApiCodecCompressed
* move slowApply to FrontStack, test it so it actually works
* remove unneeded toStrings; better error from impossible ValueTuple case
* scalafmt FrontStackSpec
* support alternative, label-free record JSON encoding
* fuse some list operations
- suggested by @stefanobaghino-da; thanks
* blue error message
Python's `os.symlink` may fail on Windows if the user has insufficient
permissions to create symbolic links. This was not noticed on CI, since
builds there are executed with administrator privileges.
This changes `dev_env_package` to only outsource the listing of
directory contents to Python, but then fall back to Bazel's own
`repository_ctx.symlink` for the creation of symbolic links (or copies
if necessary).
* Bazel: 0.24.0 -> 0.27.0
* Update rules_haskell for Bazel 0.27 compatibility
* Update bazel-deps and bazel-watcher
* Windows escape JVM flags
* load commands at top of .bzl file
Bazel 0.27 no longer allows load commands that are not at the beginning
of the file.
* Update Bazel rules
* subpackage boundary
* native is not defined in BUILD files
* yarn: @bazel/hide-bazel-files
Seems to be required since latest rules_nodejs version. Otherwise, yarn
fails with errors about existing BUILD or BUILD.bazel files.
* grpc-java plugin visibility
* Update fat_cc_library
* Nix Python3 toolchain
* Iteration over depset
* dev_env_package: Create symlinks one level deeper
To prevent symlinking the BUILD file as well. The nested BUILD file
confuses Bazel as of 0.27 and rules_nodejs cannot find the node
executable anymore.
* Update rules_nodejs
* Add managed_directories for node_modules
* hie-bios: Extract bazel-genfiles from bazel info
Bazel 0.27 changed the genfiles location which breaks the hie-core test
on macOS.
* update cc_wrapper to Bazel 0.27
* bazel info -> bazel info bazel-genfiles
* Fix typo in BUILD
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Fix running the IDE on damlc
There were two issues:
1. Missing include paths.
2. Files where the module name does not match the file name.
I’ve fixed both and added a test that we can load the damlc Main.hs.
* Update rules_haskell and static GHC
Remove patches that have been upstreamed or are no longer required.
Update still required patches to match the new rules_haskell version.
Previously we patched rules_haskell to coerce GHC into using static
Haskell libraries in most places. In particular we moved hs-libraries
entries into extra-libraries entries in the package configuration files.
A much cleaner approach is to compile GHC with a static RTS, then GHC
will by itself choose to load static Haskell libraries.
* Remove haskell_cc_import
* da-hs-daml-cli -> daml-cli
* da-hs-damlc-app -> damlc-app
* windows: root build
* windows: fixed haskell bindings tests
* windows: disable client_server_test test
* windows: marking daml_test flaky due to #1907
* windows: removing da-hs-damlc-app run from build.ps1
* windows: disable hie-core alias of currently disabled target
* Fix CPP issue with default da-ghci
The default REPL target did not exclude the //nix targets that set the
CPP language extensions.
* da-ghci -e () requires explicit target
Running the tests via the ledger-api-test-tool results in a nasty
runtime exception. This check only fixes paths if it detects a bazel
environment.
Fixes#1841
* Update the hie-bios commit SHA
* Also update the SHA in our bazel WORKSPACE
* Update the hash too
* Tutorial for Emacs integration
* Update hie-bios patch
* Fixes#1204: Release bindings and codegens to Maven Central.
Upload the Java and Scala Bindings with the respective code
generator binaries to Sonatype Open Source Repository
Host for synchronization with Maven Central.
* Fixes#1600. Improve Bazel Scala source JAR generation.
Include sources from JAR files defined as 'sources' to a Bazel Scala
target. The improvement will allow inclusion of generated source targets
in sources JARs of dependencies.
* Use temporary directory in the dar_to_scala() rule.
Place the output of the Scala code generator in a temporary
directory instead of a directory at the 'primary' output path as Bazel
rules should never have directory dependencies.
Extra command line arguments to the client_server_test target are passed
to the ledger-api-test-tool instead of the arguments defined in the
bazel target.
For example:
`bazel run //ledger/sandbox:conformance-test-static-time --list` will
pass the --list parameter to the ledger-api-test-tool (to print all
available tests), instead of actually running the tests in static-time mode.
The Ledger API Test Tool takes the flag --all-tests to run all defaul
and optional tests
The tool also allows multiple occurrences of the --include and --exclude
flags.
Also removes StandaloneSemanticTestRunner.
Fixes#1371
* Update rules_haskell
- rules_haskell now handles the global package db within Bazel
https://github.com/tweag/rules_haskell/pull/859
- We no longer use the Nix provided c2hs. So, we drop it.
- Rename `ghcWithC2hs` to `ghcStatic` to clarify that that's where the
static linking patches are applied.
- Extend package-db patches to align Nix store paths with the new $out.
This works around a restriction in current rules_haskell, where
the paths in the package config files must have the same prefix as
the path to the package config files themselves.
- Don't exclude haskell libraries from extra-libraries entries.
* Drop redundant unix-compat override
This is a left-over from when the package was patched.
* Windows GHC bindist includes ffi header
* Drop unused language-c Nix override
* windows: fixed daml-lf tests for Windows by using Bazel's rlocation
* more consistent logging on CI; publishing Windows test logs on failure
* windows: fix daml-lf engine tests
* windows: add diff tool to msys
* Fixes 895: Improve DA Bazel rules for building javadocs.
Extend the da_java_library Bazel macro to also build the Javadoc for the
target. Add the Javadoc artefacts to the release procedure.
* enable Return wartremover wart
* remove return keyword from various places in daml-lf
* remove return keyword from various places in ledger
* simpler ImmArray equals
* move traverseEitherStrictly to point of use
* remove return from ledger-api-server-example
* Hazel: Shorten target names
Previously, Hazel would generate library and binary targets, that
repeated the package name in their target name. This easily lead to too
long paths on Windows, which could induce errors with code that did not
use API functions with long path support.
This change modifies Hazel to name the library target "lib" and shorten
the binary target names to "bin" or just the Cabal exe component name.
This change had further reaching consequences, because the package name
in the generated version macros was derived from the library target
name. rules_haskell has been extended to allow to override that default
behaviour.
* data-default: Remove custom build definitions
These had been introduced to resolve issues on Windows due to too long
target names. Hazel has meanwhile been patched to generate such shorter
target names by default, making the custom builds superfluous.
* Hazel: unshorten cbits name
This is a temporary workaround for otherwise clashing cbits library
names in the case of static only linking.
* Delete an entirely unused module
* Delete an entirely unused module
* Switch the compiler to use EUnit over mkEUnit
* Delete an unused module
* Whitespace only
* Clean up the API for World, don't expose the internals, better creation functions
* Clean up the type checker environment, don't expose the internals of Gamma, add a few helper functions
* Delete unused functions
* Explicit module export lists
* Fix the nub replacement hints
* Turn on the warning that we require module export lists
* Add an explicit export list
* Update rules_haskell: Generates version macros
- rules_haskell now generates a version macros header file that is
passed on to preprocessors such as c2hs or hsc2hs.
- The haskell_import rule was also renamed to haskell_toolchain_library.
* Drop unix-compat patch
This was necessary to work around missing version macros.
* Move to using proto3-wire from upstream
* Move to upstream proto3-suite, with some custom patches in my fork
* Delete the BUILD.bazel for hte proto3 stuff, not used and the test was failing
* Delete the old proto3-wire and proto3-suite forks
* Delete proto3-wire
* Prettify BUILD.bazel files, sort the deps
* Remove some special cases from the license checker
* Delete unused Nix files from grpc-haskell
* Switch to upstream proto3-suite
* Make old-time work on Windows
* Formatting
* Patch rules_haskell to use a response file for -optP to avoid overflowing argument size limits on Windows
* Update 3rdparty/haskell/BUILD.old-time
Co-Authored-By: neil-da <35463327+neil-da@users.noreply.github.com>
* Update the comments in old-time
* Use the revised location of proto3-suite
* haskell: windows: always link system libraries
Modifies the patch to rules_haskell so that Windows system library are
not only applied to grpc, but to all targets on Windows.
* windows: test //daml-assistant:daml on CI
* Fix network build on Windows
Some files were not added to the build, which led missing symbols at
link time.
* Drop dll.a files from Windows GHC bindist
Those files greatly confuse GHC when linking statically.
* Add some Windows system libraries
These libraries are needed when linking GRPC.
* Statically link pthread on Windows
Otherwise the executables fail at runtime because they cannot find the
shared object.
* Build and run damlc on CI
* Try to fix package_db/* nullglob error
* Fix powershell command
* Cleanup package db rule
* Make formatting ugly again
* Add semantic test for the reference server
Currently the semantic test is failing. Likely due to the
location annotation changes.
* Do not compare location annotations in isReplayedBy
The location annotations may not, and do not need to, match due to the
fact that the reconstructed update expression may not exactly match
the original one, and since the interpreter currently picks the closest
location annotation we cannot guarantee that they exactly match.
* scalafmt.
* buildifier.
* client_server_test: Increase timeout to 60s
Spawning a JVM-based server can easily take a long time on a very
loaded system (e.g. when running `bazel test //ledger/...` with enough
parallelism), so better have a high default timeout.
* ledger/api-server-damlonx: Address code review
* Fix client_server_test runner compilation. Bump timeout.
* Mark the reference server semantic test exclusive
* fmt reference/BUILD.bazel
* Inline c2hs expansion.
* Patch unix-compat for Windows
unix-compat fails on Windows due to missing version macros in hsc files.
This patches unix-compat inlining the effect of the corresponding
version macro evaluation.
* Add grpc-haskell to Windows CI
* Fix formatting
* Move unix-compat.patch
Moved to bazel_tools, where all other Bazel patches reside.
* Remove .chi files
Those don't need to be checked in.
* Add FIXME on checked in c2hs files.
* Remove long gone patch
The "shorten-source-dirs" patch was removed some time ago but was still
referenced in bazel_tools.
* Drop upstreamed rules_haskell patches
Some patches have been upstreamed to rules_haskell.
* Build haskell-ide-core on CI
* Apply buildifier
* Update rules_haskell
The latest version uses stackage's mirror of Hackage.
* Add buildifier targets.
The tool allows to check and format BUILD files in the repo.
To check if files are well formatted, run:
bazel run //:buildifier
To fix badly-formatted files run:
bazel run //:buildifier-fix
* Cleanup dade-copyright-headers formatting.
* Fix dade-copyright-headers on files with just the copyright.
* Run buildifier automatically on CI via 'fmt.sh'.
* Reformat all BUILD files with buildifier.
Excludes autogenerated Bazel files.
* Add client_server_test bazel rule
This adds a generic mechanism for constructing a test that launches
a TCP server and runs a test-suite against it. The rule orchestrates
passing the port number from the server to the client via a temporary
file and takes care of killing the server when the client process exits.
Still to be done is figuring out a good way to pass additional arguments
to the server and client programs (where these arguments could be other
bazel rules).
* Add missing copyright headers
* Add support for arguments to client_server_test
Note that this does not support passing in labels (":my_data_file"),
nor does it support "$(location :my_data_file)" string macros.
* Add data attribute to client_server_test
* Add support for location expansion in client_server_test args
* client_server_test: Address code review
* client_server_test: Bump waiting for port file write to 5s
* Fetch status.proto from remote, simplify JS gRPC codegen
Fetch the `status.proto` file (part of the standard gRPC distribution)
from a distribution channel. _Moreover_, use the recently introduced
`proto_gen` rule to simplify how the gRPC code for the Node.js bindings
are generated (and remove the need to have `google/rpc/status.proto`
locally in the repository.
* Add plugin_runfiles option to proto_gen
This allows use to add additional files to the bazel sandbox so that
plugins can refer to them. This will subsequently be used by the
protoc-gen-doc plugin.
Also, pass the plugin options via --name_opt parameter.
* Add missing status.proto dependency /language-support/java and /ledger
* Build proto docs using the proto_gen rule
To make this work, I had to turn on the bazel build flag
`--protocopt=--include_source_info` because we cannot turn enable this
flag only for specific build rules.
* Make /ledger-api/grpc-definitions:docs public again
* Revert to the old style of passing plugin arguments to --name_out=options:path
* Suppress output of unzipping
* Fix link for google.rpc.Status in proto-docs
We used to have a workaround for the command line length limit on
Windows, which exceeded when building ghc-lib. The ghc-lib package was
recently split into two smaller packages and this workaround is not
needed anymore.
* Update rules_haskell
This updates to the latest rules_haskell, which fixes a few issues on
Windows. Most importantly it flags a few Windows libraries as "system"
libraries, preventing Hazel to fail because they are not provided
through Bazel.
* Fix the streaming-commons build on Windows
This modifies our custom streaming-commons BUILD file to make it work on
Windows. In particular it swaps some system libraries, passes the
`-DWINDOWS` flag and enables the build of an extra module.
* Clean up bazel_tools BUILD file
This wraps a very long line for legibility.
* Fix shared object issues on Windows
* This fixes rules_haskell to use the correct Windows path separator on Windows.
GHC expects the LD_LIBRARY_PATH variable to be a list of semi-colon separated
paths, as opposed to a list of colon separated paths:
51fd357119/compiler/ghci/Linker.hs (L1646-L1650)
* This fixes the name of Haskell shared objects on Windows. By default
Bazel's cc_library generates '.so' files, whereas GHC expects a `.dll`
(or a few other extensions, non of which are `.so`):
51fd357119/rts/linker/PEi386.c (L684)
* Build daml-lf-ast on Windows CI
