Commit Graph

185 Commits

Author SHA1 Message Date
Moritz Kiefer
0a23318fb1
Drop redundant isystem patch (#7085)
As pointed out by the comment, we should not need this with
stack_snapshot anymore.

changelog_begin
changelog_end
2020-08-11 16:33:14 +02:00
Andreas Herrmann
27e7d4cf69
Update rules_haskell (#7077)
* Update rules_haskell

Removes the warning about Bazel 3.3.1 being too recent.

* Remove unused rules_haskell patch

changelog_begin
changelog_end

* fmt

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-08-11 10:14:56 +00:00
Stephen Compall
96624a7677
use -Ywarn-unused for all Scala code (#6907)
* add -Ywarn-unused to all scalac options

* remove some unused arguments

* remove some unused definitions

* remove some unused variable names

* suppress some unused variable names

* changeExtension doesn't use baseName

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* work around no plugins in scenario interpreter perf tests

* remove many more unused things

* remove more unused things, restore some used things

* remove more unused things, restore a couple signature mistakes

* missed import

* unused argument

* remove more unused loggingContexts

* some unused code in triggers

* some unused code in sandbox and kvutils

* some unused code in repl-service and daml-script

* some unused code in bindings-rxjava tests

* some unused code in triggers runner

* more comments on silent usages

- suggested by @cocreature; thanks

* fix missing reference in TestCommands

* more unused in triggers

* more unused in sandbox

* more unused in daml-script

* more unused in ledger-client tests

* more unused in triggers

* more unused in kvutils

* more unused in daml-script

* more unused in sandbox

* remove unused in ledger-api-test-tool

* suppress final special case for codegen unused warnings

.../com/daml/sample/mymain/ContractIdNT.scala:24: warning: parameter value ev 0 in method ContractIdNT Value is never used
      implicit def `ContractIdNT Value`[a_a1dk](implicit `ev 0`: ` lfdomainapi`.Value[a_a1dk]): ` lfdomainapi`.Value[_root_.com.daml.sample.MyMain.ContractIdNT[a_a1dk]] = {
                                                         ^
.../com/daml/sample/mymain/ContractIdNT.scala:41: warning: parameter value eva_a1dk in method ContractIdNT LfEncodable is never used
      implicit def `ContractIdNT LfEncodable`[a_a1dk](implicit eva_a1dk: ` lfdomainapi`.encoding.LfEncodable[a_a1dk]): ` lfdomainapi`.encoding.LfEncodable[_root_.com.daml.sample.MyMain.ContractIdNT[a_a1dk]] = {
                                                               ^

* one more unused in daml-script

* special scaladoc rules may need silencer, too

* unused in compatibility/sandbox-migration

* more commas, a different way to `find`

- suggested by @remyhaemmerle-da; thanks
2020-08-07 13:16:09 -04:00
Andreas Herrmann
cf24597e70
Factor out reproducibility flags for tar and gzip (#6884)
* Factor out tar/gzip reproducibility flags

* use mktgz in package-app

* Bazel managed tar/gzip

* Remove quiet = True

As stated in the comment this is no longer required with Bazel >= 3.0.

* Build package-app as a sh_binary

This way Bazel will manage the runtime dependencies tar, gzip, mktgz,
and patchelf.

package-app.sh changes directory so it needs to make sure that all paths
are absolute and that the runfiles tree/manifest location is forwarded
to programs called by package-app.sh.

* Avoid file path too long errors

* Fix readlink -f on MacOS

* Document abspath

changelog_begin
changelog_end

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-08-05 14:27:14 +00:00
Andreas Herrmann
8e706a9bde
Remove vendored pkg_tar (#6934)
* Use Bazel builtin pkg_tar rule

* Use @rules_pkg//:pkg.bzl%pkg_tar

The pkg_tar rule builtin to Bazel has been deprecated.
See https://docs.bazel.build/versions/master/be/pkg.html

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-07-30 15:53:16 +00:00
Stephen Compall
fd07a26510
check for scaladoc comments that are not actually used (#6802)
* add -Xlint:doc-detached

- reverts 1feae964e3 from #6798

* attach several scaladocs where they'll actually be included

* no changelog

* attach several more scaladocs where they'll actually be included

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-28 20:32:30 +00:00
Moritz Kiefer
6581bba285
Upgrade rules_scala (#6883)
changelog_begin
changelog_end
2020-07-28 08:53:12 +00:00
Moritz Kiefer
15bf131fed
Enable assertions in Haskell builds (#6853)
GHC hates its users and defaults to optimizing out assertions. We
fixed that in Buck at some point but clearly that got lost when
migrating to Bazel.

Turns out enabling assertions catches bugs. This insight was brought to
you from the people that also brought you “Turns out writing tests
catches bugs”.

fixes #5624

changelog_begin
changelog_end
2020-07-24 14:29:25 +00:00
Andreas Herrmann
3c768915b3
Pin rules_apple (#6845)
This does not change the version of rules_apple, it only pins the http
archive instead of fetching via git tag.

To avoid Bazel warnings of the following form since Bazel 3.3.1

```
DEBUG: Rule 'build_bazel_rules_apple' indicated that a canonical reproducible form can be obtained by modifying arguments commit = "ff6a37b24fcbbd525a5bf61692a12c810d0ee3c1", shallow_since = "1559833568 -0700" and dropping ["tag"]
DEBUG: Repository build_bazel_rules_apple instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule git_repository defined at:
  /home/aj/.cache/bazel/_bazel_aj/f66bee630c6a2cd906f92a0f5cdf8769/external/bazel_tools/tools/build_defs/repo/git.bzl:195:33: in <toplevel>
```

changelog_begin
changelog_end

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-07-24 09:47:05 +02:00
Andreas Herrmann
4b1438276c
Update Bazel 2.1.0 --> 3.3.1 (#6761)
* Upgrade nixpkgs revision

* Remove unused minio

It used to be used as a gateway to push the Nix cache to GCS, but has
since been replaced by nix-store-gcs-proxy.

* Update Bazel on Windows

changelog_begin
changelog_end

* Fix hlint warnings

The nixpkgs update implied an hlint update which enabled new warnings.

* Fix "Error applying patch"

Since Bazel 2.2.0 the order of generating `WORKSPACE` and `BUILD` files
and applying patches has been reversed. The allows users to define
patches to these files that will not be immediately overwritten.
However, it also means that patches on another repository's original
`WORKSPACE` file will likely become invalid.

* a948eb7255
* https://github.com/bazelbuild/bazel/issues/10681

Hint: If you're generating a patch with `git` then you can use the
following command to exclude the `WORKSPACE` file.

```
git diff ':(exclude)WORKSPACE'
```

* Update rules_nixpkgs

* nixpkgs location expansion escaping

* Drop --noincompatible_windows_native_test_wrapper

* client_server_test using sh_inline_test

client_server_test used to produce an executable shell script in form of
a text file output. However, since the removal of
`--noincompatible_windows_native_test_wrapper` this no longer works on
Windows since `.sh` files are not directly executable on Windows.

This change fixes the issue by producing the script file in a dedicated
rule and then wrapping it in a `sh_test` rule which also works on
Windows.

* daml_test using sh_inline_test

* daml_doc_test using sh_inline_test

* _daml_validate_test using sh_inline_test

* damlc_compile_test using sh_inline_test

* client_server_test find .exe on Windows

* Bump Windows cache for Bazel update

Remove `clean --expunge` after merge.

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-07-23 09:46:04 +02:00
Stephen Compall
4355406259
add more scalac 2.12 warnings (#6798)
* set many extra scalac -Xlint options for all Scala projects

CHANGELOG_BEGIN
CHANGELOG_END

* move NoCopy to its own file

package.scala:18: warning: it is not recommended to define classes/objects inside of package objects.
If possible, define trait NoCopy in package data instead.
  trait NoCopy {
        ^

* move more traits, classes, and objects to proper packages

- note that `package` is itself a scoping construct, so if your reason
  is the apparent aesthetic of placing a bunch of things in one `package
  object`, that is easily remedied by deleting the `object` keyword

* fix some type-parameter-shadow warnings

- I'm generally in favor of sensible name-shadowing, following the
  "deliberately hide variables that should not be accessed here" school
  of thought.  But I think type name shadowing isn't quite as valuable
  and more likely to confuse than general variable shadowing, so have
  experimentally linted it out.

  Example warning:

EventsTableFlatEventsRangeQueries.scala:11: warning: type parameter
 Offset defined in trait EventsTableFlatEventsRangeQueries shadows class
 Offset defined in package v1. You may want to rename your type
 parameter, or possibly remove it.
private[events] sealed trait EventsTableFlatEventsRangeQueries[Offset] {
                                                               ^

* fix more package-object-classes warnings

* fix an inaccessible warning

ContractsService.scala:197: warning: method searchDb in class ContractsService references private class ContractsFetch.
Classes which cannot access ContractsFetch may be unable to override searchDb.
  def searchDb(dao: dbbackend.ContractDao, fetch: ContractsFetch)(
      ^

* enable -Xlint:infer-any

- continuing the saga of #6116, #6132

* enable -explaintypes for more detailed type errors

* missed header for NoCopy; probably should have left it in the package file

* misspelling in comment

* revert -Xlint:doc-detached

- there are a lot of these fixes, and they are noisy, so shifting to a
  separate PR
- thanks to @leo-da for pointing out
2020-07-21 08:18:01 -04:00
Robert Autenrieth
a03a1ecc0b
Add basic participant integration API scaladoc (#6790)
* Add basic scaladoc

for participant integration API

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-20 21:31:32 +02:00
Andreas Herrmann
4be01e4df6
Update ghcide 0.1.0 --> 0.2.0 (#6745)
* Update rules_haskell hie-bios support

* Decouple Haskell ghcide and DAML ghcide

Creates a separate `stack_snapshot` to pull in `ghcide` for the Haskell
IDE use case independent of the `ghcide` for DAML. This allows to update
these two `ghcide` instances independently. As DAML uses `ghcide` the
library updates can be involved if the API experienced breaking changes.
At the same time we may wish to update `ghcide` for Haskell earlier to
make use of new features and stay compatible with rules_haskell's ghcide
support.

* Fix Haddock warnings reported by ghcide

changelog_begin
changelog_end

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-07-16 16:30:09 +00:00
Moritz Kiefer
0fe4769b6b
Reduce the number of runfiles in the compatibility tests (#6670)
Details are in a comment.

changelog_begin
changelog_end
2020-07-09 17:15:16 +00:00
Samir Talwar
28873913d9
Bazel: Upgrade Buildifier. (#6625)
Buildifier now comes with a handy attachment to catch single `\`
characters inside strings and replace them with `\\` if the escape
sequence is invalid. Skylark/Python will do this at runtime anyway; this
just makes it clearer what the actual behavior is.

I needed to change `\` characters at the end of lines to `\\` manually
in order to stop Buildifier from simply concatenating the lines
together. Everything else was automatic.

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-06 16:07:47 +00:00
Remy
6497dfa507
Override the version of JOPT Simple from scala_rules (#6621)
CHANGELOG_BEGIN
CHANGELOG_END
2020-07-06 17:08:01 +02:00
Andreas Herrmann
0c16823c1b
Upgrade rules_haskell and pin stack_snapshot (#6548)
* Update rules_haskell

* Pin stack_snapshot repositories

* Document stack_snapshot_json

CHANGELOG_BEGIN
CHANGELOG_END

* Don't pin stack_snapshot on Windows

The lock file is generated on Unix and includes unix specific
dependencies, e.g. `unix`. Most developers don't have easy access to a
Windows machine, so regenerating the lock file for Windows would be
inconvenient.

* upgrade stack 2.1.3 --> 2.3.1 on Windows

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-07-02 18:55:09 +02:00
Stephen Compall
7622862312
upgrade to Wartremover 2.4.9 (#6551)
* upgrade to wartremover 2.4.9

* simplify wart list and list JavaConversions as disabled

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* delete long-obsolete, contradictory comment

* also upgrade wartremover in compatibility (leaving aside maven_install.json)

* update compatibility maven_install.json to match
2020-07-01 15:02:58 +00:00
Stephen Compall
a51d0db8ff
set scalac -Xsource:2.13 -Ypartial-unification globally (#6469)
* add -Xsource:2.13, -Ypartial-unification to common_scalacopts

* add now-referenced scalaz-core where needed

* work around bad type signatures in scalatest Aggregating, Containing

* unused Any suppression

* work around bad partial-unification wrought by type alias

* remove unused Conversions import

- not required in 4f68cfc480 either, so unsure how it's survived this long

* work around Future.traverse; remove unused show import

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* remove unused bounds

* remove -Ypartial-unification and -Xsource:2.13 where they were explicitly passed

* longer comment on what the options do

- suggested by @stefanobaghino-da; thanks

* forget Future.traverse, just use scalaz, it knows how to do this
2020-06-24 16:51:24 -04:00
Andreas Herrmann
0544323dfc
upgrade rules_haskell (#6284)
* upgrade rules_haskell

CHANGELOG_BEGIN
CHANGELOG_END

* Use autogenerated stack_snapshot tools

* ghcide import-dirs flags

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-06-10 17:21:31 +02:00
Andreas Herrmann
921266a926
Enable profiling builds of Cabal packages (#6240)
* Upgrade rules_haskell to latest master

changelog_begin
changelog_end

* Update rules_haskell

* Cabal: Track _p.a and .a outputs

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-06-08 10:03:05 +00:00
Stephen Compall
15350a7bc2
disable warts.Any and remove most suppressions (#6132)
* disable Any wart

* first pass removal of Any suppressions for false positives

* second pass removal of Any suppressions for false positives

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* third pass removal of Any suppressions for false positives

* fourth pass removal of Any suppressions for false positives

* reformat newly single-suppressions into single lines

- suggested by @SamirTalwar-DA; thanks
2020-05-28 16:53:41 +00:00
Stefano Baghino
0f2b45f8d7
Remove unused zlib patch (#6137)
Leftover from #6063

Should have been removed along with
114ba4ecfb

changelog_begin
changelog_end
2020-05-28 08:43:13 +00:00
Andreas Herrmann
92a2b5a28c
Enforce a dependency on node_nix (#6035)
The workspace for the vendored node wrapper script `@nodejs_linux_amd64`
did previously not record a dependency on the nixpkgs provided node
workspace. This patch enforces that dependency by introducing a dummy
read of the vendored node binary.

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-05-19 16:58:00 +00:00
Andreas Herrmann
1b711d0ac6
Hermetic nixpkgs_cc_toolchain (#5976)
* update rules_nixpkgs

* Use hermetic nixpkgs cc toolchain

CHANGELOG_BEGIN
CHANGELOG_END

* Work around Bazel's cc toolchain autodetection

* Use --crosstool_top for hermetic cc toolchain

When using --incompatible_enable_cc_toolchain_resolution instead
cc actions still depend on
`external/local_config_cc/builtin_include_directory_paths`
as well as
`external/nixpkgs_cc_toolchain_config/builtin_include_directory_paths`.

* override local_config_cc

* remove unused attribute

* Fix posix toolchain on Windows

* nixpkgs cc toolchain not on Windows

* Fix nixpkgs cc toolchain on MacOS

* nixpkgs cc toolchain uses bin/cc

* Use darwin.binutils on MacOS

* Remove clang(++) and gcc (g++) symlinks

The toolchain only considers `bin/cc` and having the other symlinks
around could lead to confusion

* Use hermetic toolchain in compatibility workspace

* Avoid empty linker flags

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-05-18 18:03:51 +02:00
Samir Talwar
57a8d0b37e
CI: Run PostgreSQL once for all Scala tests. (#5919) 2020-05-14 09:06:34 +02:00
Andreas Herrmann
35fe61ed06
Update Bazel 2.0.0 --> 2.1.0 (#5651)
* Update Bazel 2.0.0 --> 2.1.0

CHANGELOG_BEGIN
CHANGELOG_END

* Update rules_haskell

For Bazel 2.1.0 compatibility.

* nix: grpc: patch glibc version incompatibility

* Fix hlint warnings

* CcToolchainInfo.compiler

https://docs.bazel.build/versions/2.1.0/skylark/lib/CcToolchainInfo.html#compiler

* CcToolchainInfo.ar_executable

* texlive: add missing transparent package

* Leave a debug note on the docs target

* Ignore FutureWarning in //docs:docs-no-pdf

* Isolate PR

This is to avoid any interference with pipelines prior to the Bazel
update.

* Remove isolation

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-05-11 11:47:54 +02:00
Moritz Kiefer
a47c734401
Fix MacOS version linking issues (#5868)
This should fix the following issue that we see constantly on CI:

```
ld: warning: object file _ was built for newer OSX version (10.15) than being linked (10.14)
```

The issue was that the CC toolchain was not fully used in
haskell_cabal_package. --with-gcc (which is really --with-cc) only
applies when Cabal is calling the C compiler. However, in most cases
it is actually GHC itself which calls the C compiler. To make sure
that the right compiler is used in those cases, we have to pass
`-pgmc` and friends to GHC. This matches what rules_haskell does for
non cabal targets.

changelog_begin
changelog_end
2020-05-07 09:36:45 +02:00
Moritz Kiefer
49e19ebed1
Make compat tests work on windows (#5732)
* Make compat tests work on windows

This required some changes to the daml_sdk rule since the read-only
installation by the assistant breaks Bazel completely. We could only
apply those changes on Windows but I think I prefer the consistency
across platforms here over trying to stay close to how the SDK is
installed on user machines given that the SDK installation is not
something we’ve had issues with.

I’ve excluded the postgresql tests for now. I don’t expect them to be
particularly hard to fix but I’ve already spent almost 2 days on this
and having some tests run on Windows seems like a clear improvement
over running no tests on Windows :)

changelog_begin
changelog_end

* Remove todo

changelog_begin
changelog_end
2020-04-28 16:06:36 +02:00
Gary Verhaegen
7ceda5678a
run compatibility tests on macos (#5723)
This PR extends the existing Linux compatibility tests to run on macOS
too. Fixes #5692.

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2020-04-27 14:55:16 +02:00
Andreas Herrmann
5f83cd1175
Upgrade rules_haskell (#5631)
CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-04-20 15:54:56 +00:00
Moritz Kiefer
db37e4c497
Patch zip library to not use temp files (#5621)
This should hopefully fix the issues we have been seeing on CI. While
I’m not super keen on including non-upstreamable patches it seems
better than having CI be flaky.

changelog_begin
changelog_end
2020-04-20 08:35:11 +00:00
Gerolf Seitz
cadaa59ad8
Fix redirects, java-bindings javadoc, and live-preview.sh (#5446)
* Fix redirects, java-bindings javadoc, and live-preview.sh

- javadoc_library now supports sources from filegroups as well
- //language-support/java:javadoc now generates javadoc for ledger-api, java bindings, rxjava bindings
- live-preview.sh refers to the correct javadoc target //language-support/java:javadoc
- removed leading / from redirects.map

* Only generate daml-lf javadocs if not on windows

CHANGELOG_BEGIN
CHANGELOG_END
2020-04-07 09:53:26 +02:00
Gerolf Seitz
a2d785e3ee
Use com.daml as root package (#5343)
Packages com.digitalasset.daml and com.daml have been unified under com.daml

Ledger API and DAML-LF DEV protos have also been moved from `com/digitalasset`
to `com/daml` on the file system.
Protos for already released DAML LF versions (1.6, 1.7, 1.8) stay in the
package `com.digitalasset`.

CHANGELOG_BEGIN
[SDK] All Java and Scala packages starting with
``com.digitalasset.daml`` and ``com.digitalasset`` are now consolidated
under ``com.daml``. Simply changing imports should be enough to
migrate your code.
CHANGELOG_END
2020-04-05 19:49:57 +02:00
Samir Talwar
e8f1704102
bazel_tools: Attempt to reduce the Scala process heap size further. (#5362)
Especially the initial heap size.

CHANGELOG_BEGIN
CHANGELOG_END
2020-04-02 08:37:23 +00:00
Samir Talwar
4b9b2e8e61
Reduce the default maximum heap size for all Scala processes. (#5350)
* bazel_tools: Set `unused_dependency_checker_mode` in one place.

* bazel_tools: Set the default max heap size for Scala processes to 2GB.

And the default initial max heap size to 512MB.

CHANGELOG_BEGIN
CHANGELOG_END

* bazel_tools: Set the `scalac` heap size to 2GB and stack size to 2MB.

* bazel_tools: Delete `da_scala_macro_library`, as it's unused.

* bazel_tools: Revert the description of `da_scala_library_suite`.

Misread it.
2020-04-01 19:14:20 +00:00
Gerolf Seitz
329320bad9
Organize maven coordinates (#5272)
* Use com.daml as groupId for all artifacts

CHANGELOG_BEGIN
[SDK] Changed the groupId for Maven artifacts to ``com.daml``.
CHANGELOG_END

* Add 2 additional maven related checks to the release binary

1. Check that all maven upload artifacts use com.daml as the groupId
2. Check that all maven upload artifacts have a unique artifactId

* Address @cocreature's comments in https://github.com/digital-asset/daml/pull/5272#pullrequestreview-385026181
2020-04-01 11:41:18 +02:00
Andreas Herrmann
89a9f5c7d2
tarball reproducibility (#5258)
* integration-tests reproducibility

* package-app reproducibility

* Make remaining tar czf reproducible

* package-app

CHANGELOG_BEGIN
CHANGELOG_END

* Reproducibility of remaing tar invocation

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-03-31 10:09:52 +02:00
Gary Verhaegen
1872c668a5
replace DAML Authors with DA in copyright headers (#5228)
Change requested by Manoj.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 01:26:10 +01:00
Andreas Herrmann
40e439ad15
Remove unused //compiler/daml-licenses and //:notices-gen (#5221)
* Remove unused da_doc_package

The only use-site was `//compiler/daml-licenses:daml-licenses`, which
itself was unused.

* Remove unused notices-gen

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-03-26 17:33:11 +00:00
Moritz Kiefer
c004c90fcb
Remove skydoc (#5208)
Nobody uses it, it’s deprecated anyway.

changelog_begin
changelog_end
2020-03-26 11:14:34 +00:00
Moritz Kiefer
be7f0802c6
Bump ghcide (#5103)
This includes some completion improvements.

changelog_begin
changelog_end
2020-03-20 16:43:59 +00:00
Andreas Herrmann
559c78003e
Update rules_haskell (#4751)
* Update rules_haskell

The workaround for linking against `Cffi` in the REPL has been
upstreamed in a more generalized form.

CHANGELOG_BEGIN
CHANGELOG_END

* ghcide: Use rules_haskell's hie-bios support

* Document `ghcide` Bazel integration

* Rename files to match module names

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-03-13 16:49:34 +01:00
Gerolf Seitz
d617922618
Remove InMemoryKVParticipantState (#4674)
This removes the sample/reference implementation of kvutils
InMemoryKVParticipantState.
This used to be the only implementation of kvutils, but now with the
simplified kvutils api we have ledger-on-memory and ledger-on-sql.

InMemoryKVParticipantState was also used for the ledger dump utility,
which now uses ledger-on-memory.

* Runner now supports a multi participant configuration
This change removes the "extra participants" config and goes for consistent
participant setup with --participant.

* Run all conformance tests in the repository in verbose mode.

This means we'll print stack traces on error, which should make it
easier to figure out what's going on with flaky tests on CI.

This doesn't change the default for other users of the
ledger-api-test-tool; we just add the flag for:

  - ledger-api-test-tool-on-canton
  - ledger-on-memory
  - ledger-on-sql
  - sandbox




Fixes #4225.

CHANGELOG_BEGIN
CHANGELOG_END
2020-02-26 15:45:35 +01:00
Moritz Kiefer
cde562600c
Bump rules_haskell (#4687)
* Bump rules_haskell

Still checking if that helps with GHC 8.8 but we should upgrade this
either way.

changelog_begin
changelog_end

* disable grpc patch

* shut up buildifier

* delete unused ghci grpc patch

* Fix Cffi library not found issues

* Update deps.bzl

Co-Authored-By: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>

Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
2020-02-26 10:25:56 +00:00
Gary Verhaegen
5a117dc358
introduce new release process (#4513)
Context
=======

After multiple discussions about our current release schedule and
process, we've come to the conclusion that we need to be able to make a
distinction between technical snapshots and marketing releases. In other
words, we need to be able to create a bundle for early adopters to test
without making it an officially-supported version, and without
necessarily implying everyone should go through the trouble of
upgrading. The underlying goal is to have less frequent but more stable
"official" releases.

This PR is a proposal for a new release process designed under the
following constraints:

- Reuse as much as possible of the existing infrastructure, to minimize
  effort but also chances of disruptions.
- Have the ability to create "snapshot"/"nightly"/... releases that are
  not meant for general public consumption, but can still be used by savvy
  users without jumping through too many extra hoops (ideally just
  swapping in a slightly-weirder version string).
- Have the ability to promote an existing snapshot release to "official"
  release status, with as few changes as possible in-between, so we can be
  confident that the official release is what we tested as a prerelease.
- Have as much of the release pipeline shared between the two types of
  releases, to avoid discovering non-transient problems while trying to
  promote a snapshot to an official release.
- Triggerring a release should still be done through a PR, so we can
  keep the same approval process for SOC2 auditability.

The gist of this proposal is to replace the current `VERSION` file with
a `LATEST` file, which would have the following format:

```
ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7
```

This file would be maintained with a script to reduce manual labor in
producing the version string. Other than that, the process will be
largely the same, with releases triggered by changes to this `LATEST`
and the release notes files.

Version numbers
===============

Because one of the goals is to reduce the velocity of our published
version numbers, we need a different version scheme for our snapshot
releases. Fortunately, most version schemes have some support for that;
unfortunately, the SDK sits at the intersection of three different
version schemes that have made incompatible choices. Without going into
too much detail:

- Semantic versioning (which we chose as the version format for the SDK
  version number) allows for "prerelease" version numbers as well as
  "metadata"; an example of a complete version string would be
  `1.2.3-nightly.201+server12.43`. The "main" part of the version string
  always has to have 3 numbers separated by dots; the "prerelease"
  (after the `-` but before the `+`) and the "metadata" (after the `+`)
  parts are optional and, if present, must consist of one or more segments
  separated by dots, where a segment can be either a number or an
  alphanumeric string. In terms of ordering, metadata is irrelevant and
  any version with a prerelease string is before the corresponding "main"
  version string alone. Amongst prereleases, segments are compared in
  order with purely numeric ones compared as numbers and mixed ones
  compared lexicographically. So 1.2.3 is more recent than 1.2.3-1,
  which is itself less recent than 1.2.3-2.
- Maven version strings are any number of segments separated by a `.`, a
  `-`, or a transition between a number and a letter. Version strings
  are compared element-wise, with numeric segments being compared as
  numbers. Alphabetic segments are treated specially if they happen to be
  one of a handful of magic words (such as "alpha", "beta" or "snapshot"
  for example) which count as "qualifiers"; a version string with a
  qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`,
  which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a
  special ordering amongst qualifiers. Other alphabetic segments are
  compared alphabetically and count as being "after" their prefix
  (`1.2.3-really-final-this-time` counts as being released after `1.2.3`).
- GHC package numbers are comprised of any number of numeric segments
  separated by `.`, plus an optional (though deprecated) alphanumeric
  "version tag" separated by a `-`. I could not find any official
  documentation on ordering for the version tag; numeric segments are
  compared as numbers.
- npm uses semantic versioning so that is covered already.

After much more investigation than I'd care to admit, I have come up
with the following compromise as the least-bad solution. First,
obviously, the version string for stable/marketing versions is going to
be "standard" semver, i.e. major.minor.patch, all numbers, which works,
and sorts as expected, for all three schemes. For snapshot releases, we
shall use the following (semver) format:

```
0.13.53-alpha.20200214.3025.ef5d32b7
```

where the components are, respectively:

- `0.13.53`: the expected version string of the next "stable" release.
- `alpha`: a marker that hopefully scares people enough.
- `20200214`: the date of the release commit, which _MUST_ be on
  master.
- `3025`: the number of commits in master up to the release commit
  (included). Because we have a linear, append-only master branch, this
  uniquely identifies the commit.
- `ef5d32b7ù : the first 8 characters of the release commit sha. This is
  not strictly speaking necessary, but makes it a lot more convenient to
  identify the commit.

The main downsides of this format are:

1. It is not a valid format for GHC packages. We do not publish GHC
  packages from the SDK (so far we have instead opted to release our
  Haskell code as separate packages entirely), so this should not be an
  issue. However, our SDK version currently leaks to `ghc-pkg` as the
  version string for the stdlib (and prim) packages. This PR addresses
  that by tweaking the compiler to remove the offending bits, so `ghc-pkg`
  would see the above version number as `0.13.53.20200214.3025`, which
  should be enough to uniquely identify it. Note that, as far as I could
  find out, this number would never be exposed to users.
2. It is rather long, which I think is good from a human perspective as
  it makes it more scary. However, I have been told that this may be
  long enough to cause issues on Windows by pushing us past the max path
  size limitation of that "OS". I suggest we try it and see what
  happens.

The upsides are:

- It clearly indicates it is an unstable release (`alpha`).
- It clearly indicates how old it is, by including the date.
- To humans, it is immediately obvious which version is "later" even if
  they have the same date, allowing us to release same-day patches if
  needed. (Note: that is, commits that were made on the same day; the
  release date itself is irrelevant here.)
- It contains the git sha so the commit built for that release is
  immediately obvious.
- It sorts correctly under all schemes (modulo the modification for
  GHC).

Alternatives I considered:

- Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would
  be accepted by all schemes, but will not sort as expected under semantic
  versioning (though Maven will be fine). I have no idea how it will sort
  under GHC.
- Not having any non-numeric component, e.g. `0.13.53.20200214.3025`.
  This is not valid semantic versioning and is therefore rejected by
  npm.
- Not having detailed info: just go with `0.13.53-snapshot`. This is
  what is generally done in the Java world, but we then lose track of what
  version is actually in use and I'm concerned about bug reports. This
  would also not let us publish to the main Maven repo (at least not more
  than once), as artifacts there are supposed to be immutable.
- No having a qualifier: `0.13.53-3025` would be acceptable to all three
  version formats. However, it would not clearly indicate to humans that
  it is not meant as a stable version, and would sort differently under
  semantic versioning (which counts it as a prerelease, i.e. before
  `0.13.53`) than under maven (which counts it as a patch, so after
  `0.13.53`).
- Just counting releases: `0.13.53-alpha.1`, where we just count the
  number of prereleases in-between `0.13.52` and the next. This is
  currently the fallback plan if Windows path length causes issues. It
  would be less convenient to map releases to commits, but it could still
  be done via querying the history of the `LATEST` file.

Release notes
=============

> Note: We have decided not to have release notes for snapshot releases.

Release notes are a bit tricky. Because we want the ability to make
snapshot releases, then later on promote them to stable releases, it
follows that we want to build commits from the past. However, if we
decide post-hoc that a commit is actually a good candidate for a
release, there is no way that commit can have the appropriate release
notes: it cannot know what version number it's getting, and, moreover,
we now track changes in commit messages. And I do not think anyone wants
to go back to the release notes file being a merge bottleneck.

But release notes need to be published to the releases blog upon
releasing a stable version, and the docs website needs to be updated and
include them.

The only sensible solution here is to pick up the release notes as of
the commit that triggers the release. As the docs cron runs
asynchronously, this means walking down the git history to find the
relevant commit.

> Note: We could probably do away with the asynchronicity at this point.
> It was originally included to cover for the possibility of a release
> failing. If we are releasing commits from the past after they have been
> tested, this should not be an issue anymore. If the docs generation were
> part of the synchronous release step, it would have direct access to the
> correct release notes without having to walk down the git history.
>
> However, I think it is more prudent to keep this change as a future step,
> after we're confident the new release scheme does indeed produce much more
> reliable "stable" releases.

New release process
===================

Just like releases are currently controlled mostly by detecting
changes to the `VERSION` file, the new process will be controlled by
detecting changes to the `LATEST` file. The format of that file will
include both the version string and the corresponding SHA.

Upon detecting a change to the `LATEST` file, CI will run the entire
release process, just like it does now with the VERSION file. The main
differences are:

1. Before running the release step, CI will checkout the commit
  specified in the LATEST file. This requires separating the release
  step from the build step, which in my opinion is cleaner anyway.
2. The `//:VERSION` Bazel target is replaced by a repository rule
  that gets the version to build from an environment variable, with a
  default of `0.0.0` to remain consistent with the current `daml-head`
  behaviour.

Some of the manual steps will need to be skipped for a snapshot release.
See amended `release/RELEASE.md` in this commit for details.

The main caveat of this approach is that the official release will be a
different binary from the corresponding snapshot. It will have been
built from the same source, but with a different version string. This is
somewhat mitigated by Bazel caching, meaning any build step that does
not depend on the version string should use the cache and produce
identical results. I do not think this can be avoided when our artifact
includes its own version number.

I must note, though, that while going through the changes required after
removing the `VERSION` file, I have been quite surprised at the sheer number of
things that actually depend on the SDK version number. I believe we should
look into reducing that over time.

CHANGELOG_BEGIN
CHANGELOG_END
2020-02-25 17:01:23 +01:00
Samir Talwar
9bdfbb0534
Avoid opening a server to the world when finding a free port. (#4683)
* Avoid opening a server to the world when finding a free port.

This is very annoying on macOS because we get a focus-stealing popup for
a split second, asking for permission to allow the server through the
firewall. The popup pretty much always disappears before it can even be
read, when the server is closed.

This is almost certainly not an attack vector, because:
  - we only do this in tests,
  - the server is open for only a few milliseconds,
  - nothing is served,
  - and finding the port is tricky, because it's effectively random.

Nevertheless, it's very annoying.

CHANGELOG_BEGIN
CHANGELOG_END

* Extract a Bazel package for finding free ports.

We seem to do it in 4 different places, which I think is enough to
remove the duplication.
2020-02-25 13:09:28 +00:00
Moritz Kiefer
8d81399c0f
Add an experimental DAML script REPL (#4660)
As mentioned in the title, this is still very experimental and needs
more work before we want to advertise it. However, the code is in a
somewhat reasonable shape, there are tests and I think even in the
current state it is already useful. Also this PR is already getting
very large so I don’t want to hold off much longer before merging this.

It is included in the SDK but hidden from `damlc --help` and `daml
--help` until the most pressing issues are addressed (primarily around
making sure that it doesn’t just shut down if you have a type error
and better error messages in general).

changelog_begin
changelog_end
2020-02-24 11:06:27 +01:00
Moritz Kiefer
668373ca85
Embed shake data-files via TH (#4584)
This is a bit ugly but after spending some time digging into the
issues in rules_haskell around data-files, this seems like the most
sensible option especially given that we also want to ship them in the
SDK which woud require additional work even if we do fix it in
rules_haskell.

fixes #4457

changelog_begin
changelog_end
2020-02-18 17:53:27 +01:00
Moritz Kiefer
d68d197948
Disable scaladoc on the MacOS CI (#4524)
* Disable scaladoc on the MacOS CI

It is still built by default locally.

fixes #4441

changelog_begin
changelog_end

* Change tag name
2020-02-14 14:08:24 +00:00