Commit Graph

101 Commits

Author SHA1 Message Date
Gary Verhaegen
80652bd51f
report-std-change: handle GitHub errors (#4814)
There have been a few GitHub glitches last week that resulted in a few
commits on master not being associated with a PR (though they really
were created from merging a PR, and the correct PR number is in their
title).

This makes the report script crash on not finding the PR, so this PR
fixes that. And a comment.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-05 10:06:58 +01:00
Gary Verhaegen
c2240df083
change standard change report time (#4804)
In the current setup, the report generation starts at 4. However, the
daily killing of all machines also starts at 4, giving the report little
chance of ever finishing.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-04 18:22:32 +01:00
Gary Verhaegen
6a1c75cfd4
fix Windows signing (maybe) (#4753)
During the latest attempt at making a snapshot release (#4749),
everything seemingly went well except for the step of signing the
Windows installer.

Based on a very obscure erorr message (Access Denied) and a bit of
Google search, my current hypothesis is that the signing fails because
the artifact produced by Bazel is read-only. This was not an issue in
the previous setup because we were getting the installer from an Azure
internal download between different jobs. We are now getting the binary
directly from Bazel.

This also adds a `set -e` to the relevant Bash snippet, because ideally
they should always have one.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-04 13:18:49 +01:00
Gary Verhaegen
86bce50b9a
fix passing is_release through (#4745)
Somehow, in the current setup, the publish steps do not get executed on
master. This is what Azure reports:

```
Evaluating: and(succeeded(), eq('$(is_release)', 'true'),
eq(variables['Build.SourceBranchName'], 'master'), eq('linux', 'linux'))
Expanded: and(True, eq('$(is_release)', 'true'),
eq(variables['Build.SourceBranchName'], 'master'), eq('linux', 'linux'))
Result: False
```

So it looks like, in the condition, `${{parameters.is_release}}`
evaluates to the literal string `$(is_release)`. If we look at the point
of invocation of the ~function~ template, we can see:

```
      - template: ci/build-unix.yml
        parameters:
          release_tag: $(release_tag)
          name: 'linux'
          is_release: $(is_release)
```

so it does not seem completely crazy. However, according to the
documentation, we should expect that to be replaced by the value of the
corresponding variable, as per:

```
    variables:
      release_sha: $[ dependencies.check_for_release.outputs['out.release_sha'] ]
      release_tag: $[ coalesce(dependencies.check_for_release.outputs['out.release_tag'], '0.0.0') ]
      trigger_sha: $[ dependencies.check_for_release.outputs['out.trigger_sha'] ]
      is_release: $[ dependencies.check_for_release.outputs['out.is_release'] ]
```

What's interesting here is that, within `build-unix.yml`, we are also
using `release_tag` in the exact same way:

```
  - bash: ./build.sh "_$(uname)"
    displayName: 'Build'
    env:
      DAML_SDK_RELEASE_VERSION: ${{parameters.release_tag}}
```

and this time output from the build seems to show the value being
correctly substituted:

```
damlc - Compiler and IDE backend for the Digital Asset Modelling
Language
SDK Version: 0.13.55-snapshot.20200226.3266.d58bb459

Usage: <interactive> COMMAND
  Invoke the DAML compiler. Use -h for help.
```

My current guess is that the (undocumented, as far as I can tell)
evaluation order is as follows:

1. In the template, syntactically replace all the parameters.
2. In the job definition, replace the call to the template with the code
of the template. So it is as if we had written the template directly in
the `azure-pipelines.yml` file, with `$(release_tag)` and
`$(is_release)`.
3. Run the build. When we reach the time to run this specific job,
we can evaluate the expressions for the variables and replace them in
the rest of the job.

So what is going wrong? I believe the issue is with the quotes,
preventing the substitution of `is_release`. They came directly from the
[documented
syntax](https://docs.microsoft.com/en-us/azure/devops/pipelines/process/conditions?view=azure-devops&tabs=yaml#use-a-template-parameter-as-part-of-a-condition),
but if the above evaluation order is correct, they should not be there.

There are actually two things going wrong here. The first one is that
the syntax `$()` is used to substitute a value in what Azure considers a
string. This is the case for `env` keys. However, the `condition` key
is not a string, it is an Azure "expression". Expressions have their own
evaluation rules and syntax, and in particular, `$()` is not a
substitution rule there, so when it sees `$()` in a string in an
expression (due to the quoptes), it leaves it alone.

Removing the quotes does not directly help, though, as we then end with

```
condition: eq($(is_release), 'true')
```

and `$()` is not valid syntax in an expression. The way to use variables
in an expression is `variables.name` (or `variables["name"]`, because
why have only one?).

So that means we have to pass variables to the template in different
ways depending on how they will be used. So much fun.

CHANGELOG_BEGIN
CHANGELOG_END
2020-02-27 14:33:20 +01:00
Gary Verhaegen
1c2b921c14
fix tarball generation (#4738)
The existing approach is a historical accident. The reason for the
additional tarball/install jobs was that, in my original attempt, the
build steps would still build the current commit, as opposed to the
target commit.

This is not such an issue on Linux, but setting up the build environment
on Windows and macOS _again_ for no good reason is a pure waste of time
(and effort in getting it right). Now that the build steps build the
target commit (with the env var set), we can go back to the way things
were previously: just take the build products directly from the build
step.

CHANGELOG_BEGIN
CHANGELOG_END
2020-02-27 10:48:38 +01:00
Moritz Kiefer
8c14d16718
Disable pdf docs builds on macos (#4724)
This disables the PDF docs builds on MacOS on CI (they are still built
locally by default) and removes them from the Nix closure by
introducing a separate ci-cached attribute that filters out texlive.

Since we built `nix-build nix -A tools -A cached` on CI, I’ve also
removed all the Tex stuff from tools which only means that it ends up
in PATH which nobody seems to care about.

changelog_begin
changelog_end
2020-02-26 14:52:08 +00:00
Gary Verhaegen
ed8c68f0a0
notify Samir on #team-daml-ci (#4720)
CHANGELOG_BEGIN
CHANGELOG_END
2020-02-26 13:11:13 +00:00
Gary Verhaegen
5a117dc358
introduce new release process (#4513)
Context
=======

After multiple discussions about our current release schedule and
process, we've come to the conclusion that we need to be able to make a
distinction between technical snapshots and marketing releases. In other
words, we need to be able to create a bundle for early adopters to test
without making it an officially-supported version, and without
necessarily implying everyone should go through the trouble of
upgrading. The underlying goal is to have less frequent but more stable
"official" releases.

This PR is a proposal for a new release process designed under the
following constraints:

- Reuse as much as possible of the existing infrastructure, to minimize
  effort but also chances of disruptions.
- Have the ability to create "snapshot"/"nightly"/... releases that are
  not meant for general public consumption, but can still be used by savvy
  users without jumping through too many extra hoops (ideally just
  swapping in a slightly-weirder version string).
- Have the ability to promote an existing snapshot release to "official"
  release status, with as few changes as possible in-between, so we can be
  confident that the official release is what we tested as a prerelease.
- Have as much of the release pipeline shared between the two types of
  releases, to avoid discovering non-transient problems while trying to
  promote a snapshot to an official release.
- Triggerring a release should still be done through a PR, so we can
  keep the same approval process for SOC2 auditability.

The gist of this proposal is to replace the current `VERSION` file with
a `LATEST` file, which would have the following format:

```
ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7
```

This file would be maintained with a script to reduce manual labor in
producing the version string. Other than that, the process will be
largely the same, with releases triggered by changes to this `LATEST`
and the release notes files.

Version numbers
===============

Because one of the goals is to reduce the velocity of our published
version numbers, we need a different version scheme for our snapshot
releases. Fortunately, most version schemes have some support for that;
unfortunately, the SDK sits at the intersection of three different
version schemes that have made incompatible choices. Without going into
too much detail:

- Semantic versioning (which we chose as the version format for the SDK
  version number) allows for "prerelease" version numbers as well as
  "metadata"; an example of a complete version string would be
  `1.2.3-nightly.201+server12.43`. The "main" part of the version string
  always has to have 3 numbers separated by dots; the "prerelease"
  (after the `-` but before the `+`) and the "metadata" (after the `+`)
  parts are optional and, if present, must consist of one or more segments
  separated by dots, where a segment can be either a number or an
  alphanumeric string. In terms of ordering, metadata is irrelevant and
  any version with a prerelease string is before the corresponding "main"
  version string alone. Amongst prereleases, segments are compared in
  order with purely numeric ones compared as numbers and mixed ones
  compared lexicographically. So 1.2.3 is more recent than 1.2.3-1,
  which is itself less recent than 1.2.3-2.
- Maven version strings are any number of segments separated by a `.`, a
  `-`, or a transition between a number and a letter. Version strings
  are compared element-wise, with numeric segments being compared as
  numbers. Alphabetic segments are treated specially if they happen to be
  one of a handful of magic words (such as "alpha", "beta" or "snapshot"
  for example) which count as "qualifiers"; a version string with a
  qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`,
  which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a
  special ordering amongst qualifiers. Other alphabetic segments are
  compared alphabetically and count as being "after" their prefix
  (`1.2.3-really-final-this-time` counts as being released after `1.2.3`).
- GHC package numbers are comprised of any number of numeric segments
  separated by `.`, plus an optional (though deprecated) alphanumeric
  "version tag" separated by a `-`. I could not find any official
  documentation on ordering for the version tag; numeric segments are
  compared as numbers.
- npm uses semantic versioning so that is covered already.

After much more investigation than I'd care to admit, I have come up
with the following compromise as the least-bad solution. First,
obviously, the version string for stable/marketing versions is going to
be "standard" semver, i.e. major.minor.patch, all numbers, which works,
and sorts as expected, for all three schemes. For snapshot releases, we
shall use the following (semver) format:

```
0.13.53-alpha.20200214.3025.ef5d32b7
```

where the components are, respectively:

- `0.13.53`: the expected version string of the next "stable" release.
- `alpha`: a marker that hopefully scares people enough.
- `20200214`: the date of the release commit, which _MUST_ be on
  master.
- `3025`: the number of commits in master up to the release commit
  (included). Because we have a linear, append-only master branch, this
  uniquely identifies the commit.
- `ef5d32b7ù : the first 8 characters of the release commit sha. This is
  not strictly speaking necessary, but makes it a lot more convenient to
  identify the commit.

The main downsides of this format are:

1. It is not a valid format for GHC packages. We do not publish GHC
  packages from the SDK (so far we have instead opted to release our
  Haskell code as separate packages entirely), so this should not be an
  issue. However, our SDK version currently leaks to `ghc-pkg` as the
  version string for the stdlib (and prim) packages. This PR addresses
  that by tweaking the compiler to remove the offending bits, so `ghc-pkg`
  would see the above version number as `0.13.53.20200214.3025`, which
  should be enough to uniquely identify it. Note that, as far as I could
  find out, this number would never be exposed to users.
2. It is rather long, which I think is good from a human perspective as
  it makes it more scary. However, I have been told that this may be
  long enough to cause issues on Windows by pushing us past the max path
  size limitation of that "OS". I suggest we try it and see what
  happens.

The upsides are:

- It clearly indicates it is an unstable release (`alpha`).
- It clearly indicates how old it is, by including the date.
- To humans, it is immediately obvious which version is "later" even if
  they have the same date, allowing us to release same-day patches if
  needed. (Note: that is, commits that were made on the same day; the
  release date itself is irrelevant here.)
- It contains the git sha so the commit built for that release is
  immediately obvious.
- It sorts correctly under all schemes (modulo the modification for
  GHC).

Alternatives I considered:

- Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would
  be accepted by all schemes, but will not sort as expected under semantic
  versioning (though Maven will be fine). I have no idea how it will sort
  under GHC.
- Not having any non-numeric component, e.g. `0.13.53.20200214.3025`.
  This is not valid semantic versioning and is therefore rejected by
  npm.
- Not having detailed info: just go with `0.13.53-snapshot`. This is
  what is generally done in the Java world, but we then lose track of what
  version is actually in use and I'm concerned about bug reports. This
  would also not let us publish to the main Maven repo (at least not more
  than once), as artifacts there are supposed to be immutable.
- No having a qualifier: `0.13.53-3025` would be acceptable to all three
  version formats. However, it would not clearly indicate to humans that
  it is not meant as a stable version, and would sort differently under
  semantic versioning (which counts it as a prerelease, i.e. before
  `0.13.53`) than under maven (which counts it as a patch, so after
  `0.13.53`).
- Just counting releases: `0.13.53-alpha.1`, where we just count the
  number of prereleases in-between `0.13.52` and the next. This is
  currently the fallback plan if Windows path length causes issues. It
  would be less convenient to map releases to commits, but it could still
  be done via querying the history of the `LATEST` file.

Release notes
=============

> Note: We have decided not to have release notes for snapshot releases.

Release notes are a bit tricky. Because we want the ability to make
snapshot releases, then later on promote them to stable releases, it
follows that we want to build commits from the past. However, if we
decide post-hoc that a commit is actually a good candidate for a
release, there is no way that commit can have the appropriate release
notes: it cannot know what version number it's getting, and, moreover,
we now track changes in commit messages. And I do not think anyone wants
to go back to the release notes file being a merge bottleneck.

But release notes need to be published to the releases blog upon
releasing a stable version, and the docs website needs to be updated and
include them.

The only sensible solution here is to pick up the release notes as of
the commit that triggers the release. As the docs cron runs
asynchronously, this means walking down the git history to find the
relevant commit.

> Note: We could probably do away with the asynchronicity at this point.
> It was originally included to cover for the possibility of a release
> failing. If we are releasing commits from the past after they have been
> tested, this should not be an issue anymore. If the docs generation were
> part of the synchronous release step, it would have direct access to the
> correct release notes without having to walk down the git history.
>
> However, I think it is more prudent to keep this change as a future step,
> after we're confident the new release scheme does indeed produce much more
> reliable "stable" releases.

New release process
===================

Just like releases are currently controlled mostly by detecting
changes to the `VERSION` file, the new process will be controlled by
detecting changes to the `LATEST` file. The format of that file will
include both the version string and the corresponding SHA.

Upon detecting a change to the `LATEST` file, CI will run the entire
release process, just like it does now with the VERSION file. The main
differences are:

1. Before running the release step, CI will checkout the commit
  specified in the LATEST file. This requires separating the release
  step from the build step, which in my opinion is cleaner anyway.
2. The `//:VERSION` Bazel target is replaced by a repository rule
  that gets the version to build from an environment variable, with a
  default of `0.0.0` to remain consistent with the current `daml-head`
  behaviour.

Some of the manual steps will need to be skipped for a snapshot release.
See amended `release/RELEASE.md` in this commit for details.

The main caveat of this approach is that the official release will be a
different binary from the corresponding snapshot. It will have been
built from the same source, but with a different version string. This is
somewhat mitigated by Bazel caching, meaning any build step that does
not depend on the version string should use the cache and produce
identical results. I do not think this can be avoided when our artifact
includes its own version number.

I must note, though, that while going through the changes required after
removing the `VERSION` file, I have been quite surprised at the sheer number of
things that actually depend on the SDK version number. I believe we should
look into reducing that over time.

CHANGELOG_BEGIN
CHANGELOG_END
2020-02-25 17:01:23 +01:00
Gary Verhaegen
433f484188
stop producing/publishing the Bazel execution logs (#4527)
They can weigh close to 1GB, and the internal Azure networks are
unreliable enough that this adds up significant amounts of time to our
already slow CI pipeline (usually around a minute, but I've seen at
least one case where it took almost 28 minutes).

Given that we pretty much never look a them, 🔥.

CHANGELOG_BEGIN
CHANGELOG_END
2020-02-14 14:34:01 +01:00
Gary Verhaegen
d96b225dad
remove jfrog credentials after use (#4522)
Not critically important, but it seems like good hygiene.

CHANGELOG_BEGIN
CHANGELOG_END
2020-02-14 13:54:28 +01:00
nickchapman-da
d084dba531
Subscribe to CI notifications (#4494)
changelog_begin
changelog_end
2020-02-12 17:36:03 +00:00
Andreas Herrmann
655e4b3b55
Update CI nix version (#4443)
* Update CI nix version

For `--option http2 false` to take effect requires Nix 2.3.2.

CHANGELOG_BEGIN
CHANGELOG_END

* Set option `http2 = false` dev-env nix config

This is less likely to overlook an instance than manually adding
`--option http2 false` to each Nix invocation.

Setting `--option htt2p false` also had no effect on the multi-user Nix
installation on the Linux CI machines due to
```
WARNING: option '--disk_cache' was expanded to from both option '--config linux' (source /nix/store/2xnfb2l39d2b4nxw5vwmqz5hjwhw0caw-daml-bazelrc) and option '--config linux' (source /nix/store/2xnfb2l39d2b4nxw5vwmqz5hjwhw0caw-daml-bazelrc)
```

Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
2020-02-07 15:05:52 +00:00
Gary Verhaegen
064b26c75e
standard change extract script (#4416)
As part of our SOC2/ISO certification, we need to be able to evidence a
list of "Standard Changes" for the DAML SDK project. This commit adds a
script that extracts, for a given month, the list of Standard Changes
that happened along with relevant information (author, reviewer, etc.).

It also adds a definition for a monthly cron job on Azure to run the
script and send the result to Slack, @-mentioning Martin.

CHANGELOG_BEGIN
CHANGELOG_END
2020-02-07 15:16:03 +01:00
Gary Verhaegen
47bd131f15
add copyright headers to yml files (#4407)
We seem to have forgotten about them in the copyright scripts.

CHANGELOG_BEGIN
CHANGELOG_END
2020-02-06 12:54:07 +01:00
Gary Verhaegen
00224c2480
ci: alert Slack on PR build completion (#4286)
One of the outputs of our brainstorming about how to make CI better was
that it is annoying to have to "babysit" pull requests. This PR attempts
to introduce a notification mechanism by which Azure will notify people
on Slack when a build finishes, so they know they need to go and rerun
or merge the corresponding PR.

This commit also changes the existing $Slack.URL variable to
$Slack.team-daml, to make more explicit where the Slack message is being
sent to (Slack works with one token per destination channel). Both
$Slack.URL and $Slack.team-daml are currently defined as the same token
in Azure.

CHANGELOG_BEGIN
CHANGELOG_END
2020-02-03 16:29:13 +01:00
Moritz Kiefer
60e32a98c4
Disable release step on macos (#4330)
Originally we ran the release step on both Linux and MacOS to handle
platform dependent artifacts, in particular, damlc.jar. However, we
don’t have any platform dependent artifacts that are uploaded as part
of the release script anymore and I hope we will never have to add any
in the future.

So this PR, removes the code for handling platform dependent artifacts
in the release step and disables the release step on MacOS (while
still setting the variables like we do on Windows).

Currently the release step still costs us ~2 minutes on MacOS which is
already our slowest platform so hopefully this will speed things up a
bit.

changelog_begin
changelog_end
2020-01-31 17:08:47 +01:00
Moritz Kiefer
1a0fb138fb
Ignore missing maven deps on macos (#4201)
Currently, pretty much all of our builds are bottlenecked on
MacOS (mainly because the builders are slower and have worse
caching). The release step adds > 3min to each build which is a bit
annoying. It turns out that removing the calls to `bazel query` which
are used to check for missing Maven dependencies speeds this up by >
5x. Given that the check is platform independent anyway, we can just
disable it on MacOS.

changelog_begin
changelog_end
2020-01-24 16:17:08 +00:00
Robin Krom
811d740df5
upload typescript packages to npm registry (#4185)
* make packages public

This uploads the typescript npm packages of the language support to the
npm registry in the release process.

CHANGELOG_BEGIN
CHANGELOG_END

* address moritz/gary's review

* generate the .npmrc file

* adding debug output

just in case the upload will fail in the next release.

* reverse package order
2020-01-23 18:31:50 +01:00
Gary Verhaegen
8a1b46f4fd
docs cron: use GitHub-Flavoured Markdown (#4141)
This patch changes the call to the GitHub API that translates the
release notes from markdown to HTML to use gfm instead of plain
markdown. gfm is a superset of markdown that adds the following:

- GitHub usernames (`@`-mentions) are turned into links to the user's
  profile page.
- Issues and PR numbers (`#1234`) are turned into links to the
  corresponding issue or PR.
- Existing git shas are turned into links to the corresponding commit.

An example of this feature missing is the release notes for
[v0.13.42](https://blog.daml.com/release-notes/0.13.42-1), where
intended links such as

> - Rename argument in active contract to payload. See #3826.

are not rendered.

CHANGELOG_BEGIN
CHANGELOG_END
2020-01-21 14:46:02 +01:00
Gary Verhaegen
f2827e0207
docs cron: remove links to missing versions (#4123)
We have recently added the option for this script to not build some
versions (because they are too old and external dependencies have
changed from under them). We also have changed the GitHub call to get
all the history of releases.

This PR changes the logic to generate the `versions.json` file so that
it only contains versions that we have either built or copied over.

Consequently, it also changes the logic to decide whether this job
should run to depend only on the latest version, rather than the whole
list of versions.

CHANGELOG_BEGIN
CHANGELOG_END
2020-01-21 13:50:40 +01:00
Gary Verhaegen
760f9d4d37
docs cron: follow github pagination links (#4115)
The GitHub API is paginated (30 items by default). This creates two
problems:

1. At the moment, older versions silently drop from the docs website,
  without us having made any explicit decision about it.
2. When we prepare a new version, it gets created as a pre-release
  version. Our script filters that out, but that happens on our end so
  we end up with 29 published versions and the list is different form the
  existing one. If the prerelease then gets dropped, the oldest version
  comes back.

It is possible that we will sometime decide we do not want to keep old
documentation around forever, but that should be an explicit decision.
This patch changes the logic to fetch the list of versions from GitHub
so that we always get all the published versions (barring race
conditions inherent to that kind of paginated API).

CHANGELOG_BEGIN
CHANGELOG_END
2020-01-20 18:47:47 +01:00
Gary Verhaegen
8811006617
docs cron: more reliable checksums (#4102)
The docs build is currently not reproducible as it include to-the-minute
time-of-build information. It also includes some Sphinx binary caches
which I suppose will also not be reproducible (though I have not checked
the details there).

This commit attempts to remove all sources of non-reproducibility from
the docs build, though this is hard to test without having a stable,
older release to compare with.

CHANGELOG_BEGIN
CHANGELOG_END
2020-01-20 16:21:34 +01:00
Gary Verhaegen
b8a588e9c0
docs cron: sort versions.json (#4062)
CHANGELOG_BEGIN
CHANGELOG_END
2020-01-16 12:32:31 +01:00
Gary Verhaegen
11be496e15
docs cron: create temp dirs (#4061)
CHANGELOG_BEGIN
CHANGELOG_END
2020-01-16 00:40:26 +01:00
Gary Verhaegen
45c474b3d5
try to fix docs again (#4060)
CHANGELOG_BEGIN
CHANGELOG_END
2020-01-15 22:50:46 +01:00
Gary Verhaegen
ada0ad07ca
docs cron: special case 0.13.43 for scala http issue (#4058)
CHANGELOG_BEGIN
CHANGELOG_END
2020-01-15 21:48:37 +01:00
Gary Verhaegen
e96db012ed
fix logic bug in docs release cron (#3998)
The latest changes to the docs cron have introduced a bug whereby the
"latest" version is determined including prereleases.

CHANGELOG_BEGIN
CHANGELOG_END
2020-01-09 13:02:42 +01:00
Gary Verhaegen
40fd4b3626
docs cron: do not rebuild old versions (#3944)
This commit makes two conceptually independent changes:

1. It adds a checksum file to each version folder. This allows the
script to detect when a version has not been correctly uploaded.
2. It changes the script to first download all the docs website, and
then reuse existing version folders where appropriate (i.e. when their
folder matches its checksum).

The hope is that this will reduce the time it takes to deploy a new
version, as only the current version should be rebuilt (in addition to
previous, failed versions).

The first time this cron runs (upon next release as per the current
setup), however, it will still rebuild all existing versions as they do
not currently have a checksum.

CHANGELOG_BEGIN
CHANGELOG_END
2020-01-09 12:17:35 +01:00
Gary Verhaegen
89af1550b1
check for changelog (#3963)
* check for changelog
2020-01-07 17:19:50 +01:00
Moritz Kiefer
42c586f8d4
Bump ghcide (#3943)
* Bump ghcide

* Fix ghcide build

* Include bugfix for Windows
2020-01-04 07:51:51 +01:00
Gary Verhaegen
f8c247cadf
partial fix for docs cron (#3941)
This commit aims at mitigating two issues we have noticed with the
0.13.41 release:

1. The initial cron run for that release got interrupted at the 50
minutes mark, which happened to be right in the middle of the s3 upload.
This means it had already changed the versions.json file, but had not
finished updating the actual html files. Right now, the docs.daml.com
website shows version 0.13.41 in the drop-down, but actually displays
the content for 0.13.40. Additionally, trying to explicitly visit the
website for 0.13.41 (https://docs.daml.com/0.13.41) yields a 404. Note
that this also means the cron job did not reach the "tell HubSpot"
point, so 0.13.41 did not get announced.
2. As the script also did not reach the "clear cache" step, subsequent
runs have been rebuilding the documentation for no reason as the
sequence of steps was: check versions.json through HTTP, get cached one,
see it's not up-to-date, build docs, check versions.json through s3 API,
bypassing the cache, see it's up-to-date, stop.

To address those issues, this PR changes the cron to:
1. Increase the timeout to 2h instead of 50 minutes.
2. Always check the versions.json file through s3, rather than go
through the HTTP cache first.

These are not complete solutions but I'm not sure how to do better given
that s3 does not have atomic operations.
2020-01-03 14:43:22 +01:00
Gary Verhaegen
878429e3bf
update copyright notices to 2020 (#3939)
copyright update 2020

* update template
* run script: `dade-copyright-headers update .`
* update script
* manual adjustments
* exclude frozen proto files from further header checks (by adding NO_AUTO_COPYRIGHT files)
2020-01-02 21:21:13 +01:00
Peter Garmaz
03d55fa86a daml-sdk container improvements (#3902)
* re-add cleanup for /tmp to remove 700ish mb of unneeded temp files made by the sdk installer

* Set WORKDIR to daml user home dir so that sdk tools can create files

* add daml sdk config defaults for auto-install and update-check sdk install RUN command

* add --no-cache to apk add to reduce size a little

* add line return to end of daml-config.yaml
2019-12-19 08:31:15 +01:00
Gary Verhaegen
adceb3a6b2
checkout current sha after daily docs (#3559)
Currently if the docs script fails, the Slack message we get mentions the commit title of the docs version that failed to build, which is not super useful. This ensures we get back to the current commit regardless of what happens with the Haskell script.
2019-11-21 15:00:19 +01:00
Andreas Herrmann
33e47828e3
Bazel 1.1 (#3249)
* bazel: 0.28.1 --> 1.1.0

* bazel-watcher sha256

* Fix missing line in patch

* proto_source_root --> strip_import_prefix

See https://github.com/bazelbuild/bazel/issues/7153 for details.

* Update rules_nixpkgs

Required to avoid errors of the form
```
ERROR: An error occurred during the fetch of repository 'node_nix':
   parameter 'sep' may not be specified by name, for call to method split(sep, maxsplit = None) of 'string'
```

and
```
ERROR: An error occurred during the fetch of repository 'node_nix':
   Traceback (most recent call last):
	File "/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 149
		_execute_or_fail(repository_ctx, <3 more arguments>)
	File "/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 318, in _execute_or_fail
		fail(<1 more arguments>)

Cannot build Nix attribute 'nodejs'.
Command: [/Users/runner/.nix-profile/bin/nix-build, /private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/node_nix/nix/bazel.nix, "-A", "nodejs", "--out-link", "bazel-support/nix-out-link", "-I", "nixpkgs=/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/nixpkgs/nixpkgs"]
Return code: 1
Error output:
src/main/tools/process-tools.cc:173: "setitimer": Invalid argument
```

* Update rules_scala

* .proto has been removed, use [ProtoInfo] instead

See
https://docs.bazel.build/versions/1.1.0/be/protocol-buffer.html#proto_library

* python3_nix add nix_file attribute

To avoid the following error

```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/BUILD:66:1: //:nix_python3_runtime depends on @python3_nix//:bin/python in repository @python3_nix which failed to fetch. no such package '@python3_nix//': Traceback (most recent call last):
        File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 149
                _execute_or_fail(repository_ctx, <3 more arguments>)
        File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 318, in _execute_or_fail
                fail(<1 more arguments>)

Cannot build Nix attribute 'python3'.
Command: [/home/aj/.nix-profile/bin/nix-build, "-E", "import <nixpkgs> { config = {}; overlays = []; }", "-A", "python3", "--out-link", "bazel-support/nix-out-link", "-I", "nixpkgs=/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/nixpkgs/nixpkgs"]
Return code: 1
Error output:
error: anonymous function at /home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/nixpkgs/nixpkgs.nix:3:1 called with unexpected argument 'config', at (string):1:1
```

* rules_haskell unnamed string.split(_, maxsplit = _)

The keyword argument may no longer be named.

* string.replace(_, _, maxsplit = _) may not be named

* Move proto sources from deps to data

Fixes

```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/daml-lf/archive/BUILD.bazel:150:1: in deps attribute of scala_test rule //daml-lf/archive:daml_lf_archive_reader_tests_test_suite_src_test_scala_com_digitalasset_daml_lf_archive_DecodeV1Spec.scala: '//daml-lf/archive:daml_lf_1.6_archive_proto_srcs' does not have mandatory providers: 'JavaInfo'. Since this rule was created by the macro 'da_scala_test_suite', the error might have been caused by the macro implementation
```

* Define sha256 for haskell_ghc__paths

Bazel 1.1.0 fails on missing hashes.

* Disable --incompatible_windows_native_test_wrapper

* //compiler/daml-extension don't modify sources

Modifying sources in-place can cause issues on Windows, where build
actions are not sandboxed and changes on sources can affect other build
steps.

* bazel-genfiles --> bazel-bin

The bazel-genfiles symlink has been removed since Bazel 1.0.
See https://github.com/bazelbuild/bazel/issues/8651

* Mark dev_env_tool repository rule as configure

See
https://docs.bazel.build/versions/1.1.0/skylark/lib/globals.html#repository_rule

* Move data deps into data attribute

* Mark dev_env_tool as local = True

* Manually fetch @makensis_dev_env
2019-11-11 10:06:03 +01:00
Gary Verhaegen
c1662527f5
docs cron: fix content-type header (#3289) 2019-10-30 14:46:57 +01:00
Gary Verhaegen
c4aa296a5e add debug prints for docs cron (#3281) 2019-10-30 10:23:12 +00:00
Gary Verhaegen
536188abce fix spurious HubSpot announce bug (#3266) 2019-10-29 13:26:08 +00:00
Gary Verhaegen
1e1e08d3c9
rewrite docs cron in Haskell (#3235) 2019-10-28 18:26:06 +00:00
associahedron
b7e6088bcd
Delete temp directory in get-daml.sh (#3233)
* Delete temp directory after get-daml.sh

* Revert dockerfile change from #3224
2019-10-18 17:02:02 +01:00
Martin Huschenbett
dcf1ab9e5a Shrink the docker image for the SDK by 57% (#3224)
* Shrink the docker image for the SDK by 57%

Wiping out the `/tmp` dir after installing the SDK does wonders.

@associahedron I wonder if we should do this in the assistant?

* Update release notes
2019-10-18 12:45:09 +00:00
Moritz Kiefer
25f33c2928
Add a daml user to docker image and fix PATH (#3160)
Previously, we were installing the SDK as root which is probably not a
good idea. This PR adds a new `daml` user and fixes PATH (`$HOME` and
`~` both don’t work in this context).
2019-10-11 13:03:26 +02:00
Gary Verhaegen
99ea93168d
update copyright notices (#2499) 2019-08-13 17:23:03 +01:00
Gary Verhaegen
6e841952c0
record build times on macos (#2379) 2019-08-02 14:34:12 +01:00
Gary Verhaegen
4fadebc4ab
cache nix on macos (#2344) 2019-07-31 10:23:43 +01:00
Gary Verhaegen
f23fe7cd98 record build times & statuses (#2016) 2019-07-05 11:35:30 +00:00
Gary Verhaegen
03cbb2efe9
extract slack fail message (#2009) 2019-07-04 15:22:22 +01:00
Moritz Kiefer
a5fdc51662 Restart on nix compression errors (#1986)
We have seen this error on CI at least 2 times so lets restart
automatically when we encounter it.
2019-07-03 12:00:43 +00:00
Moritz Kiefer
61eace44fe
Switch to nixpkgs-unstable (#1921)
The main reason for doing this is that it saves us the trouble of
having to backport changes, e.g., a newer MacOS toolchain.
2019-06-27 18:47:42 +02:00
Gary Verhaegen
ad84566803 rebuild images if Dockerfile changes (#1850) 2019-06-24 16:37:57 +00:00