Commit Graph

193 Commits

Author SHA1 Message Date
Edward Amsden
d3c5aa74c1
build: A 'smart' docker image
This commit adds the `docker-image` attribute to the main Nix
entrypoint, invoking `nix/pkgs/docker-image` which will build
a 'smart' docker image that can load keyfiles or a pier and
boot a ship

It includes a README for the official docker image, suitable
for posting as the README to a Docker Hub or similar docker
image repository.
2021-01-13 07:37:28 -05:00
Joe Bryan
3e4d495500 nix: remove lib/base64 from solid pill staging 2020-12-08 16:22:58 -08:00
Joe Bryan
5980fbedd0 ci: updates herb call to -test, matching new invocation 2020-12-04 21:46:05 -08:00
Joe Bryan
548440a85d nix: trims trailing whitespace from solid pill builder 2020-11-20 15:23:56 -08:00
Joe Bryan
b1e3b0def2 hoon: moves $xray into userspace 2020-11-20 15:23:56 -08:00
Joe Bryan
aea0f571d7 hoon: moves $plum and +plume to userspace 2020-11-20 15:23:56 -08:00
Brendan Hay
2e72724d44
build: github actions workflow configuration 2020-11-15 15:05:13 +01:00
Brendan Hay
12e1a53633
build: silence service account activation output 2020-11-06 12:00:23 +01:00
Brendan Hay
b7d417a4b2
build: minor refactoring of haskell-nix overlays 2020-11-06 10:22:34 +01:00
Brendan Hay
983261a3d8
build: move darwin install_name_tool fixup from vere to king haskell
As well as some variable naming consistency to appease the nixpkgs
gremlins and keep the git history lords on their toes.
2020-11-06 09:47:11 +01:00
Brendan Hay
767a6f85a4 build: remove {sha256,md5} output for push-storage-object effects 2020-11-05 11:39:05 -08:00
Brendan Hay
e9cf0a3518 build: add log message when destination object already exists 2020-11-05 11:39:04 -08:00
Brendan Hay
244a9b6a06 build: force google-cloud-sdk to use python3
This prevents an error where the python2 version of the SDK tries to
use the monotonic library which is unable to detect the host operating
system inside the sandbox - exploding in typical python fashion.

By using python3 it obviates the need for the monotonic dependency by
using the builtin time.monotonic call.
2020-11-05 11:39:01 -08:00
Brendan Hay
b1790b1b3a build: adding support for hercules ci effects 2020-11-05 11:38:58 -08:00
Brendan Hay
3a0b975f4d
build: expose configurable arguments when booting/testing fake ships
This also sets -l (lite boot) by default for boot-fake-ship to speed
up the initial pier creation used by pills/tests, and uses
urbit.meta.arguments by default for tests - seems a little less likely
to footgun.
2020-11-05 10:25:31 +01:00
Brendan Hay
644acd5002
build: ensure urbit tests are run with the -g argument 2020-11-05 10:25:31 +01:00
Brendan Hay
dd782e062e
vere: ensure debug symbols aren't stripped by default (by nix) 2020-11-05 10:25:30 +01:00
Brendan Hay
8d0a65bb6e
vere: remove dependency on nix-store path for static darwin binary
Poached from @eglaysher's work in #3380.
2020-11-04 20:50:45 +01:00
Brendan Hay
9e1f75da64
build: simplify build tool selection for shell.nix 2020-11-04 15:34:03 +01:00
Brendan Hay
24c63cb078
build: reformatting and triggering tests 2020-11-04 12:55:27 +01:00
Brendan Hay
684005e200
build: disable full test-suite on darwin
Linux remains enabled. This prevents CI errors while figuring out
darwinAllowLocalNetworking and sandboxing.
2020-11-04 12:45:40 +01:00
Brendan Hay
00d236501e
build: fix bind errors in darwin sandboxes
See 5f6840fbb4/src/libstore/build.cc (L2961)
2020-11-04 11:48:46 +01:00
Brendan Hay
cf5c2a40ab
build: remove unnecessary test platform dependence
An alternative approach is being considerd for more granular platform
dependence for tests.
2020-11-04 11:11:39 +01:00
Joe Bryan
52308c4977 test: cleans up output, adds |trim and |meld
with intervening |mass measurements
2020-11-03 01:18:00 -08:00
Brendan Hay
58094c265c
build: bludgeon write/EAGAIN stdout errors into submission 2020-10-30 12:00:56 +01:00
Brendan Hay
6ed18bb825
build: removing shell set options from inline buildPhase for tests 2020-10-29 17:43:23 +01:00
Brendan Hay
6599a85b99
king: ensure tests are enabled on ci and don't depend on lfs pills 2020-10-29 16:20:41 +01:00
Brendan Hay
50f66ce431
build: simplify make test and re-enable on darwin ci for testing 2020-10-29 13:17:05 +01:00
Brendan Hay
fada33dfe6
build: remove checkMaterialization flag 2020-10-28 18:19:53 +01:00
Brendan Hay
59709752eb
build: simplify pushed artefact naming and set object content-type 2020-10-28 12:13:05 +01:00
Brendan Hay
689c0eb6a0
build: ensuring release tarball contains singular parent directory
This ensures parity with the current release tarballs.
2020-10-28 11:05:58 +01:00
Brendan Hay
ce3cbf0857
build: rework all nix expressions to support hercules-ci builds
This also removes nixcrpkgs and OSX cross compilation in favour of
compiling on the target. x86_64/musl targets are still supported
on Linux.

All sources are now managed via niv (see nix/sources.json) and Haskell
package sets are provided/organised via IOHK's haskell.nix.

Some effort has been made to expose similar top-level attributes for
development, but in some cases there have been changes. Please see
the comments in the top-level default.nix and ci.nix files for usage.
2020-10-27 13:55:49 +01:00
Paul Driver
7268fdc594 libaes_siv now using tip of dfoxfranke master 2020-10-16 11:33:50 -07:00
Paul Driver
227f27d620 remove scrypt from urbit build (in urcrypt now) 2020-10-10 18:04:26 -07:00
Paul Driver
988d1fe445 pkg-config support for urcrypt, update urbit build 2020-10-09 13:40:31 -07:00
Paul Driver
88c68e9498 more flags fixing, remove some obsolete deps 2020-10-05 08:36:45 -07:00
Paul Driver
8e596ca365 more autoconfiscation 2020-10-03 17:01:56 -07:00
Paul Driver
5fbd0b2a90 mid-autoconfiscation of urcrypt 2020-10-03 15:15:14 -07:00
Paul Driver
7bca3a86cf obliterate ge-additions package, move to urcrypt 2020-10-02 09:08:56 -07:00
Paul Driver
58e4915a8d stops trying to hide pointer shapes, cleans up secp api, updates secp dependency 2020-09-29 11:39:43 -07:00
Paul Driver
f5449204e1 secp_make->urcrypt 2020-08-29 11:55:58 -07:00
Paul Driver
8d657e12c1 aes-siva-en->urcrypt 2020-08-24 14:38:26 -07:00
Paul Driver
6fdc65dea6 argon2 2020-08-10 15:19:32 -07:00
Paul Driver
cd4b8cb9b0 aes_ecb -> urcrypt 2020-08-05 16:06:04 -07:00
Paul Driver
74d06deb1b one working function in so 2020-07-30 13:07:13 -07:00
botter-nidnul
790e2dd5ec libsigsegv: disable stack vma check on arm
Adds the equivalent arm patches to the i386 patches added in #3054

These result in a dramatic speedup in running  `=a (bex 1.000.000.000)` just as they do on x86_x64

This also sneaks in a hack to `/nix/nixcrpkgs/pkgs/libsigsegv/builder.sh` that allows libsigsegv to configure itself properly when cross compiling release binaries from x86_64 to aarch64.

It won’t trigger for you, since you’re not doing that (yet) but it will make it a little easier for me to maintain my aarch64 static release binaries if that `if [ $host = aarch64-linux-musleabi ]` section is upstreamed.
2020-07-27 23:33:14 -05:00
Paul Driver
b8db4141bc urcrypt pkg with one function, building 2020-07-27 19:19:17 -07:00
Philip Monk
e8a307ae67
noun: first pass at memory compaction 2020-07-06 21:35:41 -07:00
Fang
a53e462be8
Merge branch 'master' into ipc-redux 2020-07-03 14:58:49 +02:00
Philip Monk
0574546b77
Merge branch 'ford-fusion' (#3060)
* ford-fusion: (259 commits)
  clay: remove scaffolding
  kiln: make otas continue even if they failed to apply
  metadata-store: add cleanup utility, use on-poke:def instead of no-op
  clay: flop syntax error trace
  landscape/img/codeeval.png: typo in filename
  invite-view: reinstate as potato
  goad: don't crash on pre-OTA sign
  chat-store: responded to comments, cleaned up
  -test: support other desks (full beams in args)
  arvo: use date instead of kelvin
  hoon: re-fix +slab; /tests: fix clay tests
  kiln: don't implicitly create syncs
  gall: don't make large stack trace
  ames: use +cut in +encrypt
  pill: solid
  dojo: print generator type errors
  Squashed commit of the following:
  ci: fix herb tests; update pills
  ford,kiln: nicer errors
  dojo: too many nouns
  ...

Signed-off-by: Philip Monk <phil@pcmonk.me>
2020-06-29 16:07:38 -07:00
Ted Blackman
d7b9889b4a -test: support other desks (full beams in args) 2020-06-28 05:06:16 -04:00
Joe Bryan
14faa22e0f nix: add explicit shutdown to solid-pill derivation 2020-06-26 22:34:20 -07:00
Joe Bryan
6fa194d527 Merge branch 'master' into jb/ipc-redux
* master: (147 commits)
  vere: bump version to 0.10.7
  libsigsegv: disable stack vma check
  vere: bump version to 0.10.6
  ci: add travis as trusted user
  jets: use appropriate macro
  noun: add -C to control memo cache size
  jets: restore fond/play/peek hooks
  jam: add commented-out functionality to count size of atom
  jets: cap memo cache and remove peek, play, and fond jets
  noun: add functions to count size of noun
  release: urbit-os-v1.0.23
  interface/config: fix production build
  soto: run +on-load migration once
  publish, links: restore full height
  sh/build-interface: amend for SPA
  interface/CONTRIBUTING: amend for SPA / webpack
  solid: update pill
  hood + apps: fix OTA process for feat/SPA
  hood: add version %6 for %file-server upgrade
  chat: equally size both code + s3 buttons
  ...
2020-06-26 13:33:00 -07:00
Philip Monk
2713083386
libsigsegv: disable stack vma check
This patches libsigsegv to not check the stack vma on Linux, since that
involves reading procfs, and we make very heavy use of sigsegv.  This
eliminates most of urbit's performance discrepancy between Linux and
MacOS.  These are the benchmarks used; note this is a local MBP vs a
cloud Linux server, and the MBP is almost certainly faster hardware.

We take two benchmarks, one of which decrements 10 million times and the
other simply allocates 125MB of memory.  These are the results:

cpu-heavy ==  =/ n 10.000.000 |-(?~(n n $(n (dec n))))
mem-heavy == =a (bex 1.000.000.008)

macos, cpu-heavy: 6 seconds
macos, mem-heavy: 1 second

linux-before, cpu-heavy: 30 seconds
linux-before, mem-heavy: 160 seconds

linux-after, cpu-heavy 9 seconds
linux-after, mem-heavy 1.3 seconds

This represents a 3x speedup for the cpu-heavy operation and a 120x
speedup for the memory-heavy operation.

This check was used to try to distinguish stack overflow from other
forms of segmentation fault.  In the comments in src/handler-unix.c, it
describes three heuristics it uses, depending on what's available from
the OS.  In the linux-i386 case, all three are availble, so we simply
disable the slow one.  This correctly recognizes stack overflow if you
simply alloca(10000000000).
2020-06-26 11:15:01 -07:00
Ted Blackman
82cfc691ce ci: fix herb tests; update pills 2020-06-25 02:14:16 -04:00
Joe Bryan
dcdd1e4be9 vere: removes now-unused sniproxy dependency 2020-06-23 16:04:40 -07:00
Joe Bryan
829d402dc9 Revert "TMP kill -9 urbit in test"
This reverts commit 6e98bdd3d3.
2020-06-22 14:11:38 -07:00
Joe Bryan
6e4c8361dd build: remove duplicate zlib dependency declaration 2020-06-11 20:31:26 -07:00
Joe Bryan
8fac63661f vere: removes ncurses dependency 2020-06-11 20:25:46 -07:00
Joe Bryan
c3557e0836 Merge branch 'master' into ipc-redux-kh
* master: (25 commits)
  travis: Fix resource exhaustion error in test build.
  travis: Separate job for Haskell on Linux + bump nix version
  gall: properly handle empty outstanding ack queue
  vere: bump version to 0.10.5
  dawn: switch default eth node
  jets: patch cleanup
  Update os1-bug-report.md
  jets: switch openssl to the loom allocator.
  serf: disable automatic |pack
  release: urbit-os-v1.0.20
  behn: fix +unset-timer
  jael: send breach notifications in order
  jael: don't notify breach on initial update to ship
  jael: look for correct wire
  vere: handle failed cue of private key file
  u3: switches GMP import assert to conditional
  u3: fixes rock:load size printf
  vere: bypass terminal logging on lmdb worker thread
  vere: prevents term.c from retrying tcsetattr() indefinitely
  jets: jet decryption of aes-siv mode.
  ...
2020-06-11 19:07:13 -07:00
Benjamin Summers
2501cb0e17 travis: Fix resource exhaustion error in test build. 2020-06-11 14:54:34 -07:00
~siprel
382ace5f23 travis: Various fixes and improvements.
1. Fix bug in test builds that was causing failures for some reason I
   still don't understand.

  `tee` output to stderr was failing with "resource temporarily
  unavailable". The hack that fixed it was to simply write the herb
  output to a file and dump it to stdout after it fully completes.

2. sh/cachix works without CACHIX keys. Simply doesn't upload in that case.

3. Write code to cache testbus builds (disabled for now)

4. fakeship builds get further in bootstrap sequence before committing.

  This fixes problems with PRs from forked repos and enables the scripts
  to be run locally.
2020-06-11 20:34:50 +00:00
~siprel
f853557472 travis: Cache test results in cachix. 2020-06-11 01:00:26 +00:00
~siprel
26bd5a9e4b king: Got new IPC working, still needs lots of polish. 2020-05-26 23:01:03 +00:00
Elliot Glaysher
cc3a139440 aes_siv: vendors libaes_siv since it isn't in nix.
This vendors libaes_siv so that we can use it to jet
+sivc:aes:crypto.
2020-05-20 13:51:59 -07:00
Joe Bryan
6e98bdd3d3 TMP kill -9 urbit in test 2020-05-01 00:23:59 -07:00
Jared Tobin
c078c16eba
build: remove urbit-large-log-limit derivation
Made unnecessary by 578593d45.
2020-03-01 23:04:22 +04:00
Jared Tobin
ce3c1a0e21
build: better vere label patch
Moves the secondary URBIT_VERSION assignment to a different hunk to
avoid undesirable patching conflicts.
2020-02-24 15:15:56 +04:00
Jared Tobin
05a0a9092a
build: add configurePhase for urbit derivation
46a438f introduced use of the stdenv generic builder instead of an
explicit builder.sh script.  This was reverted in 1daaf041a as it proved
to cause problems when building on some Linux variants.

This fixes the problem encountered on Linux by calling the standard
configure script with bash in an explicit configurePhase.
2020-02-24 14:40:10 +04:00
Jared Tobin
f66d750e1c
build: label large-log-limit vere version
Ensures that large log limit Vere builds are labelled as such, and are
thus easily identifiable.
2020-02-24 14:40:10 +04:00
Jared Tobin
813261dc31
Revert "Revert "Merge branch 'jt/urbit-large-log' (#2211)""
This reverts a revert, commit 1daaf041a7,
in order to fix the problems that led to it being reverted.
2020-02-24 14:37:41 +04:00
Jared Tobin
1daaf041a7
Revert "Merge branch 'jt/urbit-large-log' (#2211)"
This reverts commit 0fc9056074, reversing
changes made to 1e3a1434e0.

The changes made here evidently broke builds on some Linux variants.
2020-01-31 15:26:15 +04:00
Jared Tobin
a6e15c9418
build: add urbit-large-log derivation
Adds a derivation that uses a patched pkg/urbit/vere/lmdb.c that employs
a sixty gigabyte log limit, instead of the default forty.  This is
useful for ships e.g. ~zod that receive a lot of traffic on mainnet.
2020-01-30 13:57:03 +04:00
Jared Tobin
46a438f26e
build: use generic builder in urbit derivation
'installPhase' captures all the important stuff that builder.sh
previously did, and use of the generic builder allows stages of the
build to be more easily overridden.
2020-01-30 11:20:40 +04:00
Brendan Hay
122e01b13d
build: allow baking multiple pills into docker images
By baking (potentially) multiple pills into an image, we can provide
mainnet vs ropsten images. It is still up to the operator to pass
along the pill path(s) to the entrypoint. For example, using Docker:

docker run --tty urbit -B /share/brass.pill -J /share/ivory.pill ...

The main .image attribute still uses only the solid pill, and image-ropsten
has been provided with brass and ivory pills.

Additionally herb has been added to the image for convenience.
2019-12-18 12:02:18 +01:00
Brendan Hay
249c02fe9f
build: trimming unnecessary image configuration 2019-12-16 09:20:28 +01:00
Jared Tobin
9d02b5e88e
Merge branch 'bh/cross-debug' (#2030)
* bh/cross-debug:
  build: add support for releasing cross-compiled debug binaries

Signed-off-by: Jared Tobin <jared@tlon.io>
2019-12-06 21:19:20 +08:00
Joe Bryan
68a94f420f nix: actually enables debug symbols in release builds 2019-12-05 17:17:33 -08:00
Jared Tobin
2047d6a676
Merge branch 'debug-release' (#2029)
* debug-release:
  nix: enables debug symbols in release builds

Signed-off-by: Jared Tobin <jared@tlon.io>
2019-12-05 16:49:12 +08:00
Brendan Hay
5339a9a826
build: add support for releasing cross-compiled debug binaries
sh/cross can now be invoked as follows:

sh/cross urbit linux64
sh/cross urbit darwin

sh/cross urbit-debug linux64
sh/cross urbit-debug darwin
2019-12-05 09:38:09 +01:00
Joe Bryan
80628e0cac nix: enables debug symbols in release builds 2019-12-04 22:59:53 -08:00
Joe Bryan
13f61ad373 nix: removes unnecessary use of cat in ivory-header build 2019-12-04 22:57:07 -08:00
Philip Monk
0431c3c073
Merge remote-tracking branch 'origin/jam-cue-rock' into rc 2019-12-02 02:08:37 -08:00
Jared Tobin
25fbb5c54a
arvo-ropsten: revert ALEF to AMES [ci skip]
f035955a renamed %alef to %ames, so the Ropsten Arvo derivation should
be updated accordingly.
2019-12-02 16:04:06 +08:00
Philip Monk
4d1457bbaa
Merge remote-tracking branch 'origin/master' into philip/mall-real 2019-11-24 00:01:04 -08:00
Joe Bryan
9c82eadd11 ci: runs |pack after tests 2019-11-22 17:25:18 -08:00
Philip Monk
a5412f01de
Merge branch 'alef-testnet-merge' into philip/mall-real 2019-11-19 13:03:07 -08:00
Jared Tobin
c9c2e33f63 zuse, build: remove default testnet config
Returns the target %zuse contract configuration to mainnet, and also
tweaks the 'arvo-ropsten' build to use %alef instead of %ames.

Also fixes a merge conflict artifact in nix/ops/default.nix.
2019-11-15 19:26:43 +09:00
Brendan Hay
ee856536b4
build: remove unused let binding from image expression 2019-11-12 08:45:02 +01:00
Jared Tobin
999bc1704e
build: give arvo a high priority
0bdced981e introduced the 'arvo-ropsten' derivation.  Attempting to
install both 'arvo' and 'arvo-ropsten' via nix-env will result in a
priority error; this assigns a higher priority to 'arvo' to resolve the
conflict.

Fixes #1912.
2019-11-05 07:10:04 +08:00
Ted Blackman
9a02c64b45 Merge branch 'master' into alef-testnet-merge
Untested, has not been run.
Uses master pills for now.  Will need refreshing.
2019-11-02 23:37:01 -04:00
Joe Bryan
46934cef17 nix: fix ivory-header error conditions (detecting lfs pointers) 2019-11-01 14:47:57 -07:00
Jared Tobin
0bdced981e
build: add Ropsten derivations for arvo and pills
Arvo testnets typically use Ropsten's deploy of the Azimuth contract
instead of mainnet's.  This commit adds an 'arvo-ropsten' derivation for
Arvo as specialised to Ropsten.

In addition to using a different Azimuth address in %zuse, this also
configures %ames and :acme appropriately for running an alternate
network on Ropsten.  %ames has its protocol version incremented, and
:acme uses the Let's Encrypt staging API.

Included here are derivations for creating brass and ivory pills using
arvo-ropsten, to boot.
2019-10-24 10:03:52 +08:00
Jared Tobin
d8f5836876
Merge branch 'image-build' (#1856)
* image-build:
  build: simplify image build interface

Signed-off-by: Jared Tobin <jared@tlon.io>
2019-10-22 10:25:55 +08:00
Brendan Hay
4813d9829b
build: simplify image build interface
This removes the baked in codedump inspection and wrapper scripts
in favour of downstream tooling overriding this as necessary by
using FROM <image> in their respective dockerfile.
2019-10-22 10:18:37 +08:00
Jared Tobin
71b27b0300
Merge branch 'pretty-userspace' (#1847)
* pretty-userspace:
  pills
  hoon: moves new pretty-printer back into userspace

Signed-off-by: Jared Tobin <jared@tlon.io>
2019-10-22 10:12:05 +08:00
Joe Bryan
10645a8d23 build: always include debug symbols 2019-10-17 18:20:23 -07:00
Joe Bryan
fa01bfb0a5 hoon: moves new pretty-printer back into userspace 2019-10-17 17:52:45 -07:00
Jared Tobin
9068188e4b
build: fix 'solid' derivation [ci skip]
578adc93 introduced a 'meta' attribute for handling, amongst other
things, conditional construction of the debug executable.  It missed the
'solid' derivation, causing it to break.
2019-10-15 11:56:16 +04:00
Jared Tobin
8a4a3335d5
build: fix typo [ci skip]
s/ubit/urbit
2019-10-15 11:56:05 +04:00