daml/infra/macos/1-create-box/README.md

Note on major macOS versions

The instructions below have been tested on a macOS Catalina host to create a macOS Catalina guest. While I believe it is possible to run different combinations of the guest/host macOS versions, the documentation for the macinbox project is pretty clear that it is not possible to create Catalina images from earlier OSes, nor earlier OS images from a Catalina host.

As I only have access to Catalina host systems, I have not been able to test older versions.

Machine Setup

You should consider the following changes to the MacOS device:

• Disable Energy Saving settings
• Ensure timezone and time sync are set correctly
• Depending on requirements to manage boxes, enable Screen Share and Remote Access

Installing tools

macOS Installer App

First thing to do is start the download of the macOS installer app from the App Store. This will take a while as the installer is rather large, but you can do most of the other steps below while the download is going on.

I have tested these steps on macOS Catalina 10.15.3 with a Catalina installer created on 2020-01-23 (15.1.00).

Homebrew

Homebrew is not required for these instructions, but it can make some of the following steps easier to automate. Given that, on the one hand, some steps have to be done manually through the UI anyway, and, on the other hand, this process really only needs to be done once, on one machine, there is little value in installing Homebrew for this process. However, in case you already have Homebrew installed, I do mention the corresponding commands when they exist.

Note that if you have Homebrew installed, you also already have the CLI tools and can skip the next step.

XCode CLI Tools

Within a graphical environment, run:

xcode-select --install

This will open a graphical dialog to install the XCode Command-Line Tools.

Vagrant

To easily manage the guest VM from scripts, we will be using Vagrant.

Alternatively, installing with Homebrew:

brew cask install vagrant

These instructions have been tested with 2.2.9.

Hypervisor Selection

We provide two options for the hypervisor: VirtualBox (open-source) and VMWare Fusion (commercial license). This resulted from our testing on large Mac Mini nodes (6 Core, 64Gb, 500Gb drives) where we found VirtualBox to be less stable when attempting to use > 6 virtual cores or 32Gb or more of memory. The Guest OS would hang on boot, or experience slow processing or network to the extent that it became unusable.

VMWare Fusion

Purchase a license for

• VMware Fusion Pro 11.5.3
• Vagrant VMware Desktop Provider 2.0.3

Download the installer packages for this software. Install VMWare Fusion per vendor instructions and accept the security settings in Catalina.

In a Terminal, ensure you have vagrant 2.2.9 and upgrade if necessary or you will receive error from vagrant plugin on VM creation.

vagrant --version
brew cask install vagrant

You will need to install the vagrant plugin and license

vagrant plugin install vagrant-vmware-desktop
vagrant plugin license vagrant-vmware-desktop ~/license.lic

To verify the license installation, run:

vagrant plugin list

VirtualBox

Download and install VirtualBox, including the extension pack.

Alternatively, it can be installed from Homebrew using:

brew cask install virtualbox

This will require sudo access.

The extension pack has to be manually added after the installation of VirtualBox; this can be done through the UI, or running:

V=$(VBoxManage --version | sed 's/r.*//')
sudo VBoxManage extpack install /path/to/download/Oracle_VM_VirtualBox_Extension_Pack-$V.vbox-extpack

These instructions have been tested against 6.1.4 and 6.1.6. The extension version must match the version of VirtualBox itself. As explained above, VirtualBox does not seem to fully support our production configuration, though it may still be useful for local testing.

rbenv

The underlying scripts to create the "blank" macOS guest machine are written in Ruby, so we need to install some version of Ruby. Ruby does come with macOS, and has installers in Homebrew, but for the sake of stability we are going to go with rbenv, so we can pin down the exact Ruby version we use. This also has the advantage of not needing root access to install the macinbox gem. (Though sudo still needs to be used to run it.)

On the host machine, for the user that is going to manage the guest VM, add the following to the relevant shell init file (~/.zshrc by default):

if [ -d $HOME/.rbenv/shims ]; then
    export PATH="$HOME/.rbenv/shims:$PATH"
fi
if [ -d $HOME/.rbenv/bin ]; then
    export PATH="$HOME/.rbenv/bin:$PATH"
fi

then run the following:

git clone https://github.com/rbenv/rbenv.git ~/.rbenv
cd ~/.rbenv
git checkout c6324ff45af33a194f5658a1c6322a94da145f98

Note: this is the commit I tested these instructions with; I have no reason to believe the master branch is bad.

In a new terminal (to pick up the new PATH), run:

mkdir -p "$(rbenv root)"/plugins
git clone https://github.com/rbenv/ruby-build.git "$(rbenv root)"/plugins/ruby-build
cd "$(rbenv root)/plugins/ruby-build"
git checkout 0ef5e055659230e2fb2ae9b0928f70dc27c1c136

where, again, this is the commit I tested with, so I am providing it for maximum reproducibility, but I have no reason to mistrust the master branch.

Finally, run:

rbenv install 2.7.0
rbenv global 2.7.0

which installs Ruby 2.7.0 for the current user.

macinbox

From this directory, run:

gem install macinbox -v 4.0.0

The version I tested with is 4.0.0.

Creating base image

After all the tools are installed (and the macOS installer has finished downloading), you can create the base image ("Vagrant box") using the following command:

VMWare Fusion Variant

sudo macinbox --box-format vmware_desktop --disk 250 --memory 57344 --cpu 10 --user-script user-script.sh

VirtualBox variant

NOTE: Limited use of hardware due to possible bugs in VirtualBox.

sudo macinbox --box-format virtualbox --disk 50 --memory 4096 --cpu 1 --user-script user-script.sh

The disk size given here (in GB) will be the disk size used by the individual VMs created based on this box; memory and cpu parameters are default values that can be overridden in the Vagrantfile.

200GB disk size is the value we arrived at for Windows and Linux nodes by progressively incrementing the size each time we had a "disk is full" error.

32GB of RAM and 4 CPU cores are the values I arrived at by running our existing build against various virtual instance types on Linux and Windows (see #4520).

250Gb disk, 56Gb memory and 10 virtual cores was arrived at from testing on Mac Mini 2018 nodes with 6 core processors.

The provided "user script", which can be inspected in the current directory, adds a synthetic.conf file as part of the base macOS image we are creating, such that new VMs will start up with an available mount point on /nix.

Sharing the base box

If you want to share this base box (to avoid having to run the above on every macOS host), you can copy over the ~/.vagrant.d/boxes/macinbox directory to new machines.