272f535 · lynxjs-elm/compiler

git.iko.soy/lynxjs-elm/compiler

Free the npm package from third party dependencies

And support Linux ARM and macOS ARM.

commit272f535da29e74b6d2c8b69d88c00e0f92217c24 authorSimon Lydell <simon.lydell@gmail.com> dateWed 9 Nov 2022, 08:30 (4 years ago) parentad9d9d4

18 files changed+282−161

compare with master

unifiedsplit

installers/npm/.gitignore+2

@@ -1 +1,3 @@

11node_modules/

2+packages/*/elm

3+packages/*/elm.exe

installers/npm/PUBLISHING.md+77−4

@@ -2,22 +2,95 @@

33Here's how to update the `npm` installer.

5+## 0. Overview

7+- There is one _main npm package_ called `elm`.

8+- Then there is one _binary npm package_ for each platform, called for example `@evancz/elm_darwin_arm64`.

10+The binary packages declare which OS and CPU they are compatible with. For example:

11+

12+```json

13+ "os": [ "darwin" ],

14+ "cpu": [ "arm64" ]

15+```

16+

17+The main npm package depend on the binary packages via [optional dependencies](https://docs.npmjs.com/cli/v9/configuring-npm/package-json#optionaldependencies):

18+

19+```json

20+ "@evancz/elm_darwin_arm64": "0.19.1-0",

21+ "@evancz/elm_darwin_x64": "0.19.1-0",

22+ "@evancz/elm_linux_arm64": "0.19.1-0",

23+ ...

24+```

25+

26+When installing, `npm` fetches the metadata for all the optional dependencies and only installs the one with a matching OS and CPU. If none of them match, `npm` still considers the install successful. However, the main npm package contains an install script that gives a helpful error.

27+

528

629## 1. GitHub Release

730

831Create a [GitHub Release](https://github.com/elm/compiler/releases) with the following files:

932

10331. `binary-for-mac-64-bit.gz`

11−2. `binary-for-windows-64-bit.gz`

34+2. `binary-for-mac-arm-64-bit.gz`

12353. `binary-for-linux-64-bit.gz`

36+4. `binary-for-linux-arm-64-bit.gz`

37+5. `binary-for-windows-64-bit.gz`

1338

1439Create each of these by running the `elm` executable for each platform through `gzip elm`.

1540

1641

17−## 2. Try a beta release

42+## 2. Put the binaries in place

43+

44+Put the above files at:

45+

46+1. `packages/elm_darwin_arm64/elm`

47+2. `packages/elm_darwin_x64/elm`

48+3. `packages/elm_linux_x64/elm`

49+4. `packages/elm_linux_arm64/elm`

50+5. `packages/elm_win32_x64/elm.exe` (Note the `.exe` file extension!)

51+

52+(They are ignored by git.)

53+

54+

55+## 3. Publish the binary packages

56+

57+Repeat this for all the packages mentioned in the previous section. This uses `packages/elm_darwin_arm64` as an example.

58+

59+1. Go to the folder: `cd packages/elm_darwin_arm64`

60+2. Double-check that you put the right binary in the right package: `file elm`

61+3. Double-check that the file is executable: `ls -l elm`

62+4. In `package.json` of the binary package, bump the version for example to `"0.19.1-2"`.

63+5. In `package.json` of the main npm package, update `"optionalDependencies"` to point to the bumped version. For example: `"@evancz/elm_darwin_arm64": "0.19.1-2"`

64+6. Publish the package: `npm publish --access=public`

65+

66+ `--access=public` is needed because scoped packages are private by default.

67+

68+<details>

69+<summary>Notes about the versions of the binary packages</summary>

70+

71+- End users never have to think about them. They only need to think about the version of the main npm package.

72+

73+- The binary packages can have different versions. One can have `"0.19.1-0"` while another is at `"0.19.1-1"`. This is useful if you mess up publishing one platform: Then you can bump just that one and re-release, instead of having to re-release _all_ platforms.

74+

75+- The version of the main npm package is not related to the versions of the binary packages – they’re all independent. So the main npm package can be at `"0.19.1-6"` while the binary packages have suffixes like `-0`, `-1` and `-9`. (They all share the `0.19.1` prefix though to make things more understandable!)

76+

77+- The main npm package pins the versions of the binary packages _exactly_ – no version ranges.

78+ - This means that installing `elm@0.19.1-6` installs the exact same bytes in two years as today.

79+ - The `package.json` of each binary package says which OS and CPU it works for. `binary.js` in the main npm package has code that deals with OS and CPU too, so the main npm package needs to install binary packages with known OS and CPU declarations.

80+

81+- There is no need to use `beta` suffixes for the binary packages. Just bump the number suffix and point to it in a beta release of the main npm package. As mentioned above:

82+ - Already published versions of the main npm package depend on exact versions of the binary packages, so they won’t accidentally start downloading beta versions.

83+ - End users only see the version of the main npm package.

84+

85+</details>

86+

87+

88+## 4. Try a beta release

1889

1990In `package.json`, bump the version to `"0.19.2-beta"`.

2091

92+Double-check that `"optionalDependencies"` is in sync with the binary packages.

93+

2194```bash

2295npm publish --tag beta

2396```

@@ -34,7 +107,7 @@ The `latest` tag should not be changed, and there should be an additional `beta`

34107Try this on Windows, Linux, and Mac.

35108

36109

37−## 3. Publish final release

110+## 5. Publish final release

38111

39112Remove the `-beta` suffix from the version in `package.json`. Then run:

40113

@@ -43,7 +116,7 @@ npm publish

43116```

44117

45118

46−## 4. Tag the `latest-0.19.1` version

119+## 6. Tag the `latest-0.19.1` version

47120

48121Many compiler releases have needed multiple `npm` publications. Maybe something does not work on Windows or some dependency becomes insecure. Normal `npm` problems.

49122

installers/npm/bin/elm+7−33

@@ -1,54 +1,28 @@

11#!/usr/bin/env node

33var child_process = require('child_process');

4−var path = require('path');

5−var fs = require('fs');

86// Some npm users enable --ignore-scripts (a good security measure) so

97// they do not run the post-install hook and install.js does not run.

108// Instead they will run this script.

119//

12−// On Mac and Linux, we download the elm executable into the exact same

10+// On Mac and Linux, we hard link the elm executable into the exact same

1311// location as this file. Since npm uses symlinks on these platforms,

1412// that means that the first run will invoke this file and subsequent

1513// runs will call the elm binary directly.

1614//

17−// On Windows, we must download a file named elm.exe for it to run properly.

15+// On Windows, our binary file must be named elm.exe for it to run properly.

1816// Instead of symlinks, npm creates two files:

1917//

2018// - node_modules/.bin/elm (a bash file)

2119// - node_modules/.bin/elm.cmd (a batch file)

2220//

2321// Both files specifically invoke `node` to run the file listed at package.bin,

24−// so there is no way around instantiating node for no reason on Windows. So

25−// the existsSync check is needed so that it is not downloaded more than once.

22+// so there is no way around instantiating node for no reason on Windows.

2623

2724

28−// figure out where to put the binary (calls path.resolve() to get path separators right on Windows)

29−//

30−var binaryPath = path.resolve(__dirname, 'elm') + (process.platform === 'win32' ? '.exe' : '');

31−

32−// Run the command directly if possible, otherwise download and then run.

33−// This check is important for Windows where this file will be run all the time.

34−//

35−if (process.platform === 'win32')

36−{

37− fs.existsSync(binaryPath)

38− ? runCommand()

39− : require('../download.js')(runCommand);

40−}

41−else

42−{

43− require('../download.js')(runCommand);

44−}

45−

46−

47−function runCommand()

48−{

49− // Need double quotes and { shell: true } when there are spaces in the path on windows:

50− // https://github.com/nodejs/node/issues/7367#issuecomment-229721296

51− child_process

52− .spawn('"' + binaryPath + '"', process.argv.slice(2), { stdio: 'inherit', shell: true })

53− .on('exit', process.exit);

54−}

25+var binaryPath = require('../binary.js')();

26+child_process

27+ .spawn(binaryPath, process.argv.slice(2), { stdio: 'inherit' })

28+ .on('exit', process.exit);

installers/npm/binary.jsnew file+126

@@ -0,0 +1,126 @@

1+var fs = require('fs');

2+var package = require('./package.json');

3+var path = require('path');

7+// MAIN

8+//

9+// This function is used by install.js and by the bin/elm backup that gets

10+// called when --ignore-scripts is enabled.

11+

12+

13+module.exports = function()

14+{

15+ // figure out package of binary

16+ var version = package.version.replace(/^(\d+\.\d+\.\d+).*$/, '$1'); // turn '1.2.3-alpha' into '1.2.3'

17+ var subPackageName = '@evancz/elm_' + process.platform + '_' + process.arch;

18+

19+ verifyPlatform(version, subPackageName);

20+

21+ var fileName = process.platform === 'win32' ? 'elm.exe' : 'elm';

22+

23+ try

24+ {

25+ var subBinaryPath = require.resolve(subPackageName + '/' + fileName);

26+ }

27+ catch (error)

28+ {

29+ if (error && error.code === 'MODULE_NOT_FOUND')

30+ {

31+ exitFailure(version, missingSubPackageHelp(subPackageName));

32+ }

33+ else

34+ {

35+ exitFailure(version, 'I had trouble requiring the binary package for your platform (' + subPackageName + '):\n\n' + error);

36+ }

37+ }

38+

39+ // Yarn 2 and later ("Berry") always invokes `node` (regardless of configuration)

40+ // so we cannot do any optimizations there

41+ var isYarnBerry = /\byarn\/(?!1\.)/.test(process.env.npm_config_user_agent || "");

42+

43+ // as mentioned in bin/elm we cannot do any optimizations on Windows

44+ if (process.platform === 'win32' || isYarnBerry)

45+ {

46+ return subBinaryPath;

47+ }

48+

49+ // figure out where to put the binary

50+ var binaryPath = path.resolve(__dirname, package.bin.elm);

51+ var tmpPath = binaryPath + '.tmp';

52+

53+ // optimize by replacing the JS bin/elm with the native binary directly

54+ try

55+ {

56+ // atomically replace the file with a hard link to the binary

57+ fs.linkSync(subBinaryPath, tmpPath);

58+ fs.renameSync(tmpPath, binaryPath);

59+ }

60+ catch (error)

61+ {

62+ exitFailure(version, 'I had some trouble writing file to disk. It is saying:\n\n' + error);

63+ }

64+

65+ return binaryPath;

66+}

67+

68+

69+

70+// VERIFY PLATFORM

71+

72+

73+function verifyPlatform(version, subPackageName)

74+{

75+ if (subPackageName in package.optionalDependencies) return;

76+

77+ var situation = process.platform + '_' + process.arch;

78+ console.error(

79+ '-- ERROR -----------------------------------------------------------------------\n\n'

80+ + 'I am detecting that your computer (' + situation + ') may not be compatible with any\n'

81+ + 'of the official pre-built binaries.\n\n'

82+ + 'I recommend against using the npm installer for your situation. Check out the\n'

83+ + 'alternative installers at https://github.com/elm/compiler/releases/tag/' + version + '\n'

84+ + 'to see if there is something that will work better for you.\n\n'

85+ + 'From there I recommend asking for guidance on Slack or Discourse to find someone\n'

86+ + 'who can help with your specific situation.\n\n'

87+ + '--------------------------------------------------------------------------------\n'

88+ );

89+ process.exit(1);

90+}

91+

92+

93+

94+// EXIT FAILURE

95+

96+

97+function exitFailure(version, message)

98+{

99+ console.error(

100+ '-- ERROR -----------------------------------------------------------------------\n\n'

101+ + message

102+ + '\n\nNOTE: You can avoid npm entirely by downloading directly from:\n'

103+ + 'https://github.com/elm/compiler/releases/tag/' + version + '\n'

104+ + 'All this package does is distribute a file from there.\n\n'

105+ + '--------------------------------------------------------------------------------\n'

106+ );

107+ process.exit(1);

108+}

109+

110+

111+

112+// MISSING SUB PACKAGE HELP

113+

114+

115+function missingSubPackageHelp(subPackageName)

116+{

117+ return (

118+ 'I support your platform, but I could not find the binary package (' + subPackageName + ') for it!\n\n'

119+ + 'This can happen if you use the "--omit=optional" (or "--no-optional") npm flag.\n'

120+ + 'The "optionalDependencies" package.json feature is used by Elm to install the correct\n'

121+ + 'binary executable for your current platform. Remove that flag to use Elm.\n\n'

122+ + 'This can also happen if the "node_modules" folder was copied between two operating systems\n'

123+ + 'that need different binaries - including "virtual" operating systems like Docker and WSL.\n'

124+ + 'If so, try installing with npm rather than copying "node_modules".'

125+ );

126+}

installers/npm/download.jsdeleted−114

@@ -1,114 +0,0 @@

1−var fs = require('fs');

2−var package = require('./package.json');

3−var path = require('path');

4−var request = require('request');

5−var zlib = require('zlib');

6−

7−

8−

9−// MAIN

10−//

11−// This function is used by install.js and by the bin/elm backup that gets

12−// called when --ignore-scripts is enabled. That's why install.js is so weird.

13−

14−

15−module.exports = function(callback)

16−{

17− // figure out URL of binary

18− var version = package.version.replace(/^(\d+\.\d+\.\d+).*$/, '$1'); // turn '1.2.3-alpha' into '1.2.3'

19−

20− var platform = {

21− darwin_x64: 'mac-64-bit',

22− darwin_arm64: 'mac-64-bit-ARM',

23− win32_x64: 'windows-64-bit',

24− linux_x64: 'linux-64-bit'

25− }[process.platform + '_' + process.arch];

26−

27− verifyPlatform(version, platform);

28−

29− var url = 'https://github.com/elm/compiler/releases/download/' + version + '/binary-for-' + platform + '.gz';

30−

31− reportDownload(version, url);

32−

33− // figure out where to put the binary (calls path.resolve() to get path separators right on Windows)

34− var binaryPath = path.resolve(__dirname, package.bin.elm) + (process.platform === 'win32' ? '.exe' : '');

35−

36− // set up handler for request failure

37− function reportDownloadFailure(error)

38− {

39− exitFailure(url,'Something went wrong while fetching the following URL:\n\n' + url + '\n\nIt is saying:\n\n' + error);

40− }

41−

42− // set up decompression pipe

43− var gunzip = zlib.createGunzip().on('error', function(error) {

44− exitFailure(url, 'I ran into trouble decompressing the downloaded binary. It is saying:\n\n' + error);

45− });

46−

47− // set up file write pipe

48− var write = fs.createWriteStream(binaryPath, {

49− encoding: 'binary',

50− mode: 0o755

51− }).on('finish', callback).on('error', function(error) {

52− exitFailure(url, 'I had some trouble writing file to disk. It is saying:\n\n' + error);

53− });

54−

55− // put it all together

56− request(url).on('error', reportDownloadFailure).pipe(gunzip).pipe(write);

57−}

58−

59−

60−

61−// VERIFY PLATFORM

62−

63−

64−function verifyPlatform(version, platform)

65−{

66− if (platform) return;

67−

68− var situation = process.platform + '_' + process.arch;

69− console.error(

70− '-- ERROR -----------------------------------------------------------------------\n\n'

71− + 'I am detecting that your computer (' + situation + ') may not be compatible with any\n'

72− + 'of the official pre-built binaries.\n\n'

73− + 'I recommend against using the npm installer for your situation. Check out the\n'

74− + 'alternative installers at https://github.com/elm/compiler/releases/tag/' + version + '\n'

75− + 'to see if there is something that will work better for you.\n\n'

76− + 'From there I recommend asking for guidance on Slack or Discourse to find someone\n'

77− + 'who can help with your specific situation.\n\n'

78− + '--------------------------------------------------------------------------------\n'

79− );

80− process.exit(1);

81−}

82−

83−

84−

85−// EXIT FAILURE

86−

87−

88−function exitFailure(url, message)

89−{

90− console.error(

91− '-- ERROR -----------------------------------------------------------------------\n\n'

92− + message

93− + '\n\nNOTE: You can avoid npm entirely by downloading directly from:\n'

94− + url + '\nAll this package does is download that file and put it somewhere.\n\n'

95− + '--------------------------------------------------------------------------------\n'

96− );

97− process.exit(1);

98−}

99−

100−

101−

102−// REPORT DOWNLOAD

103−

104−

105−function reportDownload(version, url)

106−{

107− console.log(

108− '--------------------------------------------------------------------------------\n\n'

109− + 'Downloading Elm ' + version + ' from GitHub.'

110− + '\n\nNOTE: You can avoid npm entirely by downloading directly from:\n'

111− + url + '\nAll this package does is download that file and put it somewhere.\n\n'

112− + '--------------------------------------------------------------------------------\n'

113− );

114−}

installers/npm/install.js+1−4

@@ -1,4 +1 @@

1−

2−var download = require('./download.js');

3−

4−download(function() {});

1+require('./binary.js')();

installers/npm/package.json+7−4

@@ -2,7 +2,6 @@

22 "name": "elm",

33 "version": "0.19.1-5",

44 "description": "Installer for Elm: just downloads the binary into node_modules",

5− "preferGlobal": true,

65 "license": "BSD-3-Clause",

76 "repository": {

87 "type": "git",

@@ -23,7 +22,7 @@

2322 },

2423 "files": [

2524 "install.js",

26− "download.js",

25+ "binary.js",

2726 "bin",

2827 "bin/elm"

2928 ],

@@ -36,7 +35,11 @@

3635 "installer"

3736 ],

3837 "bin": { "elm": "bin/elm" },

39− "dependencies": {

40− "request": "^2.88.0"

38+ "optionalDependencies": {

39+ "@evancz/elm_darwin_arm64": "0.19.1-0",

40+ "@evancz/elm_darwin_x64": "0.19.1-0",

41+ "@evancz/elm_linux_arm64": "0.19.1-0",

42+ "@evancz/elm_linux_x64": "0.19.1-0",

43+ "@evancz/elm_win32_x64": "0.19.1-0"

4144 }

4245}

installers/npm/packages/elm_darwin_arm64/README.mdnew file+3

@@ -0,0 +1,3 @@

1+# Elm

3+This is the macOS ARM 64-bit binary for [Elm](https://elm-lang.org/).

installers/npm/packages/elm_darwin_arm64/package.jsonnew file+9

@@ -0,0 +1,9 @@

1+{

2+ "name": "@evancz/elm_darwin_arm64",

3+ "version": "0.19.1-0",

4+ "description": "The macOS ARM 64-bit binary for Elm.",

5+ "repository": "https://github.com/elm/compiler",

6+ "license": "BSD-3-Clause",

7+ "os": [ "darwin" ],

8+ "cpu": [ "arm64" ]

9+}

installers/npm/packages/elm_darwin_x64/README.mdnew file+3

@@ -0,0 +1,3 @@

1+# Elm

3+This is the macOS 64-bit binary for [Elm](https://elm-lang.org/).

installers/npm/packages/elm_darwin_x64/package.jsonnew file+9

@@ -0,0 +1,9 @@

1+{

2+ "name": "@evancz/elm_darwin_x64",

3+ "version": "0.19.1-0",

4+ "description": "The macOS 64-bit binary for Elm.",

5+ "repository": "https://github.com/elm/compiler",

6+ "license": "BSD-3-Clause",

7+ "os": [ "darwin" ],

8+ "cpu": [ "x64" ]

9+}

installers/npm/packages/elm_linux_arm64/README.mdnew file+3

@@ -0,0 +1,3 @@

1+# Elm

3+This is the Linux ARM 64-bit binary for [Elm](https://elm-lang.org/).

installers/npm/packages/elm_linux_arm64/package.jsonnew file+9

@@ -0,0 +1,9 @@

1+{

2+ "name": "@evancz/elm_linux_arm64",

3+ "version": "0.19.1-0",

4+ "description": "The Linux ARM 64-bit binary for Elm.",

5+ "repository": "https://github.com/elm/compiler",

6+ "license": "BSD-3-Clause",

7+ "os": [ "linux" ],

8+ "cpu": [ "arm64" ]

9+}

installers/npm/packages/elm_linux_x64/README.mdnew file+3

@@ -0,0 +1,3 @@

1+# Elm

3+This is the Linux 64-bit binary for [Elm](https://elm-lang.org/).

installers/npm/packages/elm_linux_x64/package.jsonnew file+9

@@ -0,0 +1,9 @@

1+{

2+ "name": "@evancz/elm_linux_x64",

3+ "version": "0.19.1-0",

4+ "description": "The Linux 64-bit binary for Elm.",

5+ "repository": "https://github.com/elm/compiler",

6+ "license": "BSD-3-Clause",

7+ "os": [ "linux" ],

8+ "cpu": [ "x64" ]

9+}

installers/npm/packages/elm_win32_x64/README.mdnew file+3

@@ -0,0 +1,3 @@

1+# Elm

3+This is the Windows 64-bit binary for [Elm](https://elm-lang.org/).

installers/npm/packages/elm_win32_x64/package.jsonnew file+9

@@ -0,0 +1,9 @@

1+{

2+ "name": "@evancz/elm_win32_x64",

3+ "version": "0.19.1-0",

4+ "description": "The Windows 64-bit binary for Elm.",

5+ "repository": "https://github.com/elm/compiler",

6+ "license": "BSD-3-Clause",

7+ "os": [ "win32" ],

8+ "cpu": [ "x64" ]

9+}

installers/npm/troubleshooting.md+2−2

@@ -42,7 +42,7 @@ Some companies have a firewall.

4242

4343These companies usually have set the `HTTP_PROXY` or `HTTPS_PROXY` environment variable on your computer. This is more common with Windows computers.

4444

45−The result is that the request for `https://github.com/elm/compiler/releases/download/0.19.1/binary-for-windows-64-bit.gz` is being sent to a "proxy server" where they monitor traffic. Maybe they rule out certain domains, maybe they check data when it comes back from the actual URL, etc.

45+The result is that the requests for npm packages are being sent to a "proxy server" where they monitor traffic. Maybe they rule out certain domains, maybe they check data when it comes back from the actual URL, etc.

4646

4747It is probably best to ask someone about the situation on this, but you can test things out by temporarily using an alternate `HTTPS_PROXY` value with something like this:

4848

@@ -55,7 +55,7 @@ set HTTPS_PROXY=http://proxy.example.com

5555npm install -g elm

5656```

5757

58−Check out [this document](https://www.npmjs.com/package/request#controlling-proxy-behaviour-using-environment-variables) for more information on how environment variables like `NO_PROXY`, `HTTP_PROXY`, and `HTTPS_PROXY` are handled by the npm.

58+Check out [this document](https://docs.npmjs.com/cli/using-npm/config/) for more information on how environment variables like [NO_PROXY](https://docs.npmjs.com/cli/using-npm/config#noproxy), [HTTP_PROXY](https://docs.npmjs.com/cli/using-npm/config#proxy), and [HTTPS_PROXY](https://docs.npmjs.com/cli/using-npm/config#https-proxy) are handled by npm.

5959

6060<br/>

6161