on main help sign in
Import everything without history
commitfa69247ce75bdb7b66fdde32b66e1205f13482e5 authorSönke Hahn <soenkehahn@gmail.com> dateWed 27 May 2026, 14:36 (4 months ago)
589 files changed+118594
.gitattributes+3 .gitignore+25 .hlint.yaml+1 .prettierrc+3 .sops.yaml+9 LICENSE+28 README.md+58 backend/.gitignore+2 backend/.golden/.gitignore+1 backend/.golden/ConfigSchemaSpec/garnix-config-schema.json/golden+215 backend/.golden/FeatureFlagConfigDbo.json+528 backend/.golden/HostSpec/adds-hosts-when-a-new-server-is-deployed/golden+33 backend/.golden/HostSpec/reflects-the-latest-deployed-versions/golden+33 backend/.golden/HostSpec/serves-configurations-for-pull-request-servers/golden+33 backend/.golden/HostSpec/serves-pull-request-configurations-when-the-branch-name-is-not-a-valid-subdomain/golden+33 backend/.golden/HostSpec/supports-exposing-more-than-one-configuration-per-repo/golden+65 backend/.hlint.yaml+17 backend/cabal.project+2 backend/data/badge_favicon.svg+4 backend/default.nix+242 backend/dev-action-runner-ssh-key+7 backend/dev-key.jwt backend/exe/GenerateYamlConfigDocs.hs+10 backend/exe/Server.hs+7 backend/exe/convertHashids.hs+18 backend/garnix.cabal+1047 backend/nixos-module.nix+636 backend/package.yaml+217 backend/src/Garnix.hs+418 backend/src/Garnix/API.hs+115 backend/src/Garnix/API/Account.hs+502 backend/src/Garnix/API/Auth.hs+335 backend/src/Garnix/API/Badges.hs+62 backend/src/Garnix/API/Builds.hs+170 backend/src/Garnix/API/Builds/Types.hs+15 backend/src/Garnix/API/Cache.hs+173 backend/src/Garnix/API/Cache/Auth.hs+68 backend/src/Garnix/API/Cache/Permissions.hs+74 backend/src/Garnix/API/Cache/Types.hs+73 backend/src/Garnix/API/Commits.hs+81 backend/src/Garnix/API/ConfigSchema.hs+7 backend/src/Garnix/API/Dev.hs+47 backend/src/Garnix/API/GhWebhooks.hs+223 backend/src/Garnix/API/Health.hs+26 backend/src/Garnix/API/Hosts.hs+198 backend/src/Garnix/API/Keys.hs+51 backend/src/Garnix/API/Modules.hs+146 backend/src/Garnix/API/Runs.hs+71 backend/src/Garnix/API/Stripe.hs+60 backend/src/Garnix/Access.hs+82 backend/src/Garnix/AccessToken.hs+28 backend/src/Garnix/AccessToken/Types.hs+45 backend/src/Garnix/Async.hs+54 backend/src/Garnix/Attribute.hs+274 backend/src/Garnix/Build.hs+81 backend/src/Garnix/Build/Action.hs+357 backend/src/Garnix/Build/Checkout.hs+63 backend/src/Garnix/Build/Evaluation.hs+216 backend/src/Garnix/Build/Flake.hs+173 backend/src/Garnix/Build/FodCheck.hs+301 backend/src/Garnix/Build/Helpers.hs+36 backend/src/Garnix/Build/MetaCheck.hs+98 backend/src/Garnix/Build/Module.hs+191 backend/src/Garnix/Build/Package.hs+266 backend/src/Garnix/Build/Reporting.hs+55 backend/src/Garnix/Build/Types.hs+21 backend/src/Garnix/BuildLogs.hs+90 backend/src/Garnix/BuildLogs/Internal.hs+117 backend/src/Garnix/BuildLogs/Types.hs+14 backend/src/Garnix/DB.hs+1995 backend/src/Garnix/DB/FeatureFlags.hs+51 backend/src/Garnix/DB/FeatureFlags/Types.hs+62 backend/src/Garnix/DB/ModuleValues.hs+330 backend/src/Garnix/Duration.hs+106 backend/src/Garnix/Entitlements.hs+450 backend/src/Garnix/ExpiringCache.hs+76 backend/src/Garnix/FlakeInputAuthorization.hs+225 backend/src/Garnix/GetAttributes.hs+114 backend/src/Garnix/GithubInterface.hs+502 backend/src/Garnix/GithubInterface/Types.hs+26 backend/src/Garnix/HetznerInterface.hs+190 backend/src/Garnix/Hosting/Deploy.hs+482 backend/src/Garnix/Hosting/Helpers.hs+227 backend/src/Garnix/Hosting/ServerPool.hs+175 backend/src/Garnix/Hosting/ServerPool/Types.hs+80 backend/src/Garnix/Incremental.hs+153 backend/src/Garnix/Log.hs+118 backend/src/Garnix/Modules.hs+45 backend/src/Garnix/Modules/Schema.hs+163 backend/src/Garnix/Modules/module-to-schema.nix+103 backend/src/Garnix/Monad.hs+726 backend/src/Garnix/Monad/Async.hs+68 backend/src/Garnix/Monad/Bubbling.hs+22 backend/src/Garnix/Monad/Concurrency.hs+17 backend/src/Garnix/Monad/ForkT.hs+107 backend/src/Garnix/Monad/Memoization.hs+35 backend/src/Garnix/Monad/Metrics.hs+202 backend/src/Garnix/Monad/NoThrow.hs+29 backend/src/Garnix/Monad/Polling.hs+27 backend/src/Garnix/Monad/Pool.hs+121 backend/src/Garnix/Monad/SubProcess.hs+115 backend/src/Garnix/Monad/SubProcess/Deprecated.hs+58 backend/src/Garnix/MonetaryCost.hs+41 backend/src/Garnix/Nix/PathInfo.hs+53 backend/src/Garnix/Nix/Plan.hs+51 backend/src/Garnix/Nix/StorePath.hs+113 backend/src/Garnix/Nix/Types.hs+117 backend/src/Garnix/NixConfig.hs+38 backend/src/Garnix/Orchestrator.hs+103 backend/src/Garnix/ParseHttpBasicAuth.hs+15 backend/src/Garnix/Password.hs+23 backend/src/Garnix/Prelude.hs+431 backend/src/Garnix/Reporters/GithubReporter.hs+97 backend/src/Garnix/Reporters/OpenSearchReporter.hs+45 backend/src/Garnix/Reporters/Utils.hs+60 backend/src/Garnix/Request.hs+46 backend/src/Garnix/S3Cache.hs+217 backend/src/Garnix/SafeUnix.hs+29 backend/src/Garnix/Sandbox.hs+102 backend/src/Garnix/StripeLib.hs+298 backend/src/Garnix/StripeLib/Types.hs+238 backend/src/Garnix/Types.hs+1808 backend/src/Garnix/Types/ExternalLenses.hs+18 backend/src/Garnix/Types/Keys.hs+94 backend/src/Garnix/Types/MakeLensHelpers.hs+25 backend/src/Garnix/UserLogs.hs+213 backend/src/Garnix/YamlConfig.hs+464 backend/ssh-key-for-tests+38 backend/ssh-key-for-tests.pub+1 backend/test/spec/Garnix/API/AccountSpec.hs+1140 backend/test/spec/Garnix/API/AuthSpec.hs+146 backend/test/spec/Garnix/API/BadgesSpec.hs+187 backend/test/spec/Garnix/API/BuildSpec.hs+389 backend/test/spec/Garnix/API/Cache/PermissionsSpec.hs+97 backend/test/spec/Garnix/API/CommitSpec.hs+461 backend/test/spec/Garnix/API/ConfigSchemaSpec.hs+28 backend/test/spec/Garnix/API/DevSpec.hs+61 backend/test/spec/Garnix/API/GhWebhooksSpec.hs+19 backend/test/spec/Garnix/API/HealthSpec.hs+16 backend/test/spec/Garnix/API/HostsSpec.hs+522 backend/test/spec/Garnix/API/KeysSpec.hs+56 backend/test/spec/Garnix/API/ModulesSpec.hs+278 backend/test/spec/Garnix/API/RunsSpec.hs+234 backend/test/spec/Garnix/AccessTokenSpec.hs+79 backend/test/spec/Garnix/ActionSpec.hs+525 backend/test/spec/Garnix/AsyncSpec.hs+91 backend/test/spec/Garnix/AttributeSpec.hs+85 backend/test/spec/Garnix/Build/CheckoutSpec.hs+42 backend/test/spec/Garnix/Build/FodCheckSpec.hs+538 backend/test/spec/Garnix/Build/ModuleSpec.hs+424 backend/test/spec/Garnix/BuildSpec.hs+1344 backend/test/spec/Garnix/DB/FeatureFlagsSpec.hs+73 backend/test/spec/Garnix/DBSpec.hs+216 backend/test/spec/Garnix/DeploySpec.hs+1583 backend/test/spec/Garnix/EntitlementsSpec.hs+96 backend/test/spec/Garnix/ExpiringCacheSpec.hs+93 backend/test/spec/Garnix/FlakeInputAuthorizationSpec.hs+139 backend/test/spec/Garnix/GetAttributesSpec.hs+214 backend/test/spec/Garnix/GithubInterfaceSpec.hs+54 backend/test/spec/Garnix/HetznerInterfaceSpec.hs+24 backend/test/spec/Garnix/Hosting/BillingSpec.hs+282 backend/test/spec/Garnix/Hosting/DeploySpec.hs+39 backend/test/spec/Garnix/Hosting/ServerPoolSpec.hs+152 backend/test/spec/Garnix/IncrementalSpec.hs+184 backend/test/spec/Garnix/Modules/SchemaSpec.hs+529 backend/test/spec/Garnix/ModulesSpec.hs+145 backend/test/spec/Garnix/Monad/BubblingSpec.hs+34 backend/test/spec/Garnix/Monad/ConcurrencySpec.hs+51 backend/test/spec/Garnix/Monad/ForkTSpec.hs+165 backend/test/spec/Garnix/Monad/MemoizationSpec.hs+57 backend/test/spec/Garnix/Monad/NoThrowSpec.hs+104 backend/test/spec/Garnix/Monad/PoolSpec.hs+134 backend/test/spec/Garnix/Monad/SubProcessSpec.hs+112 backend/test/spec/Garnix/MonadSpec.hs+145 backend/test/spec/Garnix/Nix/StorePathSpec.hs+117 backend/test/spec/Garnix/PreludeSpec.hs+35 backend/test/spec/Garnix/ProcessLogsSpec.hs+282 backend/test/spec/Garnix/Reporters/GithubReporterSpec.hs+106 backend/test/spec/Garnix/Reporters/OpenSearchReporterSpec.hs+133 backend/test/spec/Garnix/Reporters/UtilsSpec.hs+108 backend/test/spec/Garnix/S3CacheSpec.hs+833 backend/test/spec/Garnix/SandboxSpec.hs+267 backend/test/spec/Garnix/StripeLibSpec.hs+138 backend/test/spec/Garnix/TestHelpers.hs+601 backend/test/spec/Garnix/TestHelpers/Common.hs+23 backend/test/spec/Garnix/TestHelpers/Deprecated.hs+145 backend/test/spec/Garnix/TestHelpers/GithubInterface.hs+179 backend/test/spec/Garnix/TestHelpers/GithubInterface/Deprecated.hs+52 backend/test/spec/Garnix/TestHelpers/GithubInterface/Internal.hs+243 backend/test/spec/Garnix/TestHelpers/HetznerMock.hs+244 backend/test/spec/Garnix/TestHelpers/Monad.hs+422 backend/test/spec/Garnix/TestHelpers/NixosVmScripts.hs+41 backend/test/spec/Garnix/TestHelpers/Reporter.hs+55 backend/test/spec/Garnix/TestHelpers/ServerPool.hs+86 backend/test/spec/Garnix/TestHelpers/WithServer.hs+117 backend/test/spec/Garnix/TestInstances.hs+208 backend/test/spec/Garnix/TypesSpec.hs+101 backend/test/spec/Garnix/UserLogsSpec.hs+140 backend/test/spec/Garnix/YamlConfigSpec.hs+414 backend/test/spec/Integration/FlakesSpec.hs+340 backend/test/spec/Integration/bad-flake-nix/flake.nix+1 backend/test/spec/Integration/bad-flake-nix/spec.yaml+7 backend/test/spec/Integration/bad-yaml-config/garnix.yaml+5 backend/test/spec/Integration/bad-yaml-config/spec.yaml+7 backend/test/spec/Integration/basic/flake.lock+25 backend/test/spec/Integration/basic/flake.nix+44 backend/test/spec/Integration/basic/spec.yaml+11 backend/test/spec/Integration/darwin-configuration/flake.lock+61 backend/test/spec/Integration/darwin-configuration/flake.nix+42 backend/test/spec/Integration/darwin-configuration/spec.yaml+12 backend/test/spec/Integration/failed-dependency/flake.lock+25 backend/test/spec/Integration/failed-dependency/flake.nix+61 backend/test/spec/Integration/failed-dependency/spec.yaml+13 backend/test/spec/Integration/home-configuration/flake.lock+46 backend/test/spec/Integration/home-configuration/flake.nix+35 backend/test/spec/Integration/home-configuration/spec.yaml+12 backend/test/spec/Integration/nixos-configuration/flake.lock+27 backend/test/spec/Integration/nixos-configuration/flake.nix+33 backend/test/spec/Integration/nixos-configuration/spec.yaml+12 backend/test/spec/Integration/private-github-dependency-different-org-failing/flake.lock+42 backend/test/spec/Integration/private-github-dependency-different-org-failing/flake.nix+8 backend/test/spec/Integration/private-github-dependency-different-org-failing/spec.yaml+9 backend/test/spec/Integration/private-github-dependency-external-pr/flake.lock+42 backend/test/spec/Integration/private-github-dependency-external-pr/flake.nix+8 backend/test/spec/Integration/private-github-dependency-external-pr/spec.yaml+9 backend/test/spec/Integration/private-github-dependency-succeeding/flake.lock+42 backend/test/spec/Integration/private-github-dependency-succeeding/flake.nix+35 backend/test/spec/Integration/private-github-dependency-succeeding/garnix.yaml backend/test/spec/Integration/private-github-dependency-succeeding/spec.yaml+8 backend/test/spec/Integration/private-github-dependency-users-without-access-failing/flake.lock+42 backend/test/spec/Integration/private-github-dependency-users-without-access-failing/flake.nix+8 backend/test/spec/Integration/private-github-dependency-users-without-access-failing/spec.yaml+12 backend/test/spec/Integration/private-github-dependency-users-without-access-skip-check-succeeding/flake.lock+42 backend/test/spec/Integration/private-github-dependency-users-without-access-skip-check-succeeding/flake.nix+8 backend/test/spec/Integration/private-github-dependency-users-without-access-skip-check-succeeding/garnix.yaml+1 backend/test/spec/Integration/private-github-dependency-users-without-access-skip-check-succeeding/spec.yaml+11 backend/test/spec/Integration/public-github-dependency-external-pr/flake.lock+42 backend/test/spec/Integration/public-github-dependency-external-pr/flake.nix+36 backend/test/spec/Integration/public-github-dependency-external-pr/spec.yaml+9 backend/test/spec/Integration/public-github-dependency/flake.lock+42 backend/test/spec/Integration/public-github-dependency/flake.nix+34 backend/test/spec/Integration/public-github-dependency/spec.yaml+7 backend/test/spec/Integration/public-tarball-dependency/flake.lock+38 backend/test/spec/Integration/public-tarball-dependency/flake.nix+33 backend/test/spec/Integration/public-tarball-dependency/spec.yaml+7 backend/test/spec/Integration/yaml-config/flake.lock+27 backend/test/spec/Integration/yaml-config/flake.nix+20 backend/test/spec/Integration/yaml-config/garnix.yaml+5 backend/test/spec/Integration/yaml-config/spec.yaml+11 backend/test/spec/Spec.hs+1 backend/test/spec/SpecHook.hs+29 backend/test/spec/data/create-server-response.json+235 backend/test/spec/data/emptyDir/.gitignore backend/test/spec/data/flake.lock+364 backend/test/spec/data/get-servers-response.json+400 backend/test/spec/data/repo-secrets.key+3 cabal.project+1 dns/.gitignore+1 dns/Cargo.lock+2289 dns/Cargo.toml+18 dns/default.nix+32 dns/nixos-module.nix+59 dns/src/dns.rs+111 dns/src/main.rs+387 dns/src/test_helpers.rs+78 examples/default.nix+13 examples/example-multi-server-deployment.nix+91 flake.lock+251 flake.nix+169 frontend/.eslintrc.json+50 frontend/.gitignore+22 frontend/README.md+36 frontend/age-wasm/default.nix+49 frontend/age-wasm/go.mod+10 frontend/age-wasm/go.sum+6 frontend/age-wasm/index.ts+21 frontend/age-wasm/integration-test.mjs+58 frontend/age-wasm/main.go+61 frontend/age-wasm/tsconfig.json+8 frontend/default.nix+93 frontend/jest-wasm-mock.js+1 frontend/jest.config.ts+16 frontend/knip.json+12 frontend/next-env.d.ts+5 frontend/next.config.js+48 frontend/package-lock.json+15025 frontend/package.json+60 frontend/src/app/account/access_tokens/index.tsx+386 frontend/src/app/account/access_tokens/styles.module.css+61 frontend/src/app/account/gh/[slug]/page.tsx+370 frontend/src/app/account/gh/[slug]/styles.module.css+198 frontend/src/app/account/manage_plans/page.tsx+340 frontend/src/app/account/manage_plans/styles.module.css+50 frontend/src/app/account/page.tsx+24 frontend/src/app/account/styles.module.css+16 frontend/src/app/account/usage/index.tsx+104 frontend/src/app/account/usage/styles.module.css+28 frontend/src/app/build/[slug]/page.tsx+194 frontend/src/app/build/[slug]/styles.module.css+77 frontend/src/app/build/layout.tsx+7 frontend/src/app/commit/[slug]/page.tsx+72 frontend/src/app/commit/[slug]/styles.module.css+53 frontend/src/app/commit/layout.tsx+7 frontend/src/app/favicon.ico frontend/src/app/garnix-admin/createGithubApp.tsx+121 frontend/src/app/garnix-admin/page.tsx+78 frontend/src/app/garnix-admin/styles.module.css+15 frontend/src/app/globals.css+67 frontend/src/app/layout.tsx+65 frontend/src/app/login/cb/page.tsx+78 frontend/src/app/login/cb/styles.module.css+12 frontend/src/app/login/page.tsx+32 frontend/src/app/login/styles.module.css+7 frontend/src/app/logout/page.tsx+17 frontend/src/app/modules/configure/markdown/index.tsx+23 frontend/src/app/modules/configure/markdown/styles.module.css+16 frontend/src/app/modules/configure/moduleInputs/list/index.tsx+58 frontend/src/app/modules/configure/moduleInputs/list/styles.module.css+25 frontend/src/app/modules/configure/moduleInputs/listOfPackages/addPackageModal/all-nix-pkgs.ts+22211 frontend/src/app/modules/configure/moduleInputs/listOfPackages/addPackageModal/index.tsx+121 frontend/src/app/modules/configure/moduleInputs/listOfPackages/addPackageModal/styles.module.css+36 frontend/src/app/modules/configure/moduleInputs/listOfPackages/index.tsx+41 frontend/src/app/modules/configure/moduleInputs/listOfPackages/styles.module.css+26 frontend/src/app/modules/configure/moduleInputs/moduleSelector/index.tsx+46 frontend/src/app/modules/configure/moduleInputs/moduleSelector/styles.module.css+19 frontend/src/app/modules/configure/moduleInputs/nixPath.tsx+42 frontend/src/app/modules/configure/moduleInputs/repoPicker/index.tsx+69 frontend/src/app/modules/configure/moduleInputs/repoPicker/styles.module.css+42 frontend/src/app/modules/configure/moduleInputs/secretInput/index.tsx+124 frontend/src/app/modules/configure/moduleInputs/secretInput/styles.module.css+26 frontend/src/app/modules/configure/openPrModal/index.tsx+58 frontend/src/app/modules/configure/openPrModal/styles.module.css+42 frontend/src/app/modules/configure/page.test.tsx+1083 frontend/src/app/modules/configure/page.tsx+675 frontend/src/app/modules/configure/previewModal/index.tsx+49 frontend/src/app/modules/configure/previewModal/styles.module.css+36 frontend/src/app/modules/configure/resetModal/index.tsx+46 frontend/src/app/modules/configure/resetModal/styles.module.css+36 frontend/src/app/modules/configure/styles.module.css+56 frontend/src/app/og.png frontend/src/app/page.tsx+10 frontend/src/app/repo/[owner]/[repo]/page.tsx+15 frontend/src/app/run/[slug]/page.tsx+38 frontend/src/app/run/layout.tsx+7 frontend/src/app/servers/filters/index.tsx+71 frontend/src/app/servers/filters/styles.module.css+13 frontend/src/app/servers/page.tsx+228 frontend/src/app/servers/styles.module.css+56 frontend/src/app/signup/fill/page.tsx+74 frontend/src/app/signup/fill/styles.module.css+12 frontend/src/app/signup/page.tsx+70 frontend/src/app/signup/start/page.tsx+126 frontend/src/app/signup/start/styles.module.css+85 frontend/src/app/signup/styles.module.css+50 frontend/src/app/signup/tos/page.tsx+28 frontend/src/app/signup/tos/styles.module.css+20 frontend/src/components/authModalHeader/index.tsx+78 frontend/src/components/authModalHeader/styles.module.css+78 frontend/src/components/body/index.tsx+8 frontend/src/components/build/index.tsx+123 frontend/src/components/build/styles.module.css+62 frontend/src/components/buildLog/index.tsx+103 frontend/src/components/buildLog/styles.module.css+72 frontend/src/components/button/index.tsx+87 frontend/src/components/button/styles.module.css+58 frontend/src/components/commitList/index.tsx+81 frontend/src/components/commitList/styles.module.css+67 frontend/src/components/expander/index.tsx+22 frontend/src/components/expander/styles.module.css+4 frontend/src/components/formSubmitResult/index.tsx+15 frontend/src/components/formSubmitResult/styles.module.css+16 frontend/src/components/icons/account.tsx+20 frontend/src/components/icons/arrow-left.svg+3 frontend/src/components/icons/arrow-right.svg+3 frontend/src/components/icons/billing.tsx+20 frontend/src/components/icons/blocks.svg+1264 frontend/src/components/icons/blog.svg+5 frontend/src/components/icons/branch.svg+16 frontend/src/components/icons/builds.tsx+20 frontend/src/components/icons/clock.svg+17 frontend/src/components/icons/commit.svg+14 frontend/src/components/icons/cross.svg+11 frontend/src/components/icons/cube.svg+17 frontend/src/components/icons/dash.svg+4 frontend/src/components/icons/discord.tsx+32 frontend/src/components/icons/docs.svg+5 frontend/src/components/icons/documentation.tsx+20 frontend/src/components/icons/download.tsx+9 frontend/src/components/icons/email.tsx+32 frontend/src/components/icons/failure.svg+13 frontend/src/components/icons/folder.svg+16 frontend/src/components/icons/github.tsx+24 frontend/src/components/icons/globe.svg+1268 frontend/src/components/icons/hamburgerMenu.svg+6 frontend/src/components/icons/lock.svg+4 frontend/src/components/icons/login.svg+5 frontend/src/components/icons/logo.svg+31 frontend/src/components/icons/logoWhite.svg+4 frontend/src/components/icons/logout.tsx+20 frontend/src/components/icons/matrix.tsx+21 frontend/src/components/icons/menu.svg+5 frontend/src/components/icons/modules.tsx+32 frontend/src/components/icons/pending.svg+19 frontend/src/components/icons/repo.svg+25 frontend/src/components/icons/servers.tsx+20 frontend/src/components/icons/settings.tsx+20 frontend/src/components/icons/share.svg+10 frontend/src/components/icons/status.svg+18 frontend/src/components/icons/stopwatch.svg+18 frontend/src/components/icons/success.svg+11 frontend/src/components/icons/terminal.svg+10 frontend/src/components/icons/timestamp.svg+17 frontend/src/components/icons/trash.tsx+11 frontend/src/components/icons/video.svg+5 frontend/src/components/icons/x.svg+5 frontend/src/components/images/globe.svg+786 frontend/src/components/input/index.tsx+85 frontend/src/components/input/styles.module.css+18 frontend/src/components/link/index.tsx+51 frontend/src/components/link/styles.module.css+55 frontend/src/components/loading/index.tsx+99 frontend/src/components/loading/styles.module.css+14 frontend/src/components/loginAnimation/animation.json+1 frontend/src/components/loginAnimation/index.tsx+34 frontend/src/components/loginAnimation/styles.module.css+6 frontend/src/components/loginPrompt/index.tsx+34 frontend/src/components/loginPrompt/styles.module.css+11 frontend/src/components/mdxDocument/header/index.tsx+51 frontend/src/components/mdxDocument/header/styles.module.css+42 frontend/src/components/mdxDocument/index.tsx+183 frontend/src/components/mdxDocument/styles.module.css+141 frontend/src/components/modal/index.tsx+55 frontend/src/components/modal/styles.module.css+61 frontend/src/components/navigation/index.tsx+243 frontend/src/components/navigation/styles.module.css+203 frontend/src/components/portal/index.tsx+17 frontend/src/components/run/index.tsx+84 frontend/src/components/run/styles.module.css+77 frontend/src/components/sampleCode.tsx+39 frontend/src/components/select/index.tsx+37 frontend/src/components/select/styles.module.css+24 frontend/src/components/statusIcon/index.tsx+50 frontend/src/components/statusIcon/styles.module.css+30 frontend/src/components/table/index.tsx+15 frontend/src/components/table/styles.module.css+27 frontend/src/components/terminal/index.tsx+34 frontend/src/components/terminal/styles.module.css+50 frontend/src/components/text/index.tsx+87 frontend/src/components/text/styles.module.css+64 frontend/src/components/toggleSwitch/index.tsx+24 frontend/src/components/toggleSwitch/styles.module.css+42 frontend/src/components/tooltip/index.tsx+35 frontend/src/components/tooltip/styles.module.css+22 frontend/src/components/withSidebar/index.tsx+25 frontend/src/components/withSidebar/styles.module.css+65 frontend/src/hooks/useDebounced.ts+15 frontend/src/hooks/useForm.ts+98 frontend/src/hooks/useLoading.ts+55 frontend/src/hooks/useLogStream.ts+80 frontend/src/hooks/useLoginLinkForCurrentPage.ts+15 frontend/src/middleware.ts+39 frontend/src/services/account.ts+288 frontend/src/services/apiErrors.ts+7 frontend/src/services/auth.ts+114 frontend/src/services/build.ts+72 frontend/src/services/commit.ts+72 frontend/src/services/config.ts+14 frontend/src/services/index.ts+70 frontend/src/services/keys.ts+8 frontend/src/services/logs.ts+32 frontend/src/services/modules/index.test.ts+264 frontend/src/services/modules/index.ts+526 frontend/src/services/modules/nixValue.ts+51 frontend/src/services/run.ts+31 frontend/src/services/servers.ts+43 frontend/src/store/configContext.tsx+40 frontend/src/store/providers.tsx+13 frontend/src/store/userContext.tsx+87 frontend/src/utils/analytics.ts+8 frontend/src/utils/ansi/colors.ts+263 frontend/src/utils/ansi/index.test.ts+340 frontend/src/utils/ansi/index.ts+124 frontend/src/utils/appPage.tsx+36 frontend/src/utils/colors.css+20 frontend/src/utils/currency.test.ts+54 frontend/src/utils/currency.ts+18 frontend/src/utils/custom.d.ts+9 frontend/src/utils/deferred.ts+15 frontend/src/utils/dev.tsx+10 frontend/src/utils/duration.test.ts+110 frontend/src/utils/duration.ts+74 frontend/src/utils/featureFlags.ts+30 frontend/src/utils/fonts/BerlinTypeWeb-Regular.woff frontend/src/utils/fonts/MatterSQMono-Light.woff frontend/src/utils/fonts/MatterSQMono-LightItalic.woff frontend/src/utils/fonts/MatterSQMono-Regular.woff frontend/src/utils/fonts/MatterSQMono-RegularItalic.woff frontend/src/utils/fonts/index.ts+34 frontend/src/utils/format.ts+52 frontend/src/utils/index.test.ts+55 frontend/src/utils/index.ts+52 frontend/src/utils/sizes.css+6 frontend/src/utils/withPropCheck.ts+8 frontend/src/utils/zod.ts+7 frontend/tsconfig.json+28 garnix.yaml+18 hosting-gateway/default.nix+51 hosting-gateway/heartbeatmiddleware/.traefik.yml+5 hosting-gateway/heartbeatmiddleware/go.mod+3 hosting-gateway/heartbeatmiddleware/hostset.go+32 hosting-gateway/heartbeatmiddleware/plugin.go+70 hosting-gateway/nixos-module.nix+184 hosting-gateway/on-demand-resolver/.gitignore+2 hosting-gateway/on-demand-resolver/package-lock.json+59 hosting-gateway/on-demand-resolver/package.json+16 hosting-gateway/on-demand-resolver/src/index.ts+49 hosting-gateway/on-demand-resolver/src/lib.test.ts+199 hosting-gateway/on-demand-resolver/src/lib.ts+58 hosting-gateway/on-demand-resolver/tsconfig.json+16 justfile+48 nix/data/emptyDir/.gitignore nix/data/grafana-node-exporter-full.json+14119 nix/data/ssh-key-for-local-dev-secrets+7 nix/data/ssh-key-for-local-dev-secrets.pub+1 nix/modules/action-runner.nix+261 nix/modules/common.nix+79 nix/modules/custom-gc.nix+174 nix/modules/database.nix+414 nix/modules/default.nix+15 nix/modules/dev-mode.nix+94 nix/modules/fluent-bit.nix+356 nix/modules/linux-common.nix+452 nix/modules/monitoring-client.nix+157 nix/modules/monitoring-server.nix+178 nix/modules/monitoring.nix+76 nix/packages/HDBC.nix+33 nix/packages/db.nix+96 nix/packages/garnix.nix+453 nix/packages/generic-random.nix+11 nix/packages/generics-eot.nix+42 nix/packages/github-app.nix+60 nix/packages/github-webhooks.nix+43 nix/packages/iso-deriving.nix+20 nix/packages/migrate.nix+14 nix/packages/oauth2-simple.nix+46 nix/packages/opensearch-dashboards/default.nix+49 nix/packages/opensearch-dashboards/disable-nodejs-version-check.patch+10 nix/packages/servant-github-webhook.nix+62 nix/packages/withSecrets.nix+57 nix/tests/data/foo.garnix.me.cert.pem+20 nix/tests/data/foo.garnix.me.key.pem+27 nix/tests/data/id_ed25519_garnix_server_testing+7 nix/tests/data/id_ed25519_garnix_server_testing.pub+1 nix/tests/database.nix+57 nix/tests/default.nix+197 nix/tests/fluent-bit.nix+88 nix/tests/garnixServer.nix+67 nix/tests/hosting-gateway.nix+223 nix/tests/monitoring-node.nix+64 nix/tests/monitoring-server.nix+18 nix/tests/opensearch.nix+81 nix/treefmt.nix+40 opensearch/default.nix+3 opensearch/nixos-module.nix+255 secrets/dev.yaml+45 sql/README.md+30 sql/deploy/init.sql+527 sql/local-fixtures.sql+154 sql/queries/build_time_percentiles.sql+32 sql/queries/monthly_build_times.sql+15 sql/sqitch.conf+2 sql/sqitch.plan+4 watchdog/cabal.project+1 watchdog/data/ci/date+1 watchdog/data/ci/flake.lock+60 watchdog/data/ci/flake.nix+14 watchdog/data/ci/garnix.yaml+3 watchdog/data/pr-deployment/date+1 watchdog/data/pr-deployment/flake.lock+25 watchdog/data/pr-deployment/flake.nix+33 watchdog/data/pr-deployment/garnix.yaml+4 watchdog/default.nix+60 watchdog/nixos-module.nix+55 watchdog/package.yaml+73 watchdog/src/Garnix/Watchdog/Checks.hs+284 watchdog/src/Garnix/Watchdog/Github.hs+48 watchdog/src/Garnix/Watchdog/Utils.hs+39 watchdog/src/Main.hs+70 watchdog/watchdog.cabal+78
.gitattributesnew file+3
@@ -0,0 +1,3 @@
1+# Don’t render generated files by default in GitHub diffs
2+backend/garnix.cabal linguist-generated
3+nix/packages/garnix.nix linguist-generated
.gitignorenew file+25
@@ -0,0 +1,25 @@
1+.envrc.private
2+.direnv
3+/backend/dist-newstyle/
4+/backend/call-graph.*
5+/dist-newstyle/
6+result
7+result-*
8+backup-*.sql
9+backend/logs
10+!/backend/**/.git
11+.tmp
12+/postgres-tmp/
13+/pg-tmp/
14+cabal.project.local
15+*.qcow2
16+known_hosts
17+/docs/causal/*.svg
18+.nixos-test-history'
19+pg-tmp
20+dist-newstyle
21+**/.nixos-test-history
22+host-map.json
23+personal-dev.sh
24+/node_modules/
25+/tsconfig.tsbuildinfo
.hlint.yamlnew file+1
@@ -0,0 +1 @@
1+backend/.hlint.yaml
.prettierrcnew file+3
@@ -0,0 +1,3 @@
1+{
2+ "trailingComma": "all"
3+}
.sops.yamlnew file+9
@@ -0,0 +1,9 @@
1+keys:
2+ # This is generated from nix/data/ssh-key-for-local-dev-secrets.pub:
3+ - &local-dev age1kekrcc4ulyy5xperfq03dpqqd6d56k43sp5jj4jq7jp4rqha5atsc979n4
4+
5+creation_rules:
6+ - path_regex: secrets/dev.yaml$
7+ key_groups:
8+ - age:
9+ - *local-dev
LICENSEnew file+28
@@ -0,0 +1,28 @@
1+Copyright 2026 garnix, Co.
2+
3+All rights reserved.
4+
5+Redistribution and use in source and binary forms, with or without
6+modification, are permitted provided that the following conditions are met:
7+
8+ * Redistributions of source code must retain the above copyright notice, this
9+ list of conditions and the following disclaimer.
10+
11+ * Redistributions in binary form must reproduce the above copyright notice,
12+ this list of conditions and the following disclaimer in the documentation
13+ and/or other materials provided with the distribution.
14+
15+ * Neither the name of the copyright holder nor the names of its contributors
16+ may be used to endorse or promote products derived from this software
17+ without specific prior written permission.
18+
19+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
20+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
21+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
22+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
23+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
24+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
25+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
26+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
27+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
28+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
README.mdnew file+58
@@ -0,0 +1,58 @@
1+# Garnix
2+
3+Garnix is a CI service for nixified, flake-based github repos.
4+
5+## Running Garnix locally in VMs
6+
7+You can spin up a couple of qemu VMs that provide a full Garnix deployment with:
8+
9+```bash
10+nix run -L .#examples_spinUpVms
11+```
12+
13+This will use [`nixos-compose`](https://github.com/garnix-io/nixos-compose).
14+If you run:
15+
16+```bash
17+nixos-compose tap
18+nixos-compose status
19+```
20+
21+You should then be able to point your browser to the ip address of the `exampleGarnixServer` to see the hosted ci.
22+
23+And there's an admin page on `/garnix-admin` that is useful for some development tasks.
24+
25+### Setting up a GitHub app
26+
27+You _will_ need a github app for Garnix to work, both for production and for testing.
28+On the `/garnix-admin` page you can create one by pressing the 'Submit to GitHub' button.
29+That will give you a bunch of credentials that you'll have to put into the `/secrets/dev.yaml` file by running
30+
31+```bash
32+sops edit secrets/dev.yaml
33+```
34+
35+Then you have to enable your new GitHub app on a repo that you want to build through the GitHub ui.
36+Finally, you can submit a test build, with something like this:
37+
38+```bash
39+curl -v \
40+ -XPOST \
41+ http://$(nixos-compose ip exampleGarnixServer)/api/build/submit \
42+ -H 'Content-Type: application/json' \
43+ -d '{ "owner": "garnix-io", "repo": "comment", "testCommit": "8b2b57d91dd1f4d094bb944a0a0ef65319a5663f" }'
44+```
45+
46+And then you can see the build under `/repo/garnix-io/comment`, for example.
47+
48+### Developing the frontend
49+
50+You can run the frontend in development mode against a backend in a VM like this:
51+
52+```bash
53+nixos-compose up -v
54+cd frontend
55+npm run dev
56+```
57+
58+Then point your browser to [localhost:3000](http://localhost:3000).
backend/.gitignorenew file+2
@@ -0,0 +1,2 @@
1+test/spec/garage.log
2+.hspec
backend/.golden/.gitignorenew file+1
@@ -0,0 +1 @@
1+/*/*/actual
backend/.golden/ConfigSchemaSpec/garnix-config-schema.json/goldennew file+215
@@ -0,0 +1,215 @@
1+{
2+ "anyOf": [
3+ {
4+ "$comment": "config",
5+ "properties": {
6+ "actions": {
7+ "$comment": "Specifies which actions to run.",
8+ "items": {
9+ "$comment": "actions",
10+ "properties": {
11+ "on": {
12+ "$comment": "Event that triggers this action",
13+ "const": "push"
14+ },
15+ "run": {
16+ "$comment": "Name of the nix app to run as an action.",
17+ "type": "string"
18+ },
19+ "sandboxType": {
20+ "$comment": "What sandbox type. If you want to use the 'SharedResources' type, get in touch with us.",
21+ "oneOf": [
22+ {
23+ "const": "fast-startup"
24+ },
25+ {
26+ "const": "shared-resources"
27+ }
28+ ]
29+ },
30+ "withRepoContents": {
31+ "$comment": "Whether the action should run with access to the entire repo. If false (default), only the closure of the action is available.",
32+ "type": "boolean"
33+ }
34+ },
35+ "required": [
36+ "on",
37+ "run"
38+ ],
39+ "type": "object"
40+ },
41+ "type": "array"
42+ },
43+ "builds": {
44+ "$comment": "Specifies what should be built. Everything in the `include` section, minus everything in the `exclude` section, is built.",
45+ "oneOf": [
46+ {
47+ "$comment": "builds",
48+ "properties": {
49+ "branch": {
50+ "$comment": "What (optional) branch this build section is enabled for.",
51+ "type": "string"
52+ },
53+ "exclude": {
54+ "$comment": "What builds to exclude. This is a list of *attribute matchers*, of the form `x.y.z` or `x.y`. For example, `packages.x86_64-linux.*`, or `*.*`. Two-place matchers only match two-place matchers, and three-place matchers only match three-place matchers. '*' is the wildcard. This is applied *after* the 'include'. Thus, if something matches both the 'include' and the 'exclude', it will be excluded.",
55+ "items": {
56+ "type": "string"
57+ },
58+ "type": "array"
59+ },
60+ "include": {
61+ "$comment": "What builds to include. This is a list of *attribute matchers*, of the form `x.y.z` or `x.y`. For example, `packages.x86_64-linux.*`, or `*.*`. Two-place matchers only match two-place matchers, and three-place matchers only match three-place matchers. '*' is the wildcard.",
62+ "items": {
63+ "type": "string"
64+ },
65+ "type": "array"
66+ }
67+ },
68+ "type": "object"
69+ },
70+ {
71+ "items": {
72+ "$comment": "builds",
73+ "properties": {
74+ "branch": {
75+ "$comment": "What (optional) branch this build section is enabled for.",
76+ "type": "string"
77+ },
78+ "exclude": {
79+ "$comment": "What builds to exclude. This is a list of *attribute matchers*, of the form `x.y.z` or `x.y`. For example, `packages.x86_64-linux.*`, or `*.*`. Two-place matchers only match two-place matchers, and three-place matchers only match three-place matchers. '*' is the wildcard. This is applied *after* the 'include'. Thus, if something matches both the 'include' and the 'exclude', it will be excluded.",
80+ "items": {
81+ "type": "string"
82+ },
83+ "type": "array"
84+ },
85+ "include": {
86+ "$comment": "What builds to include. This is a list of *attribute matchers*, of the form `x.y.z` or `x.y`. For example, `packages.x86_64-linux.*`, or `*.*`. Two-place matchers only match two-place matchers, and three-place matchers only match three-place matchers. '*' is the wildcard.",
87+ "items": {
88+ "type": "string"
89+ },
90+ "type": "array"
91+ }
92+ },
93+ "type": "object"
94+ },
95+ "type": "array"
96+ }
97+ ]
98+ },
99+ "flakeDir": {
100+ "$comment": "The directory containing your flake.nix relative from the repo root (if not in the repo root).",
101+ "type": "string"
102+ },
103+ "fodChecks": {
104+ "$comment": "Whether FOD checks are enabled for the repo. See https://garnix.io/docs/fod-checks for more information.",
105+ "type": "boolean"
106+ },
107+ "incrementalizeBuilds": {
108+ "$comment": "Whether to override the `garnix-incrementalize` flake input to point to an parent built commit. This allows incremental builds. See our https://garnix.io/docs for more information.",
109+ "oneOf": [
110+ {
111+ "type": "boolean"
112+ },
113+ {
114+ "$comment": "ExcludesBranches",
115+ "properties": {
116+ "excludeBranches": {
117+ "$comment": "What branches *not* to incrementalize",
118+ "items": {
119+ "type": "string"
120+ },
121+ "type": "array"
122+ }
123+ },
124+ "required": [
125+ "excludeBranches"
126+ ],
127+ "type": "object"
128+ }
129+ ]
130+ },
131+ "modules": {
132+ "$comment": "Specifies which actions to run.\nmodules",
133+ "properties": {
134+ "publish": {
135+ "$comment": "Whether to publish modules from this repository.",
136+ "type": "boolean"
137+ }
138+ },
139+ "type": "object"
140+ },
141+ "servers": {
142+ "$comment": "Specifies what servers to deploy.",
143+ "items": {
144+ "$comment": "servers",
145+ "properties": {
146+ "configuration": {
147+ "$comment": "What attribute to deploy (e.g.: 'myServer' for 'nixosConfigurations.myServer')",
148+ "type": "string"
149+ },
150+ "deployment": {
151+ "$comment": "When to deploy a new server, or redeploy an existing one\ndeployment",
152+ "oneOf": [
153+ {
154+ "allOf": [
155+ {
156+ "type": "object"
157+ },
158+ {
159+ "properties": {
160+ "type": {
161+ "const": "on-pull-request"
162+ }
163+ },
164+ "required": [
165+ "type"
166+ ],
167+ "type": "object"
168+ }
169+ ],
170+ "type": "object"
171+ },
172+ {
173+ "properties": {
174+ "branch": {
175+ "$comment": "What git branch to deploy from",
176+ "type": "string"
177+ },
178+ "isPrimary": {
179+ "$comment": "If this deploy should also be reachable at «repo-name».«org-name».garnix.me",
180+ "type": "boolean"
181+ },
182+ "machine": {
183+ "$comment": "What server tier to deploy",
184+ "type": "string"
185+ },
186+ "type": {
187+ "const": "on-branch"
188+ }
189+ },
190+ "required": [
191+ "type",
192+ "branch"
193+ ],
194+ "type": "object"
195+ }
196+ ],
197+ "type": "object"
198+ }
199+ },
200+ "required": [
201+ "deployment",
202+ "configuration"
203+ ],
204+ "type": "object"
205+ },
206+ "type": "array"
207+ }
208+ },
209+ "type": "object"
210+ },
211+ {
212+ "type": "null"
213+ }
214+ ]
215+}
backend/.golden/FeatureFlagConfigDbo.jsonnew file+528
@@ -0,0 +1,528 @@
1+{
2+ "samples": [
3+ [
4+ [
5+ "BwrapBuildSandbox",
6+ {
7+ "contents": -1,
8+ "tag": "Percentage"
9+ }
10+ ],
11+ [
12+ "FodChecks",
13+ {
14+ "tag": "DisableFeature"
15+ }
16+ ],
17+ [
18+ "BwrapBuildSandbox",
19+ {
20+ "tag": "DisableFeature"
21+ }
22+ ],
23+ [
24+ "FodChecks",
25+ {
26+ "contents": 28,
27+ "tag": "Percentage"
28+ }
29+ ],
30+ [
31+ "FodChecks",
32+ {
33+ "tag": "DisableFeature"
34+ }
35+ ],
36+ [
37+ "BwrapBuildSandbox",
38+ {
39+ "tag": "DisableFeature"
40+ }
41+ ],
42+ [
43+ "BwrapBuildSandbox",
44+ {
45+ "contents": 18,
46+ "tag": "Percentage"
47+ }
48+ ],
49+ [
50+ "FodChecks",
51+ {
52+ "tag": "DisableFeature"
53+ }
54+ ],
55+ [
56+ "BwrapBuildSandbox",
57+ {
58+ "tag": "DisableFeature"
59+ }
60+ ],
61+ [
62+ "BwrapBuildSandbox",
63+ {
64+ "tag": "DisableFeature"
65+ }
66+ ],
67+ [
68+ "BwrapBuildSandbox",
69+ {
70+ "contents": 12,
71+ "tag": "Percentage"
72+ }
73+ ],
74+ [
75+ "BwrapBuildSandbox",
76+ {
77+ "contents": 23,
78+ "tag": "Percentage"
79+ }
80+ ],
81+ [
82+ "BwrapBuildSandbox",
83+ {
84+ "tag": "DisableFeature"
85+ }
86+ ],
87+ [
88+ "BwrapBuildSandbox",
89+ {
90+ "contents": -16,
91+ "tag": "Percentage"
92+ }
93+ ],
94+ [
95+ "FodChecks",
96+ {
97+ "tag": "DisableFeature"
98+ }
99+ ],
100+ [
101+ "BwrapBuildSandbox",
102+ {
103+ "contents": -2,
104+ "tag": "Percentage"
105+ }
106+ ],
107+ [
108+ "BwrapBuildSandbox",
109+ {
110+ "contents": -8,
111+ "tag": "Percentage"
112+ }
113+ ],
114+ [
115+ "BwrapBuildSandbox",
116+ {
117+ "tag": "DisableFeature"
118+ }
119+ ],
120+ [
121+ "FodChecks",
122+ {
123+ "contents": -27,
124+ "tag": "Percentage"
125+ }
126+ ],
127+ [
128+ "FodChecks",
129+ {
130+ "tag": "DisableFeature"
131+ }
132+ ],
133+ [
134+ "BwrapBuildSandbox",
135+ {
136+ "contents": -13,
137+ "tag": "Percentage"
138+ }
139+ ],
140+ [
141+ "BwrapBuildSandbox",
142+ {
143+ "tag": "DisableFeature"
144+ }
145+ ],
146+ [
147+ "BwrapBuildSandbox",
148+ {
149+ "contents": 12,
150+ "tag": "Percentage"
151+ }
152+ ],
153+ [
154+ "FodChecks",
155+ {
156+ "contents": -3,
157+ "tag": "Percentage"
158+ }
159+ ],
160+ [
161+ "BwrapBuildSandbox",
162+ {
163+ "tag": "DisableFeature"
164+ }
165+ ],
166+ [
167+ "BwrapBuildSandbox",
168+ {
169+ "tag": "DisableFeature"
170+ }
171+ ],
172+ [
173+ "BwrapBuildSandbox",
174+ {
175+ "contents": 12,
176+ "tag": "Percentage"
177+ }
178+ ],
179+ [
180+ "FodChecks",
181+ {
182+ "contents": 27,
183+ "tag": "Percentage"
184+ }
185+ ]
186+ ],
187+ [
188+ [
189+ "FodChecks",
190+ {
191+ "tag": "DisableFeature"
192+ }
193+ ],
194+ [
195+ "FodChecks",
196+ {
197+ "tag": "DisableFeature"
198+ }
199+ ],
200+ [
201+ "FodChecks",
202+ {
203+ "tag": "DisableFeature"
204+ }
205+ ],
206+ [
207+ "FodChecks",
208+ {
209+ "tag": "DisableFeature"
210+ }
211+ ],
212+ [
213+ "BwrapBuildSandbox",
214+ {
215+ "tag": "DisableFeature"
216+ }
217+ ],
218+ [
219+ "FodChecks",
220+ {
221+ "tag": "DisableFeature"
222+ }
223+ ]
224+ ],
225+ [
226+ [
227+ "BwrapBuildSandbox",
228+ {
229+ "tag": "DisableFeature"
230+ }
231+ ],
232+ [
233+ "BwrapBuildSandbox",
234+ {
235+ "tag": "DisableFeature"
236+ }
237+ ],
238+ [
239+ "FodChecks",
240+ {
241+ "contents": 11,
242+ "tag": "Percentage"
243+ }
244+ ],
245+ [
246+ "BwrapBuildSandbox",
247+ {
248+ "contents": -4,
249+ "tag": "Percentage"
250+ }
251+ ],
252+ [
253+ "FodChecks",
254+ {
255+ "tag": "DisableFeature"
256+ }
257+ ],
258+ [
259+ "BwrapBuildSandbox",
260+ {
261+ "tag": "DisableFeature"
262+ }
263+ ],
264+ [
265+ "FodChecks",
266+ {
267+ "contents": 6,
268+ "tag": "Percentage"
269+ }
270+ ],
271+ [
272+ "BwrapBuildSandbox",
273+ {
274+ "tag": "DisableFeature"
275+ }
276+ ],
277+ [
278+ "BwrapBuildSandbox",
279+ {
280+ "tag": "DisableFeature"
281+ }
282+ ],
283+ [
284+ "FodChecks",
285+ {
286+ "contents": 19,
287+ "tag": "Percentage"
288+ }
289+ ],
290+ [
291+ "FodChecks",
292+ {
293+ "tag": "DisableFeature"
294+ }
295+ ],
296+ [
297+ "FodChecks",
298+ {
299+ "tag": "DisableFeature"
300+ }
301+ ],
302+ [
303+ "FodChecks",
304+ {
305+ "contents": -18,
306+ "tag": "Percentage"
307+ }
308+ ],
309+ [
310+ "FodChecks",
311+ {
312+ "tag": "DisableFeature"
313+ }
314+ ],
315+ [
316+ "FodChecks",
317+ {
318+ "tag": "DisableFeature"
319+ }
320+ ],
321+ [
322+ "FodChecks",
323+ {
324+ "contents": 28,
325+ "tag": "Percentage"
326+ }
327+ ],
328+ [
329+ "FodChecks",
330+ {
331+ "contents": -6,
332+ "tag": "Percentage"
333+ }
334+ ],
335+ [
336+ "FodChecks",
337+ {
338+ "contents": -18,
339+ "tag": "Percentage"
340+ }
341+ ],
342+ [
343+ "BwrapBuildSandbox",
344+ {
345+ "contents": -9,
346+ "tag": "Percentage"
347+ }
348+ ]
349+ ],
350+ [
351+ [
352+ "FodChecks",
353+ {
354+ "tag": "DisableFeature"
355+ }
356+ ],
357+ [
358+ "BwrapBuildSandbox",
359+ {
360+ "contents": 20,
361+ "tag": "Percentage"
362+ }
363+ ],
364+ [
365+ "FodChecks",
366+ {
367+ "contents": -26,
368+ "tag": "Percentage"
369+ }
370+ ],
371+ [
372+ "BwrapBuildSandbox",
373+ {
374+ "contents": -14,
375+ "tag": "Percentage"
376+ }
377+ ],
378+ [
379+ "BwrapBuildSandbox",
380+ {
381+ "contents": -30,
382+ "tag": "Percentage"
383+ }
384+ ],
385+ [
386+ "FodChecks",
387+ {
388+ "contents": -26,
389+ "tag": "Percentage"
390+ }
391+ ],
392+ [
393+ "FodChecks",
394+ {
395+ "contents": -27,
396+ "tag": "Percentage"
397+ }
398+ ],
399+ [
400+ "BwrapBuildSandbox",
401+ {
402+ "tag": "DisableFeature"
403+ }
404+ ],
405+ [
406+ "BwrapBuildSandbox",
407+ {
408+ "contents": -1,
409+ "tag": "Percentage"
410+ }
411+ ]
412+ ],
413+ [
414+ [
415+ "BwrapBuildSandbox",
416+ {
417+ "contents": 27,
418+ "tag": "Percentage"
419+ }
420+ ],
421+ [
422+ "BwrapBuildSandbox",
423+ {
424+ "tag": "DisableFeature"
425+ }
426+ ],
427+ [
428+ "FodChecks",
429+ {
430+ "contents": 16,
431+ "tag": "Percentage"
432+ }
433+ ],
434+ [
435+ "BwrapBuildSandbox",
436+ {
437+ "tag": "DisableFeature"
438+ }
439+ ],
440+ [
441+ "FodChecks",
442+ {
443+ "tag": "DisableFeature"
444+ }
445+ ],
446+ [
447+ "FodChecks",
448+ {
449+ "contents": -27,
450+ "tag": "Percentage"
451+ }
452+ ],
453+ [
454+ "BwrapBuildSandbox",
455+ {
456+ "tag": "DisableFeature"
457+ }
458+ ],
459+ [
460+ "BwrapBuildSandbox",
461+ {
462+ "contents": 7,
463+ "tag": "Percentage"
464+ }
465+ ],
466+ [
467+ "BwrapBuildSandbox",
468+ {
469+ "contents": 25,
470+ "tag": "Percentage"
471+ }
472+ ],
473+ [
474+ "BwrapBuildSandbox",
475+ {
476+ "contents": 5,
477+ "tag": "Percentage"
478+ }
479+ ],
480+ [
481+ "BwrapBuildSandbox",
482+ {
483+ "tag": "DisableFeature"
484+ }
485+ ],
486+ [
487+ "BwrapBuildSandbox",
488+ {
489+ "tag": "DisableFeature"
490+ }
491+ ],
492+ [
493+ "FodChecks",
494+ {
495+ "contents": -18,
496+ "tag": "Percentage"
497+ }
498+ ],
499+ [
500+ "FodChecks",
501+ {
502+ "tag": "DisableFeature"
503+ }
504+ ],
505+ [
506+ "FodChecks",
507+ {
508+ "contents": 22,
509+ "tag": "Percentage"
510+ }
511+ ],
512+ [
513+ "FodChecks",
514+ {
515+ "tag": "DisableFeature"
516+ }
517+ ],
518+ [
519+ "FodChecks",
520+ {
521+ "contents": 3,
522+ "tag": "Percentage"
523+ }
524+ ]
525+ ]
526+ ],
527+ "seed": 569535987
528+}
backend/.golden/HostSpec/adds-hosts-when-a-new-server-is-deployed/goldennew file+33
@@ -0,0 +1,33 @@
1+{
2+ "http": {
3+ "middlewares": {
4+ "heartbeatmiddleware": {
5+ "plugin": {
6+ "heartbeatmiddleware": {
7+ "reportEndpoint": "https://garnix.io/api/hosts/heartbeat"
8+ }
9+ }
10+ }
11+ },
12+ "routers": {
13+ "package.branch.repo.user": {
14+ "middlewares": [
15+ "heartbeatmiddleware"
16+ ],
17+ "rule": "Host(`package.branch.repo.user.garnix.me`)",
18+ "service": "package.branch.repo.user"
19+ }
20+ },
21+ "services": {
22+ "package.branch.repo.user": {
23+ "loadBalancer": {
24+ "servers": [
25+ {
26+ "url": "http://11.111.111.1"
27+ }
28+ ]
29+ }
30+ }
31+ }
32+ }
33+}
backend/.golden/HostSpec/reflects-the-latest-deployed-versions/goldennew file+33
@@ -0,0 +1,33 @@
1+{
2+ "http": {
3+ "middlewares": {
4+ "heartbeatmiddleware": {
5+ "plugin": {
6+ "heartbeatmiddleware": {
7+ "reportEndpoint": "https://garnix.io/api/hosts/heartbeat"
8+ }
9+ }
10+ }
11+ },
12+ "routers": {
13+ "package.branch.repo.user": {
14+ "middlewares": [
15+ "heartbeatmiddleware"
16+ ],
17+ "rule": "Host(`package.branch.repo.user.garnix.me`)",
18+ "service": "package.branch.repo.user"
19+ }
20+ },
21+ "services": {
22+ "package.branch.repo.user": {
23+ "loadBalancer": {
24+ "servers": [
25+ {
26+ "url": "http://11.111.111.2"
27+ }
28+ ]
29+ }
30+ }
31+ }
32+ }
33+}
backend/.golden/HostSpec/serves-configurations-for-pull-request-servers/goldennew file+33
@@ -0,0 +1,33 @@
1+{
2+ "http": {
3+ "middlewares": {
4+ "heartbeatmiddleware": {
5+ "plugin": {
6+ "heartbeatmiddleware": {
7+ "reportEndpoint": "https://garnix.io/api/hosts/heartbeat"
8+ }
9+ }
10+ }
11+ },
12+ "routers": {
13+ "nginx.pull-42.repo.owner": {
14+ "middlewares": [
15+ "heartbeatmiddleware"
16+ ],
17+ "rule": "Host(`nginx.pull-42.repo.owner.garnix.me`)",
18+ "service": "nginx.pull-42.repo.owner"
19+ }
20+ },
21+ "services": {
22+ "nginx.pull-42.repo.owner": {
23+ "loadBalancer": {
24+ "servers": [
25+ {
26+ "url": "http://11.111.111.1"
27+ }
28+ ]
29+ }
30+ }
31+ }
32+ }
33+}
backend/.golden/HostSpec/serves-pull-request-configurations-when-the-branch-name-is-not-a-valid-subdomain/goldennew file+33
@@ -0,0 +1,33 @@
1+{
2+ "http": {
3+ "middlewares": {
4+ "heartbeatmiddleware": {
5+ "plugin": {
6+ "heartbeatmiddleware": {
7+ "reportEndpoint": "https://garnix.io/api/hosts/heartbeat"
8+ }
9+ }
10+ }
11+ },
12+ "routers": {
13+ "nginx.pull-42.repo.owner": {
14+ "middlewares": [
15+ "heartbeatmiddleware"
16+ ],
17+ "rule": "Host(`nginx.pull-42.repo.owner.garnix.me`)",
18+ "service": "nginx.pull-42.repo.owner"
19+ }
20+ },
21+ "services": {
22+ "nginx.pull-42.repo.owner": {
23+ "loadBalancer": {
24+ "servers": [
25+ {
26+ "url": "http://11.111.111.1"
27+ }
28+ ]
29+ }
30+ }
31+ }
32+ }
33+}
backend/.golden/HostSpec/supports-exposing-more-than-one-configuration-per-repo/goldennew file+65
@@ -0,0 +1,65 @@
1+{
2+ "http": {
3+ "middlewares": {
4+ "heartbeatmiddleware": {
5+ "plugin": {
6+ "heartbeatmiddleware": {
7+ "reportEndpoint": "https://garnix.io/api/hosts/heartbeat"
8+ }
9+ }
10+ }
11+ },
12+ "routers": {
13+ "nginx.main.repo.owner": {
14+ "middlewares": [
15+ "heartbeatmiddleware"
16+ ],
17+ "rule": "Host(`nginx.main.repo.owner.garnix.me`)",
18+ "service": "nginx.main.repo.owner"
19+ },
20+ "psql.feat.repo.owner": {
21+ "middlewares": [
22+ "heartbeatmiddleware"
23+ ],
24+ "rule": "Host(`psql.feat.repo.owner.garnix.me`)",
25+ "service": "psql.feat.repo.owner"
26+ },
27+ "psql.main.repo.owner": {
28+ "middlewares": [
29+ "heartbeatmiddleware"
30+ ],
31+ "rule": "Host(`psql.main.repo.owner.garnix.me`)",
32+ "service": "psql.main.repo.owner"
33+ }
34+ },
35+ "services": {
36+ "nginx.main.repo.owner": {
37+ "loadBalancer": {
38+ "servers": [
39+ {
40+ "url": "http://11.111.111.1"
41+ }
42+ ]
43+ }
44+ },
45+ "psql.feat.repo.owner": {
46+ "loadBalancer": {
47+ "servers": [
48+ {
49+ "url": "http://11.111.111.3"
50+ }
51+ ]
52+ }
53+ },
54+ "psql.main.repo.owner": {
55+ "loadBalancer": {
56+ "servers": [
57+ {
58+ "url": "http://11.111.111.2"
59+ }
60+ ]
61+ }
62+ }
63+ }
64+ }
65+}
backend/.hlint.yamlnew file+17
@@ -0,0 +1,17 @@
1+- arguments: [-XQuasiQuotes]
2+- ignore: { name: Avoid lambda }
3+- ignore: { name: Eta reduce }
4+- ignore: { name: Functor law }
5+- ignore: { name: Redundant <$> }
6+- ignore: { name: Redundant id }
7+- ignore: { name: Replace case with maybe }
8+- ignore: { name: Use <&> }
9+- ignore: { name: Use >=> }
10+- ignore: { name: Use <$> }
11+- ignore: { name: Use for_ }
12+- ignore: { name: Use forM_ }
13+- ignore: { name: Use list comprehension }
14+- ignore: { name: Use mapMaybe }
15+- ignore: { name: Use newtype instead of data }
16+- ignore: { name: Use unless }
17+- ignore: { name: Use withFile }
backend/cabal.projectnew file+2
@@ -0,0 +1,2 @@
1+packages: garnix.cabal
2+allow-newer: all:ghc-prim, all:base
backend/data/badge_favicon.svgnew file+4
@@ -0,0 +1,4 @@
1+<svg xmlns="http://www.w3.org/2000/svg" width="96pt" height="96pt" viewBox="0 0 34 34" fill="white">
2+ <path d="M33 1H1V33H33V1Z" fill="white" stroke="black" stroke-width="2" stroke-miterlimit="10" />
3+ <path d="M32.9937 16.9971H17.0032V24.7824" stroke="black" stroke-width="2" stroke-miterlimit="10" />
4+</svg>
backend/default.nixnew file+242
@@ -0,0 +1,242 @@
1+{ pkgs
2+, lib
3+, system
4+, flakeInputs
5+, ...
6+}:
7+let
8+ secretSetup = ''
9+ SOPS_AGE_KEY=$(${lib.getExe pkgs.ssh-to-age} -private-key -i ${../nix/data/ssh-key-for-local-dev-secrets})
10+ export SOPS_AGE_KEY
11+ '';
12+ dbSetup = ''
13+ if test -z "$DB_DIR"; then
14+ echo Set \$DB_DIR before 'dbSetup'
15+ exit 1
16+ fi
17+ mkdir -p "$DB_DIR"
18+ export PGDATA=$DB_DIR/test
19+ export PGHOST=$DB_DIR/test
20+ export PGPORT=9178
21+ export PGUSER=garnix
22+ export PGPASSWORD=garnix
23+ export PGDATABASE=garnix
24+ # For postgresql-typed
25+ export TPG_HOST=$PGHOST
26+ export TPG_DB=$PGDATABASE
27+ export TPG_USER=$PGUSER
28+ export TPG_PASS=$PGPASSWORD
29+ export TPG_SOCK=$PGHOST"/.s.PGSQL."$PGPORT
30+ '';
31+ garnixRuntimeDependencies = [
32+ pkgs.util-linux
33+ pkgs.git
34+ pkgs.nix
35+ pkgs.openssh
36+ pkgs.coreutils-full
37+ pkgs.bubblewrap
38+ pkgs.nettools
39+ flakeInputs.comment.packages.${system}.default
40+ pkgs.bzip2
41+ pkgs.age
42+ pkgs.xz
43+ pkgs.rsync
44+ ];
45+ migrate = pkgs.callPackage ../nix/packages/migrate.nix { };
46+ postgres = pkgs.postgresql_18;
47+ db = pkgs.callPackage ../nix/packages/db.nix {
48+ inherit migrate postgres;
49+ };
50+ garnixTestDependencies = [
51+ db
52+ pkgs.ps
53+ pkgs.which
54+ pkgs.bash
55+ pkgs.sops
56+ pkgs.mercurial
57+ pkgs.curl
58+ pkgs.garage
59+ pkgs.psmisc # for `fuser` in specs
60+ ];
61+ garnixDevDependencies = [
62+ (pkgs.haskell-language-server.override { supportedGhcVersions = [ "967" ]; })
63+ (pkgs.haskellPackages.ghc.withPackages (p: p.garnix.getBuildInputs.haskellBuildInputs))
64+ pkgs.ghcid
65+ pkgs.haskellPackages.cabal-install
66+ pkgs.haskellPackages.cabal2nix
67+ pkgs.haskellPackages.hlint
68+ pkgs.haskellPackages.hpack
69+ pkgs.haskellPackages.ormolu
70+ ];
71+in
72+rec {
73+ packages = {
74+ inherit postgres migrate;
75+ garnix =
76+ pkgs.runCommand "garnix"
77+ {
78+ nativeBuildInputs = [ pkgs.makeWrapper ];
79+ }
80+ ''
81+ mkdir -p $out/bin
82+ cp ${packages.garnixHaskellPackage}/bin/server $out/bin
83+ wrapProgram "$out/bin/server" \
84+ --set EMPTY_DIR ${../nix/data/emptyDir} \
85+ --prefix PATH : ${pkgs.lib.makeBinPath garnixRuntimeDependencies}
86+ '';
87+ garnixHaskellPackage =
88+ let
89+ src = with lib.fileset;
90+ toSource {
91+ root = ./..;
92+ fileset =
93+ difference
94+ (unions [
95+ ./.
96+ ../flake.lock
97+ ])
98+ (unions [
99+ ./default.nix
100+ ./nixos-module.nix
101+ ]);
102+ };
103+ in
104+ pkgs.haskell.lib.overrideCabal
105+ pkgs.haskellPackages.garnix
106+ (old: {
107+ buildDepends = (old.buildDepends or [ ]) ++ [ db ];
108+ inherit src;
109+ prePatch = ''
110+ cd backend/
111+ '';
112+ preBuild = ''
113+ ${old.preBuild or ""}
114+ export HOME=$(pwd)
115+ DB_DIR=$(pwd)/pg-tmp
116+ ${dbSetup}
117+ export EMPTY_DIR=${../nix/data/emptyDir}
118+ db new
119+ trap 'db clear' EXIT
120+ '';
121+ doCheck = false;
122+ });
123+ moduleGraph = pkgs.runCommand "moduleGraph.pdf"
124+ {
125+ buildInputs = [ pkgs.haskellPackages.graphmod pkgs.graphviz ];
126+ }
127+ ''
128+ cp -r ${./.} backend
129+ graphmod ./backend/exe/Server.hs -i./backend/src \
130+ | tred \
131+ | dot -Tpdf > $out
132+ '';
133+ };
134+ checks = {
135+ hlint = pkgs.runCommand "hlint" { buildInputs = [ pkgs.haskell.packages.ghc967.hlint ]; } ''
136+ cd ${./.}
137+ hlint src test
138+ touch $out
139+ '';
140+ check-qualified-imports =
141+ let
142+ modulesMustBeQualified = [
143+ "Garnix.DB"
144+ "Garnix.Monad.SubProcess.Deprecated"
145+ ];
146+ checkNoUnqualifiedImports = module: ''
147+ echo 'checking for unqualified `${module}`...'
148+ FOUND=$(grep -r 'import ${module}[^\.]' | grep -v 'qualified' || true)
149+ if [ -n "$FOUND" ]; then
150+ echo -e "${module} should always be qualified. Found unqualified imports:\n$FOUND"
151+ exit 1
152+ fi
153+ '';
154+ in
155+ pkgs.runCommand "check-qualified-imports"
156+ {
157+ buildInputs = [ pkgs.ack ];
158+ } ''
159+ cd ${./.}
160+ ${lib.concatLines (lib.map checkNoUnqualifiedImports modulesMustBeQualified)}
161+ touch $out
162+ '';
163+ };
164+ shellHook = ''
165+ DB_DIR=$(git rev-parse --show-toplevel)/pg-tmp
166+ ${dbSetup}
167+ ${secretSetup}
168+ export EMPTY_DIR=${../nix/data/emptyDir}
169+ '';
170+ devShellInputs = [
171+ packages.postgres
172+ pkgs.sqitchPg
173+ ]
174+ ++ garnixRuntimeDependencies
175+ ++ garnixTestDependencies
176+ ++ garnixDevDependencies;
177+ nixosModule = import ./nixos-module.nix;
178+ commands = {
179+ convertHashids = pkgs.writeShellApplication {
180+ meta.description = "converts mangled hashids to db ids";
181+ name = "convertHashids";
182+ text = ''
183+ cd backend
184+ cabal run convert-hashids -- "$@"
185+ '';
186+ };
187+
188+ readModuleSchema = pkgs.writeShellApplication {
189+ meta.description = "reads a garnix module schema and emits sql";
190+ name = "readModuleSchema";
191+ text = ''
192+ cd backend
193+ cabal build 1>&2
194+ withSecrets cabal run read-module-schema -- "$@"
195+ '';
196+ };
197+
198+ specs =
199+ pkgs.writeShellApplication {
200+ meta.description = "runs the backend test suite";
201+ name = "backend-specs";
202+ runtimeInputs =
203+ (
204+ garnixRuntimeDependencies ++
205+ garnixTestDependencies ++
206+ [
207+ (pkgs.haskellPackages.ghc.withPackages (p: p.garnix.getBuildInputs.haskellBuildInputs))
208+ pkgs.haskellPackages.cabal-install
209+ pkgs.yaml2json
210+ pkgs.jq
211+ ]
212+ );
213+ text = ''
214+ tempDir=$(mktemp -d /tmp/garnix-specs.XXXXXXXX)
215+ cd "$tempDir"
216+ export HOME="$tempDir/home"
217+ mkdir "$HOME"
218+ DB_DIR="$tempDir/pg-tmp"
219+ ${dbSetup}
220+ db new
221+ trap 'db clear; rm $tempDir -rf' EXIT
222+
223+ export EMPTY_DIR=${../nix/data/emptyDir}
224+ ${secretSetup}
225+
226+ git config --global user.email "you@example.com"
227+ git config --global user.name "Your Name"
228+ git config --global init.defaultBranch main
229+ WATCHDOG_GITHUB_ACCESS_TOKEN=$(sops --decrypt ${./../secrets/dev.yaml} | yaml2json | jq -r .watchdog_github_access_token)
230+ mkdir -p ~/.config/nix
231+ echo access-tokens = github.com="$WATCHDOG_GITHUB_ACCESS_TOKEN" > ~/.config/nix/nix.conf
232+
233+ cp -r ${./..} src
234+ chmod a+rwX -R src
235+ chmod go-rwx src/backend/ssh-key-for-tests
236+ cd src/backend
237+ cabal configure --ghc-options="-O0"
238+ cabal run spec -- --skip @skip-ci --fail-on=focused
239+ '';
240+ };
241+ };
242+}
backend/dev-action-runner-ssh-keynew file+7
@@ -0,0 +1,7 @@
1+-----BEGIN OPENSSH PRIVATE KEY-----
2+b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW
3+QyNTUxOQAAACCLE2AI+5QT6Qx0lwOJM9O/j276hJSYTxuWy28WI9ueWQAAAJDYrQNF2K0D
4+RQAAAAtzc2gtZWQyNTUxOQAAACCLE2AI+5QT6Qx0lwOJM9O/j276hJSYTxuWy28WI9ueWQ
5+AAAEBOPXXwJKFa4FjGZmjdlElTvhYP1gRqARCvOB0zRpWDAosTYAj7lBPpDHSXA4kz07+P
6+bvqElJhPG5bLbxYj255ZAAAAC2FsZXhAZ2Fybml4AQI=
7+-----END OPENSSH PRIVATE KEY-----
backend/dev-key.jwtnew file

Binary file not shown.

backend/exe/GenerateYamlConfigDocs.hsnew file+10
@@ -0,0 +1,10 @@
1+module Main (main) where
2+
3+import Autodocodec.Yaml
4+import Data.Text.IO qualified as T
5+import Garnix.Prelude
6+import Garnix.YamlConfig
7+
8+main :: IO ()
9+main = do
10+ T.putStr $ renderPlainSchemaViaCodec @GarnixConfig
backend/exe/Server.hsnew file+7
@@ -0,0 +1,7 @@
1+module Main (main) where
2+
3+import Garnix
4+import Garnix.Prelude
5+
6+main :: IO ()
7+main = run
backend/exe/convertHashids.hsnew file+18
@@ -0,0 +1,18 @@
1+import Data.Text.IO qualified as T
2+import Garnix.Prelude
3+import System.Environment (getArgs)
4+import Text.Read (readMaybe)
5+
6+main :: IO ()
7+main = do
8+ args <- fmap cs <$> getArgs
9+ forM_ args $ \arg -> do
10+ case cs arg ^? hashIdText of
11+ Just hashId -> output hashId
12+ Nothing -> case readMaybe arg of
13+ Just int -> output (int ^. re hashIdInt)
14+ Nothing -> error $ "cannot parse " <> cs arg
15+
16+output :: HashId -> IO ()
17+output hashId =
18+ T.putStrLn $ (hashId ^. re hashIdText) <> " - " <> show (hashId ^. hashIdInt)
backend/garnix.cabalnew file+1047
@@ -0,0 +1,1047 @@
1+cabal-version: 1.12
2+
3+-- This file has been generated from package.yaml by hpack version 0.38.3.
4+--
5+-- see: https://github.com/sol/hpack
6+
7+name: garnix
8+version: 0.1.0.0
9+homepage: https://github.com/jkarni/garnix#readme
10+bug-reports: https://github.com/jkarni/garnix/issues
11+author: Julian K. Arni
12+maintainer: jkarni@riseup.net
13+copyright: AllRightsReserved
14+license: BSD3
15+build-type: Simple
16+extra-source-files:
17+ test/spec/data/create-server-response.json
18+ test/spec/data/get-servers-response.json
19+ test/spec/data/emptyDir/.gitignore
20+
21+source-repository head
22+ type: git
23+ location: https://github.com/jkarni/garnix
24+
25+library
26+ exposed-modules:
27+ Garnix
28+ Garnix.Access
29+ Garnix.AccessToken
30+ Garnix.AccessToken.Types
31+ Garnix.API
32+ Garnix.API.Account
33+ Garnix.API.Auth
34+ Garnix.API.Badges
35+ Garnix.API.Builds
36+ Garnix.API.Builds.Types
37+ Garnix.API.Cache
38+ Garnix.API.Cache.Auth
39+ Garnix.API.Cache.Permissions
40+ Garnix.API.Cache.Types
41+ Garnix.API.Commits
42+ Garnix.API.ConfigSchema
43+ Garnix.API.Dev
44+ Garnix.API.GhWebhooks
45+ Garnix.API.Health
46+ Garnix.API.Hosts
47+ Garnix.API.Keys
48+ Garnix.API.Modules
49+ Garnix.API.Runs
50+ Garnix.API.Stripe
51+ Garnix.Async
52+ Garnix.Attribute
53+ Garnix.Build
54+ Garnix.Build.Action
55+ Garnix.Build.Checkout
56+ Garnix.Build.Evaluation
57+ Garnix.Build.Flake
58+ Garnix.Build.FodCheck
59+ Garnix.Build.Helpers
60+ Garnix.Build.MetaCheck
61+ Garnix.Build.Module
62+ Garnix.Build.Package
63+ Garnix.Build.Reporting
64+ Garnix.Build.Types
65+ Garnix.BuildLogs
66+ Garnix.BuildLogs.Internal
67+ Garnix.BuildLogs.Types
68+ Garnix.DB
69+ Garnix.DB.FeatureFlags
70+ Garnix.DB.FeatureFlags.Types
71+ Garnix.DB.ModuleValues
72+ Garnix.Duration
73+ Garnix.Entitlements
74+ Garnix.ExpiringCache
75+ Garnix.FlakeInputAuthorization
76+ Garnix.GetAttributes
77+ Garnix.GithubInterface
78+ Garnix.GithubInterface.Types
79+ Garnix.HetznerInterface
80+ Garnix.Hosting.Deploy
81+ Garnix.Hosting.Helpers
82+ Garnix.Hosting.ServerPool
83+ Garnix.Hosting.ServerPool.Types
84+ Garnix.Incremental
85+ Garnix.Log
86+ Garnix.Modules
87+ Garnix.Modules.Schema
88+ Garnix.Monad
89+ Garnix.Monad.Async
90+ Garnix.Monad.Bubbling
91+ Garnix.Monad.Concurrency
92+ Garnix.Monad.ForkT
93+ Garnix.Monad.Memoization
94+ Garnix.Monad.Metrics
95+ Garnix.Monad.NoThrow
96+ Garnix.Monad.Polling
97+ Garnix.Monad.Pool
98+ Garnix.Monad.SubProcess
99+ Garnix.Monad.SubProcess.Deprecated
100+ Garnix.MonetaryCost
101+ Garnix.Nix.PathInfo
102+ Garnix.Nix.Plan
103+ Garnix.Nix.StorePath
104+ Garnix.Nix.Types
105+ Garnix.NixConfig
106+ Garnix.Orchestrator
107+ Garnix.ParseHttpBasicAuth
108+ Garnix.Password
109+ Garnix.Prelude
110+ Garnix.Reporters.GithubReporter
111+ Garnix.Reporters.OpenSearchReporter
112+ Garnix.Reporters.Utils
113+ Garnix.Request
114+ Garnix.S3Cache
115+ Garnix.SafeUnix
116+ Garnix.Sandbox
117+ Garnix.StripeLib
118+ Garnix.StripeLib.Types
119+ Garnix.Types
120+ Garnix.Types.ExternalLenses
121+ Garnix.Types.Keys
122+ Garnix.Types.MakeLensHelpers
123+ Garnix.UserLogs
124+ Garnix.YamlConfig
125+ other-modules:
126+ Paths_garnix
127+ hs-source-dirs:
128+ src
129+ default-extensions:
130+ BangPatterns
131+ ConstraintKinds
132+ DataKinds
133+ DeriveAnyClass
134+ DeriveFunctor
135+ DeriveGeneric
136+ DeriveTraversable
137+ DerivingStrategies
138+ DerivingVia
139+ FlexibleContexts
140+ FlexibleInstances
141+ FunctionalDependencies
142+ GADTs
143+ GeneralizedNewtypeDeriving
144+ ImportQualifiedPost
145+ InstanceSigs
146+ LambdaCase
147+ MultiParamTypeClasses
148+ MultiWayIf
149+ NamedFieldPuns
150+ NoImplicitPrelude
151+ NumericUnderscores
152+ OverloadedLabels
153+ OverloadedStrings
154+ PackageImports
155+ PolyKinds
156+ QuantifiedConstraints
157+ QuasiQuotes
158+ RankNTypes
159+ RecordWildCards
160+ ScopedTypeVariables
161+ StandaloneDeriving
162+ TupleSections
163+ TypeApplications
164+ TypeFamilies
165+ TypeOperators
166+ ViewPatterns
167+ ghc-options: -Wall -Werror -Wno-name-shadowing -fno-warn-star-is-type -threaded -rtsopts "-with-rtsopts=-T -N" -j
168+ build-depends:
169+ aeson
170+ , aeson-pretty
171+ , aeson-qq
172+ , amazonka
173+ , amazonka-s3
174+ , attoparsec
175+ , auto-update
176+ , autodocodec
177+ , autodocodec-schema
178+ , base >=4.7 && <5
179+ , base64-bytestring
180+ , bytestring
181+ , casing
182+ , containers
183+ , cookie
184+ , cradle
185+ , crypton
186+ , data-default-class
187+ , directory
188+ , exceptions
189+ , extra
190+ , fast-logger
191+ , file-embed
192+ , filepath
193+ , format-numbers
194+ , generic-lens
195+ , generics-eot
196+ , getopt-generics
197+ , github
198+ , github-app
199+ , github-webhooks
200+ , hashable
201+ , hashids
202+ , hashtables
203+ , http-client
204+ , http-client-tls
205+ , http-conduit
206+ , http-media
207+ , http-types
208+ , interpolate
209+ , iso-deriving
210+ , jose
211+ , jwt
212+ , lens
213+ , lens-aeson
214+ , lens-regex-pcre
215+ , lifted-async
216+ , lifted-base
217+ , monad-control
218+ , mtl
219+ , network-uri
220+ , nix-derivation
221+ , oauth2-simple
222+ , openapi3
223+ , postgresql-typed
224+ , pretty-show
225+ , prettyprinter
226+ , process
227+ , prometheus
228+ , random
229+ , resource-pool
230+ , resourcet
231+ , retry
232+ , row-types
233+ , row-types-aeson
234+ , safe-exceptions
235+ , servant
236+ , servant-auth
237+ , servant-auth-server
238+ , servant-github-webhook
239+ , servant-rawm
240+ , servant-rawm-server
241+ , servant-server
242+ , shake
243+ , stm
244+ , streaming
245+ , streaming-bytestring
246+ , string-conversions
247+ , strip-ansi-escape
248+ , stripe-concepts
249+ , stripe-signature
250+ , systemd
251+ , tagged
252+ , template-haskell
253+ , temporary
254+ , text
255+ , time
256+ , tls
257+ , transformers-base
258+ , typed-process
259+ , unix
260+ , unordered-containers
261+ , uuid
262+ , vector
263+ , wai
264+ , wai-app-static
265+ , wai-extra
266+ , warp
267+ , wreq
268+ , yaml
269+ default-language: Haskell2010
270+
271+executable convert-hashids
272+ main-is: convertHashids.hs
273+ hs-source-dirs:
274+ exe
275+ default-extensions:
276+ BangPatterns
277+ ConstraintKinds
278+ DataKinds
279+ DeriveAnyClass
280+ DeriveFunctor
281+ DeriveGeneric
282+ DeriveTraversable
283+ DerivingStrategies
284+ DerivingVia
285+ FlexibleContexts
286+ FlexibleInstances
287+ FunctionalDependencies
288+ GADTs
289+ GeneralizedNewtypeDeriving
290+ ImportQualifiedPost
291+ InstanceSigs
292+ LambdaCase
293+ MultiParamTypeClasses
294+ MultiWayIf
295+ NamedFieldPuns
296+ NoImplicitPrelude
297+ NumericUnderscores
298+ OverloadedLabels
299+ OverloadedStrings
300+ PackageImports
301+ PolyKinds
302+ QuantifiedConstraints
303+ QuasiQuotes
304+ RankNTypes
305+ RecordWildCards
306+ ScopedTypeVariables
307+ StandaloneDeriving
308+ TupleSections
309+ TypeApplications
310+ TypeFamilies
311+ TypeOperators
312+ ViewPatterns
313+ ghc-options: -Wall -Werror -Wno-name-shadowing -fno-warn-star-is-type -threaded -rtsopts "-with-rtsopts=-T -N" -j
314+ build-depends:
315+ aeson
316+ , aeson-pretty
317+ , aeson-qq
318+ , amazonka
319+ , amazonka-s3
320+ , attoparsec
321+ , auto-update
322+ , autodocodec
323+ , autodocodec-schema
324+ , base >=4.7 && <5
325+ , base64-bytestring
326+ , bytestring
327+ , casing
328+ , containers
329+ , cookie
330+ , cradle
331+ , crypton
332+ , data-default-class
333+ , directory
334+ , exceptions
335+ , extra
336+ , fast-logger
337+ , file-embed
338+ , filepath
339+ , format-numbers
340+ , garnix
341+ , generic-lens
342+ , generics-eot
343+ , getopt-generics
344+ , github
345+ , github-app
346+ , github-webhooks
347+ , hashable
348+ , hashids
349+ , hashtables
350+ , http-client
351+ , http-client-tls
352+ , http-conduit
353+ , http-media
354+ , http-types
355+ , interpolate
356+ , iso-deriving
357+ , jose
358+ , jwt
359+ , lens
360+ , lens-aeson
361+ , lens-regex-pcre
362+ , lifted-async
363+ , lifted-base
364+ , monad-control
365+ , mtl
366+ , network-uri
367+ , nix-derivation
368+ , oauth2-simple
369+ , openapi3
370+ , postgresql-typed
371+ , pretty-show
372+ , prettyprinter
373+ , process
374+ , prometheus
375+ , random
376+ , resource-pool
377+ , resourcet
378+ , retry
379+ , row-types
380+ , row-types-aeson
381+ , safe-exceptions
382+ , servant
383+ , servant-auth
384+ , servant-auth-server
385+ , servant-github-webhook
386+ , servant-rawm
387+ , servant-rawm-server
388+ , servant-server
389+ , shake
390+ , stm
391+ , streaming
392+ , streaming-bytestring
393+ , string-conversions
394+ , strip-ansi-escape
395+ , stripe-concepts
396+ , stripe-signature
397+ , systemd
398+ , tagged
399+ , template-haskell
400+ , temporary
401+ , text
402+ , time
403+ , tls
404+ , transformers-base
405+ , typed-process
406+ , unix
407+ , unordered-containers
408+ , uuid
409+ , vector
410+ , wai
411+ , wai-app-static
412+ , wai-extra
413+ , warp
414+ , wreq
415+ , yaml
416+ default-language: Haskell2010
417+
418+executable generate-yaml-config-docs
419+ main-is: GenerateYamlConfigDocs.hs
420+ hs-source-dirs:
421+ exe
422+ default-extensions:
423+ BangPatterns
424+ ConstraintKinds
425+ DataKinds
426+ DeriveAnyClass
427+ DeriveFunctor
428+ DeriveGeneric
429+ DeriveTraversable
430+ DerivingStrategies
431+ DerivingVia
432+ FlexibleContexts
433+ FlexibleInstances
434+ FunctionalDependencies
435+ GADTs
436+ GeneralizedNewtypeDeriving
437+ ImportQualifiedPost
438+ InstanceSigs
439+ LambdaCase
440+ MultiParamTypeClasses
441+ MultiWayIf
442+ NamedFieldPuns
443+ NoImplicitPrelude
444+ NumericUnderscores
445+ OverloadedLabels
446+ OverloadedStrings
447+ PackageImports
448+ PolyKinds
449+ QuantifiedConstraints
450+ QuasiQuotes
451+ RankNTypes
452+ RecordWildCards
453+ ScopedTypeVariables
454+ StandaloneDeriving
455+ TupleSections
456+ TypeApplications
457+ TypeFamilies
458+ TypeOperators
459+ ViewPatterns
460+ ghc-options: -Wall -Werror -Wno-name-shadowing -fno-warn-star-is-type -threaded -rtsopts "-with-rtsopts=-T -N" -j
461+ build-depends:
462+ aeson
463+ , aeson-pretty
464+ , aeson-qq
465+ , amazonka
466+ , amazonka-s3
467+ , attoparsec
468+ , auto-update
469+ , autodocodec
470+ , autodocodec-schema
471+ , autodocodec-yaml
472+ , base >=4.7 && <5
473+ , base64-bytestring
474+ , bytestring
475+ , casing
476+ , containers
477+ , cookie
478+ , cradle
479+ , crypton
480+ , data-default-class
481+ , directory
482+ , exceptions
483+ , extra
484+ , fast-logger
485+ , file-embed
486+ , filepath
487+ , format-numbers
488+ , garnix
489+ , generic-lens
490+ , generics-eot
491+ , getopt-generics
492+ , github
493+ , github-app
494+ , github-webhooks
495+ , hashable
496+ , hashids
497+ , hashtables
498+ , http-client
499+ , http-client-tls
500+ , http-conduit
501+ , http-media
502+ , http-types
503+ , interpolate
504+ , iso-deriving
505+ , jose
506+ , jwt
507+ , lens
508+ , lens-aeson
509+ , lens-regex-pcre
510+ , lifted-async
511+ , lifted-base
512+ , monad-control
513+ , mtl
514+ , network-uri
515+ , nix-derivation
516+ , oauth2-simple
517+ , openapi3
518+ , postgresql-typed
519+ , pretty-show
520+ , prettyprinter
521+ , process
522+ , prometheus
523+ , random
524+ , resource-pool
525+ , resourcet
526+ , retry
527+ , row-types
528+ , row-types-aeson
529+ , safe-exceptions
530+ , servant
531+ , servant-auth
532+ , servant-auth-server
533+ , servant-github-webhook
534+ , servant-rawm
535+ , servant-rawm-server
536+ , servant-server
537+ , shake
538+ , stm
539+ , streaming
540+ , streaming-bytestring
541+ , string-conversions
542+ , strip-ansi-escape
543+ , stripe-concepts
544+ , stripe-signature
545+ , systemd
546+ , tagged
547+ , template-haskell
548+ , temporary
549+ , text
550+ , time
551+ , tls
552+ , transformers-base
553+ , typed-process
554+ , unix
555+ , unordered-containers
556+ , uuid
557+ , vector
558+ , wai
559+ , wai-app-static
560+ , wai-extra
561+ , warp
562+ , wreq
563+ , yaml
564+ default-language: Haskell2010
565+
566+executable server
567+ main-is: Server.hs
568+ hs-source-dirs:
569+ exe
570+ default-extensions:
571+ BangPatterns
572+ ConstraintKinds
573+ DataKinds
574+ DeriveAnyClass
575+ DeriveFunctor
576+ DeriveGeneric
577+ DeriveTraversable
578+ DerivingStrategies
579+ DerivingVia
580+ FlexibleContexts
581+ FlexibleInstances
582+ FunctionalDependencies
583+ GADTs
584+ GeneralizedNewtypeDeriving
585+ ImportQualifiedPost
586+ InstanceSigs
587+ LambdaCase
588+ MultiParamTypeClasses
589+ MultiWayIf
590+ NamedFieldPuns
591+ NoImplicitPrelude
592+ NumericUnderscores
593+ OverloadedLabels
594+ OverloadedStrings
595+ PackageImports
596+ PolyKinds
597+ QuantifiedConstraints
598+ QuasiQuotes
599+ RankNTypes
600+ RecordWildCards
601+ ScopedTypeVariables
602+ StandaloneDeriving
603+ TupleSections
604+ TypeApplications
605+ TypeFamilies
606+ TypeOperators
607+ ViewPatterns
608+ ghc-options: -Wall -Werror -Wno-name-shadowing -fno-warn-star-is-type -threaded -rtsopts "-with-rtsopts=-T -N" -j
609+ build-depends:
610+ aeson
611+ , aeson-pretty
612+ , aeson-qq
613+ , amazonka
614+ , amazonka-s3
615+ , attoparsec
616+ , auto-update
617+ , autodocodec
618+ , autodocodec-schema
619+ , base >=4.7 && <5
620+ , base64-bytestring
621+ , bytestring
622+ , casing
623+ , containers
624+ , cookie
625+ , cradle
626+ , crypton
627+ , data-default-class
628+ , directory
629+ , exceptions
630+ , extra
631+ , fast-logger
632+ , file-embed
633+ , filepath
634+ , format-numbers
635+ , garnix
636+ , generic-lens
637+ , generics-eot
638+ , getopt-generics
639+ , github
640+ , github-app
641+ , github-webhooks
642+ , hashable
643+ , hashids
644+ , hashtables
645+ , http-client
646+ , http-client-tls
647+ , http-conduit
648+ , http-media
649+ , http-types
650+ , interpolate
651+ , iso-deriving
652+ , jose
653+ , jwt
654+ , lens
655+ , lens-aeson
656+ , lens-regex-pcre
657+ , lifted-async
658+ , lifted-base
659+ , monad-control
660+ , mtl
661+ , network-uri
662+ , nix-derivation
663+ , oauth2-simple
664+ , openapi3
665+ , postgresql-typed
666+ , pretty-show
667+ , prettyprinter
668+ , process
669+ , prometheus
670+ , random
671+ , resource-pool
672+ , resourcet
673+ , retry
674+ , row-types
675+ , row-types-aeson
676+ , safe-exceptions
677+ , servant
678+ , servant-auth
679+ , servant-auth-server
680+ , servant-github-webhook
681+ , servant-rawm
682+ , servant-rawm-server
683+ , servant-server
684+ , shake
685+ , stm
686+ , streaming
687+ , streaming-bytestring
688+ , string-conversions
689+ , strip-ansi-escape
690+ , stripe-concepts
691+ , stripe-signature
692+ , systemd
693+ , tagged
694+ , template-haskell
695+ , temporary
696+ , text
697+ , time
698+ , tls
699+ , transformers-base
700+ , typed-process
701+ , unix
702+ , unordered-containers
703+ , uuid
704+ , vector
705+ , wai
706+ , wai-app-static
707+ , wai-extra
708+ , warp
709+ , wreq
710+ , yaml
711+ default-language: Haskell2010
712+
713+test-suite spec
714+ type: exitcode-stdio-1.0
715+ main-is: Spec.hs
716+ other-modules:
717+ Garnix.AccessTokenSpec
718+ Garnix.ActionSpec
719+ Garnix.API.AccountSpec
720+ Garnix.API.AuthSpec
721+ Garnix.API.BadgesSpec
722+ Garnix.API.BuildSpec
723+ Garnix.API.Cache.PermissionsSpec
724+ Garnix.API.CommitSpec
725+ Garnix.API.ConfigSchemaSpec
726+ Garnix.API.DevSpec
727+ Garnix.API.GhWebhooksSpec
728+ Garnix.API.HealthSpec
729+ Garnix.API.HostsSpec
730+ Garnix.API.KeysSpec
731+ Garnix.API.ModulesSpec
732+ Garnix.API.RunsSpec
733+ Garnix.AsyncSpec
734+ Garnix.AttributeSpec
735+ Garnix.Build.CheckoutSpec
736+ Garnix.Build.FodCheckSpec
737+ Garnix.Build.ModuleSpec
738+ Garnix.BuildSpec
739+ Garnix.DB.FeatureFlagsSpec
740+ Garnix.DBSpec
741+ Garnix.DeploySpec
742+ Garnix.EntitlementsSpec
743+ Garnix.ExpiringCacheSpec
744+ Garnix.FlakeInputAuthorizationSpec
745+ Garnix.GetAttributesSpec
746+ Garnix.GithubInterfaceSpec
747+ Garnix.HetznerInterfaceSpec
748+ Garnix.Hosting.BillingSpec
749+ Garnix.Hosting.DeploySpec
750+ Garnix.Hosting.ServerPoolSpec
751+ Garnix.IncrementalSpec
752+ Garnix.Modules.SchemaSpec
753+ Garnix.ModulesSpec
754+ Garnix.Monad.BubblingSpec
755+ Garnix.Monad.ConcurrencySpec
756+ Garnix.Monad.ForkTSpec
757+ Garnix.Monad.MemoizationSpec
758+ Garnix.Monad.NoThrowSpec
759+ Garnix.Monad.PoolSpec
760+ Garnix.Monad.SubProcessSpec
761+ Garnix.MonadSpec
762+ Garnix.Nix.StorePathSpec
763+ Garnix.PreludeSpec
764+ Garnix.ProcessLogsSpec
765+ Garnix.Reporters.GithubReporterSpec
766+ Garnix.Reporters.OpenSearchReporterSpec
767+ Garnix.Reporters.UtilsSpec
768+ Garnix.S3CacheSpec
769+ Garnix.SandboxSpec
770+ Garnix.StripeLibSpec
771+ Garnix.TestHelpers
772+ Garnix.TestHelpers.Common
773+ Garnix.TestHelpers.Deprecated
774+ Garnix.TestHelpers.GithubInterface
775+ Garnix.TestHelpers.GithubInterface.Deprecated
776+ Garnix.TestHelpers.GithubInterface.Internal
777+ Garnix.TestHelpers.HetznerMock
778+ Garnix.TestHelpers.Monad
779+ Garnix.TestHelpers.NixosVmScripts
780+ Garnix.TestHelpers.Reporter
781+ Garnix.TestHelpers.ServerPool
782+ Garnix.TestHelpers.WithServer
783+ Garnix.TestInstances
784+ Garnix.TypesSpec
785+ Garnix.UserLogsSpec
786+ Garnix.YamlConfigSpec
787+ Integration.FlakesSpec
788+ SpecHook
789+ Garnix
790+ Garnix.Access
791+ Garnix.AccessToken
792+ Garnix.AccessToken.Types
793+ Garnix.API
794+ Garnix.API.Account
795+ Garnix.API.Auth
796+ Garnix.API.Badges
797+ Garnix.API.Builds
798+ Garnix.API.Builds.Types
799+ Garnix.API.Cache
800+ Garnix.API.Cache.Auth
801+ Garnix.API.Cache.Permissions
802+ Garnix.API.Cache.Types
803+ Garnix.API.Commits
804+ Garnix.API.ConfigSchema
805+ Garnix.API.Dev
806+ Garnix.API.GhWebhooks
807+ Garnix.API.Health
808+ Garnix.API.Hosts
809+ Garnix.API.Keys
810+ Garnix.API.Modules
811+ Garnix.API.Runs
812+ Garnix.API.Stripe
813+ Garnix.Async
814+ Garnix.Attribute
815+ Garnix.Build
816+ Garnix.Build.Action
817+ Garnix.Build.Checkout
818+ Garnix.Build.Evaluation
819+ Garnix.Build.Flake
820+ Garnix.Build.FodCheck
821+ Garnix.Build.Helpers
822+ Garnix.Build.MetaCheck
823+ Garnix.Build.Module
824+ Garnix.Build.Package
825+ Garnix.Build.Reporting
826+ Garnix.Build.Types
827+ Garnix.BuildLogs
828+ Garnix.BuildLogs.Internal
829+ Garnix.BuildLogs.Types
830+ Garnix.DB
831+ Garnix.DB.FeatureFlags
832+ Garnix.DB.FeatureFlags.Types
833+ Garnix.DB.ModuleValues
834+ Garnix.Duration
835+ Garnix.Entitlements
836+ Garnix.ExpiringCache
837+ Garnix.FlakeInputAuthorization
838+ Garnix.GetAttributes
839+ Garnix.GithubInterface
840+ Garnix.GithubInterface.Types
841+ Garnix.HetznerInterface
842+ Garnix.Hosting.Deploy
843+ Garnix.Hosting.Helpers
844+ Garnix.Hosting.ServerPool
845+ Garnix.Hosting.ServerPool.Types
846+ Garnix.Incremental
847+ Garnix.Log
848+ Garnix.Modules
849+ Garnix.Modules.Schema
850+ Garnix.Monad
851+ Garnix.Monad.Async
852+ Garnix.Monad.Bubbling
853+ Garnix.Monad.Concurrency
854+ Garnix.Monad.ForkT
855+ Garnix.Monad.Memoization
856+ Garnix.Monad.Metrics
857+ Garnix.Monad.NoThrow
858+ Garnix.Monad.Polling
859+ Garnix.Monad.Pool
860+ Garnix.Monad.SubProcess
861+ Garnix.Monad.SubProcess.Deprecated
862+ Garnix.MonetaryCost
863+ Garnix.Nix.PathInfo
864+ Garnix.Nix.Plan
865+ Garnix.Nix.StorePath
866+ Garnix.Nix.Types
867+ Garnix.NixConfig
868+ Garnix.Orchestrator
869+ Garnix.ParseHttpBasicAuth
870+ Garnix.Password
871+ Garnix.Prelude
872+ Garnix.Reporters.GithubReporter
873+ Garnix.Reporters.OpenSearchReporter
874+ Garnix.Reporters.Utils
875+ Garnix.Request
876+ Garnix.S3Cache
877+ Garnix.SafeUnix
878+ Garnix.Sandbox
879+ Garnix.StripeLib
880+ Garnix.StripeLib.Types
881+ Garnix.Types
882+ Garnix.Types.ExternalLenses
883+ Garnix.Types.Keys
884+ Garnix.Types.MakeLensHelpers
885+ Garnix.UserLogs
886+ Garnix.YamlConfig
887+ Paths_garnix
888+ hs-source-dirs:
889+ test/spec
890+ src
891+ default-extensions:
892+ BangPatterns
893+ ConstraintKinds
894+ DataKinds
895+ DeriveAnyClass
896+ DeriveFunctor
897+ DeriveGeneric
898+ DeriveTraversable
899+ DerivingStrategies
900+ DerivingVia
901+ FlexibleContexts
902+ FlexibleInstances
903+ FunctionalDependencies
904+ GADTs
905+ GeneralizedNewtypeDeriving
906+ ImportQualifiedPost
907+ InstanceSigs
908+ LambdaCase
909+ MultiParamTypeClasses
910+ MultiWayIf
911+ NamedFieldPuns
912+ NoImplicitPrelude
913+ NumericUnderscores
914+ OverloadedLabels
915+ OverloadedStrings
916+ PackageImports
917+ PolyKinds
918+ QuantifiedConstraints
919+ QuasiQuotes
920+ RankNTypes
921+ RecordWildCards
922+ ScopedTypeVariables
923+ StandaloneDeriving
924+ TupleSections
925+ TypeApplications
926+ TypeFamilies
927+ TypeOperators
928+ ViewPatterns
929+ ghc-options: -Wall -Werror -Wno-name-shadowing -fno-warn-star-is-type -threaded -rtsopts "-with-rtsopts=-T -N" -j -O0 -fno-warn-incomplete-uni-patterns
930+ build-depends:
931+ HUnit
932+ , QuickCheck
933+ , aeson
934+ , aeson-pretty
935+ , aeson-qq
936+ , amazonka
937+ , amazonka-s3
938+ , attoparsec
939+ , auto-update
940+ , autodocodec
941+ , autodocodec-schema
942+ , base >=4.7 && <5
943+ , base64-bytestring
944+ , binary
945+ , bytestring
946+ , casing
947+ , containers
948+ , cookie
949+ , cradle
950+ , crypton
951+ , data-default-class
952+ , directory
953+ , exceptions
954+ , extra
955+ , fast-logger
956+ , file-embed
957+ , filepath
958+ , format-numbers
959+ , generic-lens
960+ , generic-random
961+ , generics-eot
962+ , getopt-generics
963+ , github
964+ , github-app
965+ , github-webhooks
966+ , hashable
967+ , hashids
968+ , hashtables
969+ , hspec
970+ , hspec-core
971+ , hspec-discover
972+ , hspec-golden
973+ , hspec-golden-aeson
974+ , http-client
975+ , http-client-tls
976+ , http-conduit
977+ , http-media
978+ , http-types
979+ , interpolate
980+ , iso-deriving
981+ , jose
982+ , jwt
983+ , lens
984+ , lens-aeson
985+ , lens-regex-pcre
986+ , lifted-async
987+ , lifted-base
988+ , mockery
989+ , monad-control
990+ , mtl
991+ , neat-interpolation
992+ , network-uri
993+ , nix-derivation
994+ , oauth2-simple
995+ , openapi3
996+ , pcre-light
997+ , port-utils
998+ , postgresql-typed
999+ , pretty-show
1000+ , prettyprinter
1001+ , process
1002+ , prometheus
1003+ , quickcheck-instances
1004+ , random
1005+ , resource-pool
1006+ , resourcet
1007+ , retry
1008+ , row-types
1009+ , row-types-aeson
1010+ , safe-exceptions
1011+ , servant
1012+ , servant-auth
1013+ , servant-auth-server
1014+ , servant-github-webhook
1015+ , servant-rawm
1016+ , servant-rawm-server
1017+ , servant-server
1018+ , shake
1019+ , silently
1020+ , stm
1021+ , streaming
1022+ , streaming-bytestring
1023+ , string-conversions
1024+ , strip-ansi-escape
1025+ , stripe-concepts
1026+ , stripe-signature
1027+ , systemd
1028+ , tagged
1029+ , template-haskell
1030+ , temporary
1031+ , text
1032+ , time
1033+ , tls
1034+ , transformers-base
1035+ , turtle
1036+ , typed-process
1037+ , unix
1038+ , unordered-containers
1039+ , uuid
1040+ , vector
1041+ , wai
1042+ , wai-app-static
1043+ , wai-extra
1044+ , warp
1045+ , wreq
1046+ , yaml
1047+ default-language: Haskell2010
backend/nixos-module.nixnew file+636
@@ -0,0 +1,636 @@
1+{ config
2+, lib
3+, pkgs
4+, flakePackages
5+, flakeInputs
6+, ...
7+}:
8+let
9+ buildLogsFluentBitPort = 8888;
10+
11+ logsDir = pkgs.writeShellScriptBin "logsDir" ''
12+ if [ -d /var/lib/garnix/logs ]; then
13+ echo "Logs dir exists. Not creating"
14+ else
15+ mkdir /var/lib/garnix/logs
16+ fi
17+ '';
18+
19+ migrateScript = with pkgs;
20+ writeShellScriptBin "migrate" ''
21+ set -euo pipefail
22+
23+ export SQITCH_PASSWORD=$(cat /run/secrets/database-password)
24+ ${lib.getBin flakePackages."backend_migrate"}/bin/sqitch deploy \
25+ "db:pg://${config.garnix.database.dbUser}:$(cat /run/secrets/database-password)@${config.garnix.database.fqdn}:${toString config.garnix.database.dbPort}/${config.garnix.database.dbName}?sslmode=${config.garnix.database.ssl.mode}&sslrootcert=${config.garnix.database.ssl.rootCert}"
26+ '';
27+ dbCheckIsReady = with pkgs;
28+ writeShellScriptBin "dbCheckIsReady" ''
29+ #!/usr/bin/env bash
30+ set -uo pipefail
31+
32+ max_retries=10
33+ retry_interval=2
34+ retries=0
35+ start_time=$(date +%s)
36+
37+ export PGPASSWORD=$(cat /run/secrets/database-password)
38+ echo $PGHOST
39+
40+ while true; do
41+ end_time=$(date +%s)
42+ ${flakePackages."backend_postgres"}/bin/psql -c "SELECT 1;" >/dev/null
43+ #shellcheck disable=SC2181
44+ if [ $? -eq 0 ]; then
45+ elapsed_time=$((end_time - start_time))
46+ echo "Postgresql is ready to handle queries after $elapsed_time seconds."
47+ break
48+ fi
49+
50+ retries=$((retries + 1))
51+
52+ if [ $retries -ge $max_retries ]; then
53+ echo "Maximum retries reached. Readiness condition not met."
54+ exit 1
55+ fi
56+
57+ echo "Readiness condition not met. Retrying in $retry_interval seconds..."
58+ sleep $retry_interval
59+ done
60+ '';
61+in
62+{
63+ options = {
64+ services = {
65+ garnixServer = {
66+ enable = lib.mkEnableOption "garnix server";
67+ url = lib.mkOption {
68+ type = lib.types.str;
69+ default = "https://app.garnix.io";
70+ };
71+ githubAppName = lib.mkOption {
72+ type = lib.types.str;
73+ default = "garnix-ci";
74+ };
75+ port = lib.mkOption {
76+ type = lib.types.int;
77+ default = 8321;
78+ };
79+ monitoringPort = lib.mkOption {
80+ type = lib.types.int;
81+ default = 8322;
82+ };
83+ metricsPort = lib.mkOption {
84+ type = lib.types.int;
85+ default = 8323;
86+ };
87+ opensearchUrl = lib.mkOption {
88+ type = lib.types.str;
89+ example = "https://<opensearch-ip>/_msearch";
90+ };
91+ testFeatures = lib.mkOption {
92+ type = lib.types.listOf (lib.types.enum [ "DevApi" "OpenSearchMocks" "StripeMocks" "CacheUploadMocks" ]);
93+ default = [ ];
94+ };
95+ provisionServerPool = lib.mkOption {
96+ type = lib.types.bool;
97+ default = false;
98+ };
99+ s3Cache = {
100+ publicBucket = lib.mkOption {
101+ type = lib.types.str;
102+ };
103+ publicBaseUrl = lib.mkOption {
104+ type = lib.types.str;
105+ };
106+ privateBucket = lib.mkOption {
107+ type = lib.types.str;
108+ };
109+ host = lib.mkOption {
110+ type = lib.types.str;
111+ };
112+ region = lib.mkOption {
113+ type = lib.types.str;
114+ default = "auto";
115+ };
116+ };
117+ };
118+
119+ frontend = {
120+ port = lib.mkOption {
121+ type = lib.types.int;
122+ default = 3000;
123+ };
124+ };
125+ };
126+ };
127+
128+ config = lib.mkIf config.services.garnixServer.enable {
129+ assertions = [{
130+ assertion =
131+ !config.garnix.devMode.enable -> (config.services.garnixServer.testFeatures == [ ]);
132+ message = ''
133+ Test features cannot be enabled in production.
134+ If you want to enable test features, set garnix.devMode.enable to true.
135+ '';
136+ }];
137+
138+ networking.firewall.allowedTCPPorts = [ 80 443 ];
139+
140+ environment.systemPackages = [ pkgs.postgresql_18 ];
141+
142+ programs.ssh = {
143+ startAgent = true;
144+ extraConfig = ''
145+ AddKeysToAgent yes
146+ Host macMini1
147+ Hostname 142.132.141.88
148+ User nix-ssh
149+ IdentityFile /run/secrets/garnix_server_remote_builder_ssh
150+ Host macMini2
151+ Hostname 142.132.141.89
152+ User nix-ssh
153+ IdentityFile /run/secrets/garnix_server_remote_builder_ssh
154+ Host garnix5
155+ Hostname 65.108.28.108
156+ User nix-ssh
157+ IdentityFile /run/secrets/garnix_server_remote_builder_ssh
158+ Host garnix6
159+ Hostname 65.108.28.106
160+ User nix-ssh
161+ IdentityFile /run/secrets/garnix_server_remote_builder_ssh
162+ Host garnix7
163+ Hostname 65.108.28.107
164+ User nix-ssh
165+ IdentityFile /run/secrets/garnix_server_remote_builder_ssh
166+ Host garnix8
167+ Hostname 88.99.75.150
168+ User nix-ssh
169+ IdentityFile /run/secrets/garnix_server_remote_builder_ssh
170+ Host garnix9
171+ Hostname 157.90.140.190
172+ User nix-ssh
173+ IdentityFile /run/secrets/garnix_server_remote_builder_ssh
174+ Host arm-server-0
175+ Hostname 65.109.75.126
176+ User nix-ssh
177+ IdentityFile /run/secrets/garnix_server_remote_builder_ssh
178+ Host arm-server-1
179+ Hostname 91.107.205.127
180+ User nix-ssh
181+ IdentityFile /run/secrets/garnix_server_remote_builder_ssh
182+ Host cache.garnix.io
183+ Hostname 37.27.121.36
184+ User cache-uploader
185+ '';
186+ };
187+
188+ nix = {
189+ settings = {
190+ cores = 4;
191+ };
192+ extraOptions = ''
193+ max-jobs = ${if config.garnix.devMode.enable then "auto" else "0"}
194+ keep-build-log = true
195+ '';
196+ buildMachines =
197+ let
198+ sshUser = "nix-ssh";
199+ sshKey = "/run/secrets/garnix_server_remote_builder_ssh";
200+ protocol = "ssh-ng";
201+ in
202+ [
203+ {
204+ hostName = "macMini1";
205+ inherit sshUser sshKey protocol;
206+ systems = [ "aarch64-darwin" "x86_64-darwin" ];
207+ maxJobs = 4;
208+ speedFactor = 1;
209+ supportedFeatures = [
210+ "big-parallel"
211+ "recursive-nix"
212+ ];
213+ mandatoryFeatures = [ ];
214+ }
215+ {
216+ hostName = "macMini2";
217+ inherit sshUser sshKey protocol;
218+ systems = [ "aarch64-darwin" "x86_64-darwin" ];
219+ maxJobs = 4;
220+ speedFactor = 1;
221+ supportedFeatures = [
222+ "big-parallel"
223+ "recursive-nix"
224+ ];
225+ mandatoryFeatures = [ ];
226+ }
227+ {
228+ hostName = "garnix5";
229+ inherit sshUser sshKey protocol;
230+ systems = [ "x86_64-linux" "i686-linux" ];
231+ maxJobs = 28;
232+ speedFactor = 4;
233+ supportedFeatures = [
234+ "nixos-test"
235+ "kvm"
236+ "big-parallel"
237+ "recursive-nix"
238+ ];
239+ mandatoryFeatures = [ ];
240+ }
241+ {
242+ hostName = "garnix6";
243+ inherit sshUser sshKey protocol;
244+ systems = [ "x86_64-linux" "i686-linux" ];
245+ maxJobs = 28;
246+ speedFactor = 4;
247+ supportedFeatures = [
248+ "nixos-test"
249+ "kvm"
250+ "big-parallel"
251+ "recursive-nix"
252+ ];
253+ mandatoryFeatures = [ ];
254+ }
255+ {
256+ hostName = "garnix7";
257+ inherit sshUser sshKey protocol;
258+ systems = [ "x86_64-linux" "i686-linux" ];
259+ maxJobs = 28;
260+ speedFactor = 4;
261+ supportedFeatures = [
262+ "nixos-test"
263+ "kvm"
264+ "big-parallel"
265+ "recursive-nix"
266+ ];
267+ mandatoryFeatures = [ ];
268+ }
269+ {
270+ hostName = "garnix8";
271+ inherit sshUser sshKey protocol;
272+ systems = [ "x86_64-linux" "i686-linux" ];
273+ maxJobs = 28;
274+ speedFactor = 4;
275+ supportedFeatures = [
276+ "nixos-test"
277+ "kvm"
278+ "big-parallel"
279+ "recursive-nix"
280+ ];
281+ mandatoryFeatures = [ ];
282+ }
283+ {
284+ hostName = "garnix9";
285+ inherit sshUser sshKey protocol;
286+ systems = [ "x86_64-linux" "i686-linux" ];
287+ maxJobs = 28;
288+ speedFactor = 4;
289+ supportedFeatures = [
290+ "nixos-test"
291+ "kvm"
292+ "big-parallel"
293+ "recursive-nix"
294+ ];
295+ mandatoryFeatures = [ ];
296+ }
297+ {
298+ hostName = "arm-server-0";
299+ inherit sshUser sshKey protocol;
300+ systems = [ "aarch64-linux" ];
301+ maxJobs = 60;
302+ speedFactor = 4;
303+ supportedFeatures = [
304+ "nixos-test"
305+ "kvm"
306+ "big-parallel"
307+ "recursive-nix"
308+ ];
309+ mandatoryFeatures = [ ];
310+ }
311+ {
312+ hostName = "arm-server-1";
313+ inherit sshUser sshKey protocol;
314+ systems = [ "aarch64-linux" ];
315+ maxJobs = 8;
316+ speedFactor = 1;
317+ supportedFeatures = [
318+ "nixos-test"
319+ "kvm"
320+ "big-parallel"
321+ "recursive-nix"
322+ ];
323+ mandatoryFeatures = [ ];
324+ }
325+ ];
326+ distributedBuilds = !config.garnix.devMode.enable;
327+ daemonIOSchedPriority = 4;
328+ };
329+
330+ virtualisation.vmVariant = { lib, ... }: {
331+ services.garnixServer = {
332+ url = "https://testing.garnix.io";
333+ githubAppName = "test-app-jkarni";
334+ };
335+ };
336+
337+ garnix = {
338+ custom-gc = {
339+ enable = true;
340+ enableTimer = true;
341+ useNixHeuristicGc = false;
342+ upperLimitPercent = null;
343+ };
344+ fluent-bit.enableNginxLogParsing = true;
345+ fluent-bit.configuration.pipelines.build-logs =
346+ let
347+ tag = "build-logs";
348+ in
349+ {
350+ input = {
351+ Name = "http";
352+ Tag = tag;
353+ listen = "::1";
354+ port = buildLogsFluentBitPort;
355+ };
356+ output = {
357+ Name = "opensearch";
358+ Match = tag;
359+ Host = config.garnix.fluent-bit.opensearch.fqdn;
360+ Port = 443;
361+ Tls = "On";
362+ "Tls.verify" = if config.garnix.devMode.enable then "Off" else "On";
363+ HTTP_User = config.garnix.fluent-bit.opensearch.basicAuth.username;
364+ HTTP_Passwd = ''''${OPENSEARCH_PASSWORD}'';
365+ Logstash_Format = "On";
366+ Logstash_Prefix = "garnix-build-logs";
367+ Logstash_DateFormat = "%Y.%m.%d";
368+ Time_Key = "@timestamp";
369+ Time_Key_Nanos = "On";
370+ Replace_Dots = "On";
371+ Suppress_Type_Name = "On";
372+ Index = "fluent-bit";
373+ };
374+ };
375+ };
376+
377+ services.garnixServer = {
378+ s3Cache = lib.mkMerge [
379+ (lib.mkIf config.garnix.devMode.enable {
380+ publicBucket = "test-public";
381+ publicBaseUrl = "https://pub-aed3ff3b65d444b3aeee39d6ea1767b0.r2.dev";
382+ privateBucket = "test-private";
383+ host = "79e0f6a031ca6d9650034b607922ba45.r2.cloudflarestorage.com";
384+ })
385+ (lib.mkIf (!config.garnix.devMode.enable) {
386+ publicBucket = "prod-public";
387+ publicBaseUrl = "https://garnix-cache.com";
388+ privateBucket = "prod-private";
389+ host = "79e0f6a031ca6d9650034b607922ba45.r2.cloudflarestorage.com";
390+ })
391+ ];
392+ };
393+
394+ systemd.services.garnixServer = {
395+ description = "The garnix server";
396+ # Coreutils is needed so we have the right version of 'tail', allowing
397+ # use of the --pid option
398+ path = with pkgs; [
399+ git
400+ config.nix.package
401+ coreutils
402+ util-linux
403+ bzip2
404+ openssh
405+ age
406+ flakeInputs.comment.packages.${stdenv.hostPlatform.system}.default
407+ ];
408+ wantedBy = [ "multi-user.target" ];
409+ wants = [ "network-online.target" ];
410+ after = [ "network-online.target" ];
411+ serviceConfig = {
412+ Type = "notify";
413+ User = config.garnix.database.dbUser;
414+ RestartSec = 1;
415+ Restart = "always";
416+ StartLimitBurst = 3;
417+ TimeoutStopSec = 5;
418+ KillSignal = "SIGINT";
419+ # If you change this, also change the build-logs-dir option
420+ StateDirectory = "garnix";
421+ PrivateTmp = false;
422+ LimitNOFILE = 1048576;
423+ Environment = [
424+ "PGPORT=${toString config.garnix.database.dbPort}"
425+ "PGUSER=${config.garnix.database.dbUser}"
426+ "PGDATABASE=${config.garnix.database.dbName}"
427+ "PGHOST=${config.garnix.database.fqdn}"
428+ "PGSSLMODE=${config.garnix.database.ssl.mode}"
429+ "PGSSLROOTCERT=${config.garnix.database.ssl.rootCert}"
430+ "TPG_TLS=true"
431+ "TPG_TLS_MODE=full"
432+ "TPG_HOST=${toString config.garnix.database.fqdn}"
433+ "TPG_PORT=${toString config.garnix.database.dbPort}"
434+ "TPG_USER=${config.garnix.database.dbUser}"
435+ "TPG_DB=${config.garnix.database.dbName}"
436+ "GARNIX_URL=${config.services.garnixServer.url}"
437+ "GITHUB_APP_NAME=${config.services.garnixServer.githubAppName}"
438+ "OPENSEARCH_URL=${config.services.garnixServer.opensearchUrl}"
439+ "S3_CACHE_REGION=${config.services.garnixServer.s3Cache.region}"
440+ "S3_CACHE_HOST=${config.services.garnixServer.s3Cache.host}"
441+ "S3_CACHE_PUBLIC_BUCKET=${config.services.garnixServer.s3Cache.publicBucket}"
442+ "S3_CACHE_PUBLIC_BASE_URL=${config.services.garnixServer.s3Cache.publicBaseUrl}"
443+ "S3_CACHE_PRIVATE_BUCKET=${config.services.garnixServer.s3Cache.privateBucket}"
444+ ];
445+ SupplementaryGroups = [ config.users.groups.keys.name ];
446+ ExecStartPre = [
447+ "${dbCheckIsReady}/bin/dbCheckIsReady"
448+ "${migrateScript}/bin/migrate"
449+ "${logsDir}/bin/logsDir"
450+ ];
451+ ExecStart = ''
452+ ${lib.getBin flakePackages."backend_garnix"}/bin/server \
453+ ${lib.concatStringsSep " " (builtins.map (testFeature: "--enable ${testFeature}") config.services.garnixServer.testFeatures)} \
454+ --port ${toString config.services.garnixServer.port} \
455+ --monitoring-port ${toString config.services.garnixServer.monitoringPort} \
456+ --metrics-port ${toString config.services.garnixServer.metricsPort} \
457+ --build-logs-reporting-port ${toString buildLogsFluentBitPort} \
458+ --build-logs-dir /var/lib/garnix/logs \
459+ ${lib.optionalString config.services.garnixServer.provisionServerPool
460+ '' --provision-server-pool''}
461+ '';
462+ };
463+ unitConfig = {
464+ StartLimitIntervalSec = 10;
465+ };
466+ };
467+
468+ systemd.services.frontend = {
469+ description = "Garnix NextJS webapp";
470+ wantedBy = [ "multi-user.target" ];
471+ wants = [ "network-online.target" ];
472+ after = [ "network-online.target" ];
473+ serviceConfig = {
474+ Type = "simple";
475+ User = config.users.users.garnix.name;
476+ RestartSec = 1;
477+ Restart = "always";
478+ KillSignal = "SIGINT";
479+ Environment = [
480+ "PORT=${toString config.services.frontend.port}"
481+ "HOSTNAME=127.0.0.1"
482+ ];
483+ ExecStart = ''
484+ ${flakePackages.frontend_default}/bin/garnix-frontend
485+ '';
486+ };
487+ };
488+
489+ services.nginx = {
490+ enable = true;
491+ recommendedProxySettings = true;
492+ recommendedOptimisation = true;
493+ recommendedGzipSettings = true;
494+ proxyTimeout = "600s";
495+ appendConfig = ''
496+ worker_processes auto;
497+ worker_rlimit_nofile 2048;
498+ '';
499+ eventsConfig = ''
500+ worker_connections 1024;
501+ '';
502+ virtualHosts."garnix.io" = {
503+ default = true;
504+ forceSSL = ! config.garnix.devMode.enable;
505+ enableACME = ! config.garnix.devMode.enable;
506+ locations."/api".proxyPass = "http://127.0.0.1:${toString config.services.garnixServer.port}";
507+ locations."@frontend".proxyPass = "http://127.0.0.1:${toString config.services.frontend.port}";
508+ locations."/" = {
509+ root = "${flakePackages.frontend_default}/public";
510+ tryFiles = "$uri @frontend";
511+ };
512+ };
513+ virtualHosts."api.garnix.io" = {
514+ default = false;
515+ forceSSL = ! config.garnix.devMode.enable;
516+ enableACME = ! config.garnix.devMode.enable;
517+ locations."/" = {
518+ proxyPass = "http://127.0.0.1:${toString config.services.garnixServer.port}/api/";
519+ extraConfig = ''
520+ if ($http_origin ~* "^https://(app\.)?garnix\.io$") {
521+ add_header Access-Control-Allow-Origin "$http_origin";
522+ }
523+ '';
524+ };
525+ };
526+ };
527+
528+ services.journald.extraConfig = ''
529+ SystemMaxUse=100G
530+ SystemMaxFiles=1000
531+ '';
532+
533+ sops.secrets = {
534+ database-password = {
535+ mode = "0440";
536+ group = config.users.users.garnix.name;
537+ };
538+ github_webhook_secret = {
539+ mode = "0440";
540+ group = config.users.users.garnix.name;
541+ };
542+ github_client_secret = {
543+ mode = "0440";
544+ group = config.users.users.garnix.name;
545+ };
546+ github_client_id = {
547+ mode = "0440";
548+ group = config.users.users.garnix.name;
549+ };
550+ github_app_id = {
551+ mode = "0440";
552+ group = config.users.users.garnix.name;
553+ };
554+ github_app_pk = {
555+ mode = "0440";
556+ group = config.users.users.garnix.name;
557+ };
558+ garnix_server_remote_builder_ssh = {
559+ mode = "0400";
560+ group = config.users.users.garnix.name;
561+ };
562+ garnix_server_remote_builder_ssh_garnix = {
563+ mode = "0400";
564+ owner = config.users.users.garnix.name;
565+ key = "garnix_server_remote_builder_ssh";
566+ };
567+ hetzner-token = {
568+ mode = "0440";
569+ group = config.users.users.garnix.name;
570+ };
571+ opensearch-garnix = {
572+ mode = "0400";
573+ owner = config.users.users.garnix.name;
574+ };
575+ repo-secrets-key = {
576+ mode = "0400";
577+ owner = config.users.users.garnix.name;
578+ };
579+ repo-secrets-key-pub = {
580+ mode = "0400";
581+ owner = config.users.users.garnix.name;
582+ };
583+ stripe-publishable-key = {
584+ mode = "0400";
585+ owner = config.users.users.garnix.name;
586+ };
587+ stripe-secret-key = {
588+ mode = "0400";
589+ owner = config.users.users.garnix.name;
590+ };
591+ stripe-webhook-secret = {
592+ mode = "0400";
593+ owner = config.users.users.garnix.name;
594+ };
595+ prometheus-node-exporter-1 = {
596+ mode = "0400";
597+ owner = config.users.users.garnix.name;
598+ };
599+ # This key was generated by `Servant.Auth.Server.writeKey`.
600+ "garnix-jwt-key" = {
601+ mode = "0400";
602+ owner = config.users.users.garnix.name;
603+ };
604+ "garnix_server_ssh_hosting" = {
605+ mode = "0400";
606+ owner = config.users.users.garnix.name;
607+ };
608+ garnix_action_runner_ssh = {
609+ mode = "0400";
610+ owner = config.users.users.garnix.name;
611+ };
612+ "s3-cache-access-key-id" = {
613+ mode = "0400";
614+ owner = config.users.users.garnix.name;
615+ };
616+ "s3-cache-secret-access-key" = {
617+ mode = "0400";
618+ owner = config.users.users.garnix.name;
619+ };
620+ "cache-priv-key" = {
621+ mode = "0400";
622+ owner = config.users.users.garnix.name;
623+ };
624+ };
625+
626+ users.users.garnix = {
627+ group = "garnix";
628+ extraGroups = [ config.users.groups.keys.name ];
629+ isNormalUser = true;
630+ description = "Garnix server user";
631+ };
632+ users.groups.garnix = { };
633+
634+ garnix.killRogueNixProcesses = false;
635+ };
636+}
backend/package.yamlnew file+217
@@ -0,0 +1,217 @@
1+name: garnix
2+version: 0.1.0.0
3+github: "jkarni/garnix"
4+license: BSD3
5+author: "Julian K. Arni"
6+maintainer: "jkarni@riseup.net"
7+copyright: "AllRightsReserved"
8+
9+default-extensions:
10+ - BangPatterns
11+ - ConstraintKinds
12+ - DataKinds
13+ - DeriveAnyClass
14+ - DeriveFunctor
15+ - DeriveGeneric
16+ - DeriveTraversable
17+ - DerivingStrategies
18+ - DerivingVia
19+ - FlexibleContexts
20+ - FlexibleInstances
21+ - FunctionalDependencies
22+ - GADTs
23+ - GeneralizedNewtypeDeriving
24+ - ImportQualifiedPost
25+ - InstanceSigs
26+ - LambdaCase
27+ - MultiParamTypeClasses
28+ - MultiWayIf
29+ - NamedFieldPuns
30+ - NoImplicitPrelude
31+ - NumericUnderscores
32+ - OverloadedLabels
33+ - OverloadedStrings
34+ - PackageImports
35+ - PolyKinds
36+ - QuantifiedConstraints
37+ - QuasiQuotes
38+ - RankNTypes
39+ - RecordWildCards
40+ - ScopedTypeVariables
41+ - StandaloneDeriving
42+ - TupleSections
43+ - TypeApplications
44+ - TypeFamilies
45+ - TypeOperators
46+ - ViewPatterns
47+
48+dependencies:
49+ - aeson
50+ - aeson-pretty
51+ - aeson-qq
52+ - amazonka
53+ - amazonka-s3
54+ - attoparsec
55+ - autodocodec
56+ - autodocodec-schema
57+ - auto-update
58+ - base >= 4.7 && < 5
59+ - base64-bytestring
60+ - bytestring
61+ - casing
62+ - containers
63+ - cookie
64+ - cradle
65+ - crypton
66+ - data-default-class
67+ - directory
68+ - exceptions
69+ - extra
70+ - fast-logger
71+ - file-embed
72+ - filepath
73+ - format-numbers
74+ - generic-lens
75+ - generics-eot
76+ - getopt-generics
77+ - github
78+ - github-app
79+ - github-webhooks
80+ - hashable
81+ - hashids
82+ - hashtables
83+ - http-client
84+ - http-client-tls
85+ - http-conduit
86+ - http-media
87+ - http-types
88+ - interpolate
89+ - iso-deriving
90+ - jose
91+ - jwt
92+ - lens
93+ - lens-aeson
94+ - lens-regex-pcre
95+ - lifted-async
96+ - lifted-base
97+ - monad-control
98+ - mtl
99+ - network-uri
100+ - nix-derivation
101+ - oauth2-simple
102+ - openapi3
103+ - postgresql-typed
104+ - prettyprinter
105+ - pretty-show
106+ - process
107+ - prometheus
108+ - random
109+ - resource-pool
110+ - resourcet
111+ - retry
112+ - row-types
113+ - row-types-aeson
114+ - safe-exceptions
115+ - servant
116+ - servant-auth
117+ - servant-auth-server
118+ - servant-github-webhook
119+ - servant-rawm
120+ - servant-rawm-server
121+ - servant-server
122+ - shake
123+ - stm
124+ - streaming
125+ - streaming-bytestring
126+ - string-conversions
127+ - strip-ansi-escape
128+ - stripe-concepts
129+ - stripe-signature
130+ - systemd
131+ - tagged
132+ - template-haskell
133+ - temporary
134+ - text
135+ - time
136+ - tls
137+ - transformers-base
138+ - typed-process
139+ - unix
140+ - unordered-containers
141+ - uuid
142+ - vector
143+ - wai
144+ - wai-app-static
145+ - wai-extra
146+ - warp
147+ - wreq
148+ - yaml
149+
150+library:
151+ source-dirs: src
152+
153+extra-source-files:
154+ - test/spec/data/create-server-response.json
155+ - test/spec/data/get-servers-response.json
156+ - test/spec/data/get-servers-response.json
157+ - test/spec/data/emptyDir/.gitignore
158+
159+ghc-options:
160+ - -Wall
161+ - -Werror
162+ - -Wno-name-shadowing
163+ - -fno-warn-star-is-type
164+ - -threaded
165+ - -rtsopts
166+ - '"-with-rtsopts=-T -N"'
167+ - -j
168+
169+executables:
170+ server:
171+ source-dirs: exe
172+ main: Server.hs
173+ other-modules: []
174+ dependencies:
175+ - garnix
176+ generate-yaml-config-docs:
177+ source-dirs: exe
178+ main: GenerateYamlConfigDocs.hs
179+ other-modules: []
180+ dependencies:
181+ - garnix
182+ - autodocodec-yaml
183+ convert-hashids:
184+ source-dirs:
185+ - exe
186+ main: convertHashids.hs
187+ other-modules: []
188+ dependencies:
189+ - garnix
190+
191+tests:
192+ spec:
193+ main: Spec.hs
194+ source-dirs:
195+ - test/spec
196+ - src
197+ ghc-options:
198+ - -O0
199+ - -fno-warn-incomplete-uni-patterns
200+ dependencies:
201+ - binary
202+ - generic-random
203+ - hspec
204+ - hspec-core
205+ - hspec-discover
206+ - hspec-golden
207+ - hspec-golden-aeson
208+ - HUnit
209+ - mockery
210+ - neat-interpolation
211+ - pcre-light
212+ - port-utils
213+ - QuickCheck
214+ - quickcheck-instances
215+ - silently
216+ - turtle
217+ - unix
backend/src/Garnix.hsnew file+418
@@ -0,0 +1,418 @@
1+module Garnix where
2+
3+import Amazonka qualified
4+import Amazonka.Auth qualified as Amazonka
5+import Amazonka.S3 qualified as Amazonka
6+import Control.Concurrent (forkIO, getNumCapabilities, newMVar)
7+import Control.Exception qualified
8+import Control.Exception.Safe qualified as Safe
9+import Cradle qualified
10+import Crypto.PubKey.RSA.Read (readRsaPem)
11+import Data.ByteString.Base64 qualified as B64
12+import Data.ByteString.Char8 qualified
13+import Data.ByteString.Char8 qualified as BSC
14+import Data.Functor ((<&>))
15+import Data.HashTable.IO qualified as HashTables
16+import Data.Pool qualified as Pool
17+import Data.Set qualified as Set
18+import Data.Text qualified as T
19+import Data.Text.IO (hPutStrLn)
20+import Data.Text.IO qualified as T
21+import Database.PostgreSQL.Typed (pgDisconnect)
22+import GHC.Conc (getNumProcessors)
23+import Garnix.API
24+import Garnix.DB qualified as DB
25+import Garnix.DB.FeatureFlags (withRecachedFeatureFlags)
26+import Garnix.DB.FeatureFlags.Types (getFeatureFlagConfig)
27+import Garnix.Duration
28+import Garnix.GithubInterface
29+import Garnix.HetznerInterface
30+import Garnix.Hosting.Deploy (stopUnusedServers)
31+import Garnix.Hosting.ServerPool qualified as ServerPool
32+import Garnix.Hosting.ServerPool.Types
33+import Garnix.Monad
34+import Garnix.Monad.Metrics (registerMetrics, serveMetrics)
35+import Garnix.Monad.Pool qualified
36+import Garnix.NixConfig (defaultNixConfig)
37+import Garnix.Prelude
38+import Garnix.StripeLib qualified
39+import Garnix.Types
40+import Garnix.UserLogs
41+import GitHub.App.Auth (AppAuth (..))
42+import GitHub.Data.Id (Id (..))
43+import GitHub.Data.Webhooks.Events
44+import Network.HTTP.Client.TLS (newTlsManager)
45+import Network.Wai.Handler.Warp qualified as Warp
46+import Network.Wai.Middleware.Gzip
47+import Servant
48+import Servant.Auth.Server
49+ ( CookieSettings (..),
50+ JWTSettings,
51+ defaultCookieSettings,
52+ defaultJWTSettings,
53+ fromSecret,
54+ )
55+import Servant.GitHub.Webhook
56+import Stripe.Concepts qualified
57+import System.Directory
58+import System.Environment (getEnv)
59+import System.Systemd.Daemon (notifyReady)
60+import WithCli (HasArguments, withCli)
61+
62+run :: IO ()
63+run = withCli runWith
64+
65+data Options = Options
66+ { enable :: [String],
67+ port :: Warp.Port,
68+ monitoringPort :: Warp.Port,
69+ metricsPort :: Warp.Port,
70+ buildLogsDir :: FilePath,
71+ buildLogsReportingPort :: Maybe Warp.Port,
72+ provisionServerPool :: Bool
73+ }
74+ deriving stock (Eq, Show, Generic)
75+ deriving anyclass (HasArguments)
76+
77+envMocks :: Set TestFeature -> IO (Maybe EnvMocks)
78+envMocks testFeatures = do
79+ foldM helper Nothing testFeatures
80+ where
81+ helper :: Maybe EnvMocks -> TestFeature -> IO (Maybe EnvMocks)
82+ helper mEnvMocks testFeature = case testFeature of
83+ DevApi -> pure mEnvMocks
84+ OpenSearchMocks -> do
85+ let envMocks = fromMaybe emptyMocks mEnvMocks
86+ (storeLogLineMock, queryOpenSearchMock) <- Garnix.UserLogs.testImplementation
87+ pure
88+ $ Just
89+ $ envMocks
90+ { storeLogLineMock = Just storeLogLineMock,
91+ queryOpenSearchMock = Just queryOpenSearchMock
92+ }
93+ StripeMocks -> do
94+ let envMocks = fromMaybe emptyMocks mEnvMocks
95+ (createCustomerMock, createSubscriptionMock, createInvoiceMock, listSubscriptionsMock, cancelSubscriptionMock, getPriceMock) <- Garnix.StripeLib.testImplementation
96+ pure
97+ $ Just
98+ $ envMocks
99+ { createCustomerMock = Just createCustomerMock,
100+ createSubscriptionMock = Just createSubscriptionMock,
101+ createInvoiceItemMock = Just createInvoiceMock,
102+ listSubscriptionsMock = Just listSubscriptionsMock,
103+ cancelSubscriptionMock = Just cancelSubscriptionMock,
104+ getPriceMock = Just getPriceMock
105+ }
106+ CacheUploadMocks -> do
107+ let envMocks = fromMaybe emptyMocks mEnvMocks
108+ s3UploadMock <- newMock (\_ -> pure ())
109+ pure
110+ $ Just
111+ $ envMocks
112+ { s3CacheUploadMock = Just s3UploadMock
113+ }
114+ FodCheckMocks -> do
115+ let envMocks = fromMaybe emptyMocks mEnvMocks
116+ fodCheckMock <- newMock (\_ -> pure ())
117+ pure $ Just $ envMocks {fodCheckMock = Just fodCheckMock}
118+
119+withEnv :: (HasCallStack) => Set TestFeature -> FilePath -> Maybe Warp.Port -> (Env -> IO a) -> IO a
120+withEnv testFeatures buildLogsDir buildLogsReportingPort action = do
121+ buildLogsDir' <- makeAbsolute buildLogsDir
122+ ghK <-
123+ lookupEnv "GITHUB_WEBHOOK_SECRET"
124+ >>= maybe (BSC.readFile "/run/secrets/github_webhook_secret") (pure . cs)
125+ ghClientSecret <-
126+ lookupEnv "GITHUB_CLIENT_SECRET"
127+ >>= maybe (cs <$> readFile "/run/secrets/github_client_secret") (pure . cs)
128+ Just emptyDir' <- lookupEnv "EMPTY_DIR"
129+ ghClientId <-
130+ lookupEnv "GITHUB_CLIENT_ID"
131+ >>= maybe (cs <$> readFile "/run/secrets/github_client_id") (pure . cs)
132+ appId <-
133+ fmap (Id . read)
134+ $ lookupEnv "GITHUB_APP_ID"
135+ >>= maybe (readFile "/run/secrets/github_app_id") pure
136+ appPkPem' <-
137+ lookupEnv "GITHUB_APP_PK"
138+ >>= maybe (BSC.readFile "/run/secrets/github_app_pk") (pure . cs)
139+ ghAppName <-
140+ lookupEnv "GITHUB_APP_NAME"
141+ >>= maybe (cs <$> readFile "/run/secrets/github_app_name") (pure . cs)
142+ sshKeys <- do
143+ envValue <- lookupEnv "GARNIX_SERVER_SSH_KEYS"
144+ let sshKeyPaths = case envValue of
145+ Nothing -> ["/run/secrets/garnix_server_ssh_hosting"]
146+ Just v -> map cs $ T.splitOn "," (cs v)
147+ forM sshKeyPaths $ \sshKey -> do
148+ doesFileExist sshKey >>= \exists ->
149+ unless exists
150+ $ error
151+ $ "ssh key not found: "
152+ <> cs sshKey
153+ liftIO $ makeAbsolute sshKey
154+ s3CacheEnv <- do
155+ amazonkaEnv <- do
156+ accessKeyId <-
157+ ( lookupEnv "S3_CACHE_ACCESS_KEY_ID"
158+ >>= maybe (BSC.readFile "/run/secrets/s3-cache-access-key-id") (pure . cs)
159+ )
160+ <&> Amazonka.AccessKey
161+ secretAccessKey <-
162+ ( lookupEnv "S3_CACHE_SECRET_ACCESS_KEY"
163+ >>= maybe (BSC.readFile "/run/secrets/s3-cache-secret-access-key") (pure . cs)
164+ )
165+ <&> Amazonka.SecretKey
166+ region <- cs <$> getEnv "S3_CACHE_REGION"
167+ host <- cs <$> getEnv "S3_CACHE_HOST"
168+ Amazonka.newEnv (pure . Amazonka.fromKeys accessKeyId secretAccessKey)
169+ <&> (#region .~ Amazonka.Region' region)
170+ <&> Amazonka.overrideService (Amazonka.setEndpoint True host 443)
171+ <&> Amazonka.overrideService (#s3AddressingStyle .~ Amazonka.S3AddressingStylePath)
172+ publicBucket <- Amazonka.BucketName . cs <$> getEnv "S3_CACHE_PUBLIC_BUCKET"
173+ publicBaseUrl <-
174+ getEnv "S3_CACHE_PUBLIC_BASE_URL"
175+ <&> cs . (\url -> if "/" `isSuffixOf` url then url else url <> "/")
176+ privateBucket <- Amazonka.BucketName . cs <$> getEnv "S3_CACHE_PRIVATE_BUCKET"
177+ cachePrivKeyFile <-
178+ lookupEnv "CACHE_PRIV_KEY_FILE"
179+ <&> fromMaybe "/run/secrets/cache-priv-key"
180+ cachePrivKeyName <- do
181+ cachePrivKey <- T.readFile cachePrivKeyFile
182+ case T.split (== ':') (cs cachePrivKey) of
183+ [name, _key] -> pure name
184+ _ -> Control.Exception.throwIO $ Control.Exception.ErrorCall "cannot parse cachePrivKey"
185+ let expiration = fromHours @Int 2
186+ let maxUploadSize = 4 * 2 ^ (30 :: Integer)
187+ isInNixosCacheMemoTable <- HashTables.new >>= newMVar
188+ pure
189+ $ S3CacheEnv
190+ { amazonkaEnv,
191+ publicBucket,
192+ publicBaseUrl,
193+ privateBucket,
194+ cachePrivKeyFile,
195+ cachePrivKeyName,
196+ expiration,
197+ maxUploadSize,
198+ isInNixosCacheMemoTable
199+ }
200+ actionServerUrl <- fromMaybe "action-runner2.garnix.io" <$> lookupEnv "GARNIX_ACTION_HOST"
201+ actionRunnerSshKey <- lookupEnv "GARNIX_ACTION_RUNNER_SSH_KEY" >>= maybe (pure "/run/secrets/garnix_action_runner_ssh") makeAbsolute
202+ curDir <- getCurrentDirectory
203+ let appPkPem = case readRsaPem appPkPem' of
204+ Right a -> a
205+ Left _ -> error "error reading GitHub App private key"
206+ mgr <- newTlsManager
207+ jwtKey <-
208+ lookupEnv "JWT_KEY"
209+ >>= maybe (BSC.readFile "/run/secrets/garnix-jwt-key") BSC.readFile
210+ <&> fromSecret . B64.decodeLenient
211+ burl <-
212+ lookupEnv "GARNIX_URL" >>= \case
213+ Nothing -> pure "https://app.garnix.io"
214+ Just u -> pure u
215+ hetznerTok <-
216+ lookupEnv "HETZNER_TOKEN"
217+ >>= maybe (BSC.readFile "/run/secrets/hetzner-token") (pure . cs)
218+ opensearchQueryUrl <- fromMaybe "https://opensearch.garnix.io/_msearch" <$> lookupEnv "OPENSEARCH_URL"
219+ opensearchPass <-
220+ lookupEnv "OPENSEARCH_API"
221+ >>= maybe (BSC.readFile "/run/secrets/opensearch-garnix") (pure . cs)
222+ dbPass <- do
223+ p <-
224+ lookupEnv "PGPASSWORD"
225+ >>= maybe (BSC.readFile "/run/secrets/database-password") (pure . cs)
226+ pure $ Data.ByteString.Char8.words p
227+ repoSecretsKeyPath <-
228+ RepoSecretsEncryptionKeyPath
229+ . fromMaybe "/run/secrets/repo-secrets-key"
230+ <$> lookupEnv "REPO_SECRETS_KEY_PATH"
231+ repoSecretsPubKey <-
232+ fmap RepoSecretsEncryptionPubKey
233+ $ lookupEnv "REPO_SECRETS_PUB_KEY"
234+ >>= maybe (T.readFile "/run/secrets/repo-secrets-key-pub") (pure . cs)
235+ dbConnectionPool <-
236+ ConnectionPool
237+ <$> Pool.newPool
238+ ( Pool.setNumStripes (Just 2)
239+ $ Pool.defaultPoolConfig
240+ (DB.getDBConnection dbPass)
241+ pgDisconnect
242+ 60
243+ 10
244+ )
245+ metrics <- registerMetrics
246+ nixEvalPool <- Garnix.Monad.Pool.newPool 50 metrics #evalQueueWaitTime #evalQueueLen
247+ s3UploadPool <- Garnix.Monad.Pool.newPool 100 metrics #s3QueueWaitTime #s3QueueLen
248+ stripe <- do
249+ publishableKey <-
250+ lookupEnv "STRIPE_PUBLISHABLE_KEY"
251+ >>= maybe (T.readFile "/run/secrets/stripe-publishable-key") (pure . cs)
252+ secretKey <-
253+ lookupEnv "STRIPE_SECRET_KEY"
254+ >>= maybe (T.readFile "/run/secrets/stripe-secret-key") (pure . cs)
255+ webhookSecret <-
256+ lookupEnv "STRIPE_WEBHOOK_SECRET"
257+ >>= maybe (T.readFile "/run/secrets/stripe-webhook-secret") (pure . cs)
258+ pure
259+ $ StripeEnv
260+ { publishableKey,
261+ secretKey,
262+ webhookSecret
263+ }
264+ Cradle.StdoutTrimmed hostname <- Cradle.run $ Cradle.cmd "hostname"
265+ mocks <- envMocks testFeatures
266+ featureFlagConfig <- getFeatureFlagConfig
267+ fodCheckPool <- Garnix.Monad.Pool.newPool 20 metrics #fodCheckQueueWaitTime #fodCheckQueueLen
268+ withDefaultLogger $ \defaultLogger -> do
269+ let env =
270+ Env
271+ { testFeatures = testFeatures,
272+ githubAppAuth = AppAuth appId appPkPem,
273+ githubAppId = appId,
274+ githubAppName = ghAppName,
275+ githubClientSecret = ghClientSecret,
276+ githubClientId = ghClientId,
277+ buildLogsReportingPort = buildLogsReportingPort,
278+ workingDir = curDir,
279+ nixXdgCacheDir = Nothing,
280+ userNixConfig = defaultNixConfig,
281+ githubWebhookSecret = ghK,
282+ githubInterface = realGithubInterface,
283+ hetznerInterface = realHetznerInterface,
284+ serverPoolConfig =
285+ [ (I2x4, 10),
286+ (I4x8, 2),
287+ (I8x16, 1),
288+ (I16x32, 1)
289+ ],
290+ cookieSettings =
291+ defaultCookieSettings
292+ { cookieXsrfSetting = Nothing,
293+ cookieIsSecure = if DevApi `elem` testFeatures then NotSecure else Secure
294+ },
295+ jwtSettings = defaultJWTSettings jwtKey,
296+ repoSecretsEncryptionKeyPath = repoSecretsKeyPath,
297+ repoSecretsEncryptionPubKey = repoSecretsPubKey,
298+ dbConn = dbConnectionPool,
299+ manager = mgr,
300+ baseUrl = cs burl,
301+ logger = defaultLogger,
302+ buildLogsDir = buildLogsDir',
303+ hetznerToken = hetznerTok,
304+ opensearchQueryUrl = opensearchQueryUrl,
305+ opensearchPassword = opensearchPass,
306+ sshUserHostingKeys = sshKeys,
307+ s3CacheEnv,
308+ action =
309+ ActionEnv
310+ { runnerHost = cs actionServerUrl,
311+ runnerSshKey = cs actionRunnerSshKey,
312+ timeoutDuration = fromHours @Int 2
313+ },
314+ nixEvalPool = nixEvalPool,
315+ s3UploadPool = s3UploadPool,
316+ stripe = stripe,
317+ mocks = mocks,
318+ spanCtx = [],
319+ metrics = metrics,
320+ emptyDir = emptyDir',
321+ hostname = hostname,
322+ githubLogDebounceDuration = fromSeconds @Int 15,
323+ featureFlagConfig,
324+ fodCheckPool
325+ }
326+ action env
327+
328+runWith :: Options -> IO ()
329+runWith opts = do
330+ hSetBuffering stdout LineBuffering
331+ hSetBuffering stderr LineBuffering
332+ testFeatures <- case mapM parseTestFeature $ enable opts of
333+ Right testFeatures -> pure $ Set.fromList testFeatures
334+ Left err -> Control.Exception.throwIO $ Control.Exception.ErrorCall $ cs err
335+ hPutStrLn stderr $ "Test features: " <> if Set.null testFeatures then "none" else T.intercalate ", " (fmap show (toList testFeatures))
336+ do
337+ n <- getNumProcessors
338+ hPutStrLn stderr $ "number of processors: " <> show n
339+ n <- getNumCapabilities
340+ hPutStrLn stderr $ "number of capabilities: " <> show n
341+ withEnv
342+ testFeatures
343+ (Garnix.buildLogsDir opts)
344+ (Garnix.buildLogsReportingPort opts)
345+ $ \env -> do
346+ serveMetrics (Garnix.metricsPort opts) (env ^. #metrics)
347+ void . forkIO . void . runM env $ forever (stopUnusedServers' *> threadDelay (fromMinutes @Int 5))
348+ if Garnix.provisionServerPool opts
349+ then void $ runM env ServerPool.initializeProvisioningPool
350+ else hPutStrLn stderr "Not provisioning server pool"
351+ let settings =
352+ Warp.defaultSettings
353+ & Warp.setPort (port opts)
354+ & Warp.setBeforeMainLoop
355+ ( do
356+ hPutStrLn stderr $ "Listening on port " <> show (port opts)
357+ void notifyReady
358+ )
359+ Warp.runSettings settings $ Garnix.toApplication env
360+ where
361+ stopUnusedServers' :: M ()
362+ stopUnusedServers' = stopUnusedServers `catchError` (\e -> log Error $ "stopUnusedServer error: " <> show e)
363+
364+type ContextList =
365+ '[ JWTSettings,
366+ CookieSettings,
367+ GitHubKey CheckSuiteEvent,
368+ GitHubKey CheckRunEvent,
369+ GitHubKey PullRequestEvent,
370+ GitHubKey PushEvent,
371+ Stripe.Concepts.WebhookSecretKey
372+ ]
373+
374+toApplication :: Env -> Application
375+toApplication env =
376+ let ghKey :: GitHubKey a
377+ ghKey = gitHubKey . pure $ env ^. #githubWebhookSecret
378+ context :: Context ContextList
379+ context =
380+ (env ^. #jwtSettings)
381+ :. (env ^. #cookieSettings)
382+ :. ghKey
383+ :. ghKey
384+ :. ghKey
385+ :. ghKey
386+ :. Stripe.Concepts.textToWebhookSecretKey (env ^. #stripe . #webhookSecret)
387+ :. EmptyContext
388+ contextProxy :: Proxy ContextList
389+ contextProxy = Proxy
390+ in gzip gzipSettings
391+ $ logRequestsMiddleware
392+ env
393+ ( \requestTraceId ->
394+ serveWithContext api context
395+ $ hoistServerWithContext api contextProxy (mToHandler env requestTraceId) (toServant wholeAPI)
396+ )
397+
398+gzipSettings :: GzipSettings
399+gzipSettings = defaultGzipSettings {gzipFiles = GzipPreCompressed GzipIgnore}
400+
401+mToHandler :: Env -> RequestTraceId -> M a -> Servant.Handler a
402+mToHandler env requestTraceId action = do
403+ r <- liftIO $ runM env $ do
404+ withSpan requestTraceId $ do
405+ logThrownErrors $ do
406+ turnRuntimeExceptionsIntoMonadicErrors $ do
407+ withRecachedFeatureFlags $ do
408+ action
409+ case r of
410+ Right v -> pure v
411+ Left e -> throwError $ servantizeError e
412+ where
413+ turnRuntimeExceptionsIntoMonadicErrors :: M a -> M a
414+ turnRuntimeExceptionsIntoMonadicErrors action =
415+ action
416+ `Safe.catch` ( \(e :: SomeException) ->
417+ throw $ UncaughtRuntimeException (show e)
418+ )
backend/src/Garnix/API.hsnew file+115
@@ -0,0 +1,115 @@
1+module Garnix.API where
2+
3+import Autodocodec.Schema (JSONSchema)
4+import Data.Text qualified as T
5+import Garnix.API.Account
6+import Garnix.API.Auth
7+import Garnix.API.Badges
8+import Garnix.API.Builds
9+import Garnix.API.Cache
10+import Garnix.API.Commits
11+import Garnix.API.ConfigSchema (garnixConfigJsonSchema)
12+import Garnix.API.Dev (DevAPI, devAPI)
13+import Garnix.API.GhWebhooks
14+import Garnix.API.Health
15+import Garnix.API.Hosts
16+import Garnix.API.Keys
17+import Garnix.API.Modules
18+import Garnix.API.Runs (RunAPI, runAPI)
19+import Garnix.API.Stripe (StripeWebhookAPI, stripeWebhookAPI)
20+import Garnix.DB qualified as DB
21+import Garnix.Monad
22+import Garnix.Prelude
23+import Garnix.Types
24+import Servant
25+import Servant.Auth.Server
26+
27+type CT = '[JSON]
28+
29+api :: Proxy (ToServant WholeAPI AsApi)
30+api = genericApi (Proxy :: Proxy WholeAPI)
31+
32+data WholeAPI r = WholeAPI
33+ { events ::
34+ r
35+ :- "api"
36+ :> "events"
37+ :> ( "github" :> ToServantApi GhWebhookAPI
38+ :<|> "stripe" :> StripeWebhookAPI
39+ ),
40+ account :: r :- "api" :> "account" :> Auth '[JWT, Cookie] AuthJwtPayload :> ToServantApi AccountAPI,
41+ build :: r :- "api" :> "build" :> Auth '[JWT, Cookie] AuthJwtPayload :> ToServantApi BuildAPI,
42+ commit :: r :- "api" :> "commits" :> Auth '[JWT, Cookie] AuthJwtPayload :> ToServantApi CommitAPI,
43+ run :: r :- "api" :> "run" :> Auth '[JWT, Cookie] AuthJwtPayload :> ToServantApi RunAPI,
44+ modules :: r :- "api" :> "modules" :> Auth '[JWT, Cookie] AuthJwtPayload :> ToServantApi ModulesAPI,
45+ dev :: r :- "api" :> "dev" :> ToServantApi DevAPI,
46+ hosts :: r :- "api" :> "hosts" :> ToServantApi HostsAPI,
47+ keys :: r :- "api" :> "keys" :> Capture "owner" GhRepoOwner :> Capture "repo" GhRepoName :> "repo-key.public" :> Get '[PlainText] PublicKey,
48+ actionKeys :: r :- "api" :> "keys" :> Capture "owner" GhRepoOwner :> Capture "repo" GhRepoName :> "actions" :> Capture "action" PackageName :> "key.public" :> Get '[PlainText] PublicKey,
49+ login :: r :- "api" :> "login" :> ToServantApi LoginAPI,
50+ signup :: r :- "api" :> "signup" :> ToServantApi SignupAPI,
51+ whoami :: r :- "api" :> "whoami" :> Auth '[JWT, Cookie] AuthJwtPayload :> Get '[JSON] (Maybe UserDto),
52+ authJwt :: r :- "api" :> "auth" :> "jwt" :> ToServantApi AuthJwtAPI,
53+ config :: r :- "api" :> "config" :> Get '[JSON] FrontendConfig,
54+ badges :: r :- "api" :> "badges" :> Capture "owner" GhRepoOwner :> Capture "repo" GhRepoName :> QueryParam "branch" Branch :> Get '[JSON] Badge,
55+ waitlist :: r :- "api" :> "waitlist" :> ReqBody '[JSON] Email :> Post '[JSON] (),
56+ cache :: r :- "api" :> "cache" :> ToServantApi CacheAPI,
57+ garnixConfigSchema :: r :- "api" :> "garnix-config-schema.json" :> Get '[JSON] JSONSchema,
58+ health :: r :- "api" :> "health" :> ToServantApi HealthAPI
59+ }
60+ deriving stock (Generic)
61+
62+data ProjectAPI r = ProjectAPI
63+ { get ::
64+ r
65+ :- Capture "gh_owner" Text
66+ :> Capture "gh_repo" Text
67+ :> "commit"
68+ :> Capture "commit" CommitHash
69+ :> Get CT (),
70+ post ::
71+ r
72+ :- Capture "gh_owner" Text
73+ :> Capture "gh_repo" Text
74+ :> "commit"
75+ :> Capture "commit" CommitHash
76+ :> QueryParam "token" GhToken
77+ :> Post '[JSON] RunResult
78+ }
79+ deriving stock (Generic)
80+
81+wholeAPI :: WholeAPI (AsServerT M)
82+wholeAPI =
83+ WholeAPI
84+ { events = toServant ghWebhookAPI :<|> stripeWebhookAPI,
85+ account = toServant . accountAPI,
86+ dev = devAPI,
87+ login = toServant loginAPI,
88+ signup = toServant signupAPI,
89+ whoami = whoAmIAPI,
90+ authJwt = toServant authJwtAPI,
91+ hosts = toServant hostsAPI,
92+ keys = Garnix.API.Keys.getRepoPublicKey,
93+ actionKeys = Garnix.API.Keys.getActionPublicKey,
94+ config = getConfig,
95+ build = toServant . buildAPI,
96+ commit = toServant . commitAPI,
97+ run = toServant . runAPI,
98+ modules = toServant . modulesAPI,
99+ badges = badgesAPI,
100+ waitlist = waitlistAPI,
101+ cache = toServant cacheAPI,
102+ garnixConfigSchema = pure garnixConfigJsonSchema,
103+ health = toServant healthAPI
104+ }
105+
106+getConfig :: M FrontendConfig
107+getConfig = do
108+ ghAppName <- view #githubAppName
109+ pure $ FrontendConfig {_frontendConfigGithubAppName = ghAppName}
110+
111+waitlistAPI :: Email -> M ()
112+waitlistAPI email = do
113+ let isValid = '@' `T.elem` getEmail email
114+ unless isValid $ throw InvalidEmail
115+ DB.addToWaitlist email
backend/src/Garnix/API/Account.hsnew file+502
@@ -0,0 +1,502 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+
3+module Garnix.API.Account where
4+
5+import Control.Concurrent.Async.Lifted
6+import Control.Lens
7+import Data.Map.Strict qualified as Map
8+import Data.Maybe
9+import Data.Row (Rec, (.==), type (.+), type (.==))
10+import Database.PostgreSQL.Typed (pgSQL)
11+import Garnix.AccessToken
12+import Garnix.AccessToken.Types
13+import Garnix.DB qualified as DB
14+import Garnix.Duration
15+import Garnix.Entitlements (ProductToken (..), getHosting, getPlanByProductToken, getPlans, setExtraUsageLimits)
16+import Garnix.Entitlements qualified as Entitlements
17+import Garnix.GithubInterface.Types
18+import Garnix.Hosting.Helpers (getBranchDeploymentBillingLineItems, groupIdentifierToLineItemDescription)
19+import Garnix.Monad
20+import Garnix.Prelude
21+import Garnix.StripeLib (AddressDto, ClientSecret, PriceId (PriceId), TaxCalculationDto, UnitAmount (UnitAmount), createInvoiceItem, taxCalculation, unitAmountFromCost)
22+import Garnix.StripeLib qualified as StripeLib
23+import Garnix.Types hiding (Admin, installationId)
24+import Servant (Put)
25+import Servant.Auth.Server
26+
27+data AccountAPI route = AccountAPI
28+ { _accountAPIUsage :: route :- "usage" :> Get '[JSON] UsageOverview,
29+ _accountAPIOrgUsage :: route :- "usage" :> Capture "org" GhRepoOwner :> Get '[JSON] OrgUsage,
30+ _accountAPISetUsageLimits :: route :- "usage" :> Capture "org" GhRepoOwner :> ReqBody '[JSON] ExtraUsageLimits :> Put '[JSON] NoContent,
31+ _accountAPIUpgradeOption :: route :- "upgrade_option" :> QueryParam "product_token" ProductToken :> Get '[JSON] UpgradeOption,
32+ _accountAPITaxes :: route :- "taxes" :> ReqBody '[JSON] TaxesRequest :> Post '[JSON] TaxCalculationDto,
33+ _accountAPIEnabledRepos :: route :- "repos" :> Get '[JSON] EnabledRepos,
34+ _accountAPISubscribe :: route :- "subscribe" :> ReqBody '[JSON] SubscribeRequestBody :> Post '[JSON] SubscribeResponse,
35+ _accountAPIUnsubscribe :: route :- "subscription" :> Capture "org" GhRepoOwner :> Delete '[JSON] NoContent,
36+ _accountAPIGetAccessTokens :: route :- "tokens" :> Get '[JSON] GetTokensResponseBody,
37+ _accountAPICreateAccessToken :: route :- "tokens" :> ReqBody '[JSON] CreateTokenRequestBody :> Post '[JSON] CreateTokenResponseBody,
38+ _accountAPIRevokeAccessToken :: route :- "tokens" :> Capture "tokenId" Int64 :> Delete '[JSON] NoContent
39+ }
40+ deriving (Generic)
41+
42+accountAPI :: AuthResult AuthJwtPayload -> AccountAPI (AsServerT M)
43+accountAPI user =
44+ AccountAPI
45+ { _accountAPIUsage = usageOverview user,
46+ _accountAPIOrgUsage = orgUsage user,
47+ _accountAPISetUsageLimits = setUsageLimits user,
48+ _accountAPIUpgradeOption = getUpgradeOptionByToken,
49+ _accountAPITaxes = taxes user,
50+ _accountAPIEnabledRepos = enabledReposOf user,
51+ _accountAPISubscribe = createSubscription user,
52+ _accountAPIUnsubscribe = cancelOrgSubscription user,
53+ _accountAPIGetAccessTokens = getAccessTokens user,
54+ _accountAPICreateAccessToken = createAccessToken user,
55+ _accountAPIRevokeAccessToken = revokeAccessToken user
56+ }
57+
58+data UsageOverview = UsageOverview
59+ { _usageOverviewByOrg :: Map.Map GhRepoOwner OrgUsage
60+ }
61+ deriving stock (Eq, Show, Generic)
62+
63+instance ToJSON UsageOverview where
64+ toEncoding = ourToEncoding
65+ toJSON = ourToJSON
66+
67+data OrgUsage = OrgUsage
68+ { _orgUsagePlan :: ProductPlan,
69+ _orgUsageCiTime :: Duration,
70+ _orgUsagePrDeploymentTime :: Duration,
71+ _orgUsageBranchDeploymentHosts :: Int64,
72+ _orgUsageUpgradeOption :: Maybe UpgradeOption,
73+ _orgUsageInstallationStatus :: InstallationStatus
74+ }
75+ deriving stock (Eq, Show, Generic)
76+
77+instance ToJSON OrgUsage where
78+ toEncoding = ourToEncoding
79+ toJSON = ourToJSON
80+
81+getOrgsUserIsAdminIn :: User -> GhToken -> M [GhRepoOwner]
82+getOrgsUserIsAdminIn user token =
83+ (GhRepoOwner (user ^. githubLogin) :)
84+ . map organizationName
85+ . filter (\membership -> role membership == Admin)
86+ <$> getInstalledOrgs token
87+
88+getUsageForOrg :: Map.Map GhRepoOwner Duration -> Map.Map GhRepoOwner ProductPlan -> GhRepoOwner -> M OrgUsage
89+getUsageForOrg usage plans org = do
90+ prDeploymentTime <- DB.getPrDeployDurationForOwner org
91+ branchDeployments <- sum <$> DB.getRunningBranchServersForOwner org
92+ upgradeOptions <- upgradeOptions org
93+ plan <- case Map.lookup org plans of
94+ Just plan -> pure plan
95+ Nothing -> throw $ OtherError "Impossible: plan missing for org passed into getPlans"
96+ installationStatus <- DB.getInstallationStatus org
97+ pure
98+ OrgUsage
99+ { _orgUsagePlan = plan,
100+ _orgUsageCiTime = fromMaybe emptyDuration $ Map.lookup org usage,
101+ _orgUsagePrDeploymentTime = prDeploymentTime,
102+ _orgUsageBranchDeploymentHosts = branchDeployments,
103+ _orgUsageUpgradeOption = upgradeOptions,
104+ _orgUsageInstallationStatus = installationStatus
105+ }
106+
107+usageOverview :: AuthResult AuthJwtPayload -> M UsageOverview
108+usageOverview (Authenticated (WebSession user ghToken)) = do
109+ orgs <- getOrgsUserIsAdminIn user ghToken
110+ usage <- DB.getCurrentMonthUsages (GhRepoOwner (user ^. githubLogin) : orgs)
111+ plans <- getPlans orgs
112+ map <- mkMapM orgs $ getUsageForOrg usage plans
113+ pure $ UsageOverview map
114+usageOverview _ = throw Unauthorized
115+
116+mkMapM :: (Monad m, Ord key) => [key] -> (key -> m value) -> m (Map.Map key value)
117+mkMapM keys f =
118+ Map.fromList
119+ <$> forM
120+ keys
121+ ( \key -> do
122+ value <- f key
123+ pure (key, value)
124+ )
125+
126+orgUsage :: AuthResult AuthJwtPayload -> GhRepoOwner -> M OrgUsage
127+orgUsage (Authenticated (WebSession user ghToken)) org = do
128+ orgsUserIsAdminIn <- getOrgsUserIsAdminIn user ghToken
129+ when (org `notElem` orgsUserIsAdminIn) $ throw NotFound
130+ usage <- DB.getCurrentMonthUsages [org]
131+ plans <- getPlans [org]
132+ getUsageForOrg usage plans org
133+orgUsage _ _ = throw Unauthorized
134+
135+setUsageLimits :: AuthResult AuthJwtPayload -> GhRepoOwner -> ExtraUsageLimits -> M NoContent
136+setUsageLimits (Authenticated (WebSession user ghToken)) org newLimits = do
137+ orgsUserIsAdminIn <- getOrgsUserIsAdminIn user ghToken
138+ when (org `notElem` orgsUserIsAdminIn) $ throw Unauthorized
139+ plan <- Entitlements.getPlan org
140+ unless (plan ^. isPaid) $ do
141+ throw $ BadRequest "User does not have an active plan"
142+ setExtraUsageLimits org newLimits
143+ pure NoContent
144+setUsageLimits _ _ _ = throw Unauthorized
145+
146+data UpgradeOption = UpgradeOption
147+ { _upgradeOptionApiKey :: Text,
148+ _upgradeOptionProductToken :: ProductToken,
149+ _upgradeOptionCurrency :: Text,
150+ _upgradeOptionUnitAmount :: Int64,
151+ _upgradeOptionPlan :: ProductPlan
152+ }
153+ deriving stock (Eq, Show, Generic)
154+
155+instance ToJSON UpgradeOption where
156+ toEncoding = ourToEncoding
157+ toJSON = ourToJSON
158+
159+getUpgradeOptionByToken :: Maybe ProductToken -> M UpgradeOption
160+getUpgradeOptionByToken token = case token of
161+ Nothing -> throw $ BadRequest "query parameter missing: token"
162+ Just token -> do
163+ apiKey <- view $ #stripe . #publishableKey
164+ result :: [UpgradeOption] <- do
165+ result <-
166+ DB.pgQuery
167+ [pgSQL|
168+ SELECT
169+ name,
170+ title,
171+ token,
172+ price_id,
173+ description,
174+ ci_minutes,
175+ pr_hosting,
176+ hosting,
177+ packages_per_flake,
178+ package_eval_timeout_in_minutes,
179+ package_build_timeout_in_minutes
180+ FROM products
181+ WHERE
182+ token = ${getProductToken token} AND
183+ price_id IS NOT NULL
184+ |]
185+ forM result $ \case
186+ ( name,
187+ title,
188+ Just token,
189+ Just priceId,
190+ description,
191+ ciMinutes :: Maybe Int64,
192+ prHosting :: Maybe Int64,
193+ hosting,
194+ packagesPerFlake,
195+ packageEvalTimeout :: Maybe Int16,
196+ packageBuildTimeout :: Maybe Int16
197+ ) -> do
198+ stripePrice <- StripeLib.getPrice (PriceId priceId)
199+ pure
200+ $ UpgradeOption
201+ { _upgradeOptionApiKey = apiKey,
202+ _upgradeOptionProductToken = ProductToken token,
203+ _upgradeOptionCurrency = stripePrice ^. #currency,
204+ _upgradeOptionUnitAmount = stripePrice ^. #unit_amount,
205+ _upgradeOptionPlan =
206+ ProductPlan
207+ { _productPlanDisplayName = fromMaybe name title,
208+ _productPlanDescription = description,
209+ _productPlanBaseCiTime = maybe emptyDuration fromMinutes ciMinutes,
210+ _productPlanMaximumPrDeploymentTime = maybe emptyDuration fromMinutes prHosting,
211+ _productPlanIncludedBranchDeploymentHosts = fromMaybe 0 hosting,
212+ _productPlanMaximumPackagesPerFlake = fromMaybe 100 packagesPerFlake,
213+ _productPlanPackageEvaluationTimeout = fromMaybe 30 packageEvalTimeout,
214+ _productPlanPackageBuildTimeout = fromMaybe 120 packageBuildTimeout,
215+ _productPlanExtraUsage = emptyUsageLimits,
216+ _productPlanIsPaid = True
217+ }
218+ }
219+ _ -> throw $ OtherError "impossible"
220+ case result of
221+ [option] -> pure option
222+ [] -> throw NotFound
223+ _ -> throw $ OtherError "impossible"
224+
225+upgradeOptions :: GhRepoOwner -> M (Maybe UpgradeOption)
226+upgradeOptions repoOwner = do
227+ res ::
228+ [ ( Text,
229+ Maybe Text,
230+ Maybe Text,
231+ Maybe Text,
232+ Maybe Int64,
233+ Maybe Int64,
234+ Maybe Int64,
235+ Maybe Int32,
236+ Maybe Text,
237+ Maybe Int16,
238+ Maybe Int16
239+ )
240+ ] <-
241+ DB.pgQuery
242+ [pgSQL|
243+ SELECT
244+ name,
245+ title,
246+ description,
247+ price_id,
248+ hosting,
249+ pr_hosting,
250+ ci_minutes,
251+ packages_per_flake,
252+ token,
253+ package_eval_timeout_in_minutes,
254+ package_build_timeout_in_minutes
255+ FROM products
256+ WHERE
257+ visible
258+ AND token IS NOT NULL
259+ AND price_id IS NOT NULL
260+ AND NOT EXISTS
261+ (SELECT 1 FROM repo_owner_has_product WHERE
262+ repo_owner_has_product.product = products.name
263+ AND repo_owner_has_product.repo_owner = ${repoOwner}
264+ )
265+ |]
266+ case res of
267+ ( name,
268+ title,
269+ description,
270+ Just priceId,
271+ branchHosting,
272+ prDeployMinutes,
273+ ciMinutes,
274+ packagesPerFlake,
275+ Just token,
276+ packageEvalTimeout,
277+ packageBuildTimeout
278+ )
279+ : _ -> do
280+ stripePrice <- StripeLib.getPrice (PriceId priceId)
281+ apiKey <- view $ #stripe . #publishableKey
282+ pure
283+ $ Just
284+ $ UpgradeOption
285+ apiKey
286+ (ProductToken token)
287+ (stripePrice ^. #currency)
288+ (stripePrice ^. #unit_amount)
289+ ( ProductPlan
290+ { _productPlanDisplayName = fromMaybe name title,
291+ _productPlanDescription = description,
292+ _productPlanBaseCiTime = maybe emptyDuration fromMinutes ciMinutes,
293+ _productPlanMaximumPrDeploymentTime = maybe emptyDuration fromMinutes prDeployMinutes,
294+ _productPlanIncludedBranchDeploymentHosts = fromMaybe 0 branchHosting,
295+ _productPlanMaximumPackagesPerFlake = fromMaybe 100 packagesPerFlake,
296+ _productPlanPackageEvaluationTimeout = fromMaybe 30 packageEvalTimeout,
297+ _productPlanPackageBuildTimeout = fromMaybe 120 packageBuildTimeout,
298+ _productPlanExtraUsage = emptyUsageLimits,
299+ _productPlanIsPaid = True
300+ }
301+ )
302+ _ : _ -> throw $ OtherError "impossible"
303+ [] -> pure Nothing
304+
305+type TaxesRequest =
306+ Rec
307+ ( "unit_amount" .== Int64
308+ .+ "currency" .== String
309+ .+ "address" .== AddressDto
310+ )
311+
312+taxes :: AuthResult AuthJwtPayload -> TaxesRequest -> M TaxCalculationDto
313+taxes user request = case user of
314+ Authenticated _ -> taxCalculation (request ^. #unit_amount) (request ^. #currency) (request ^. #address)
315+ _ -> throw Unauthorized
316+
317+type SubscribeRequestBody = Rec ("product_token" .== ProductToken .+ "github_org" .== GhRepoOwner)
318+
319+type SubscribeResponse = Rec ("client_secret" .== ClientSecret)
320+
321+createSubscription :: AuthResult AuthJwtPayload -> SubscribeRequestBody -> M SubscribeResponse
322+createSubscription (Authenticated (WebSession user ghToken)) body = do
323+ orgs <- getOrgsUserIsAdminIn user ghToken
324+ when (body ^. #github_org `notElem` orgs) $ throw Forbidden
325+ mCustomerId <- DB.getInstallationStripeCustomer (body ^. #github_org)
326+ customer <-
327+ case mCustomerId of
328+ Nothing -> do
329+ customer <-
330+ StripeLib.createCustomer
331+ (body ^. #github_org)
332+ (StripeLib.Name $ user ^. githubLogin . to getGhLogin)
333+ (user ^. email)
334+ DB.setStripeCustomerId (body ^. #github_org) (customer ^. #id)
335+ pure $ customer ^. #id
336+ Just id -> pure id
337+ plan <- getPlanByProductToken (body ^. #product_token)
338+ let throwNotFound = throw $ OtherError $ "No price_id found for product with product_token: " <> show (body ^. #product_token)
339+ case plan of
340+ Nothing -> throwNotFound
341+ (Just (_, Nothing, _)) -> throwNotFound
342+ Just (name, Just priceId, plan') -> do
343+ subscription <- StripeLib.createSubscription customer priceId name (plan' ^. displayName)
344+ pure
345+ ( #client_secret
346+ .== subscription
347+ ^. #latest_invoice
348+ . #payment_intent
349+ . #client_secret
350+ )
351+createSubscription _ _ = throw Unauthorized
352+
353+cancelOrgSubscription :: AuthResult AuthJwtPayload -> GhRepoOwner -> M NoContent
354+cancelOrgSubscription (Authenticated (WebSession user ghToken)) org = do
355+ orgsUserIsAdminIn <- getOrgsUserIsAdminIn user ghToken
356+ when (org `notElem` orgsUserIsAdminIn) $ throw NotFound
357+ mCustomer <- DB.getInstallationStripeCustomer org
358+ case mCustomer of
359+ Nothing -> throw NotFound
360+ Just customerId -> do
361+ withTextSpans
362+ [ ("stripe_customer_id", show customerId),
363+ ("repo_owner", show org)
364+ ]
365+ $ do
366+ subscriptions <- (^.. #data . traverse . #id) <$> StripeLib.listSubscriptions customerId
367+ case subscriptions of
368+ [] -> do
369+ log Critical "Failed to unsubscribe: No subscription found"
370+ throw NotFound
371+ _ : _ : _ -> do
372+ log Critical "Failed to unsubscribe: More than one subscription found"
373+ throw NotFound
374+ [subscription] -> do
375+ StripeLib.cancelSubscription subscription
376+ DB.setRequestedCancellation org True
377+ pure NoContent
378+cancelOrgSubscription _ _ = throw Unauthorized
379+
380+handleSubscriptionAdded :: StripeLib.SubscriptionCreatedOrUpdatedEvent -> M ()
381+handleSubscriptionAdded (StripeLib.SubscriptionCreatedOrUpdatedEvent _eventType customerId priceId status periodStart periodEnd) = do
382+ when (status == StripeLib.SubscriptionStatusActive) $ do
383+ repoOwner <- getGhRepoOwner customerId
384+ Entitlements.addProductByPriceId repoOwner priceId
385+ DB.updatePeriodForCustomer customerId periodStart periodEnd
386+ where
387+ getGhRepoOwner :: CustomerId -> M GhRepoOwner
388+ getGhRepoOwner customerId = do
389+ res <-
390+ DB.pgQuery
391+ [pgSQL|
392+ SELECT repo_owner FROM installations
393+ WHERE stripe_customer = ${getCustomerId customerId}
394+ |]
395+ case res of
396+ [] -> throw $ OtherError $ "Cannot find stripe customer " <> getCustomerId customerId
397+ [id'] -> pure id'
398+ _ -> throw $ OtherError "impossible: stripe_customer should be unique"
399+
400+handleInvoiceCreated :: StripeLib.InvoiceCreatedEvent -> M ()
401+handleInvoiceCreated (StripeLib.InvoiceCreatedEvent customerId invoiceId reason periodStart periodEnd) = do
402+ withTextSpans
403+ [ ("stripe_customer_id", show customerId),
404+ ("stripe_invoice_id", show invoiceId),
405+ ("billing_reason", show reason)
406+ ]
407+ $ do
408+ log Notice "Invoice created"
409+ when (reason == StripeLib.SubscriptionCycle) $ do
410+ mRepoOwner <- DB.getRepoOwnerForStripeCustomer customerId
411+ case mRepoOwner of
412+ Nothing -> do
413+ let err = "Invoice was created for an existing subscription, but we are missing the installation"
414+ log Critical err
415+ throw $ OtherError err
416+ Just repoOwner -> do
417+ plan <- Entitlements.getPlan repoOwner
418+ billOverage "CI minutes" (UnitAmount "0.6") (plan ^. baseCiTime)
419+ =<< DB.getCurrentMonthUsage repoOwner
420+ billOverage "PR deploy minutes" (UnitAmount "0.6") (plan ^. maximumPrDeploymentTime)
421+ =<< DB.getPrDeployDurationForOwner repoOwner
422+ numFreeServers <- (^. #planIncludedBranchDeploymentHosts) <$> getHosting repoOwner
423+ branchDeploymentLineItems <- getBranchDeploymentBillingLineItems numFreeServers periodStart periodEnd repoOwner
424+ forM_ branchDeploymentLineItems $ \lineItem -> do
425+ let groupName =
426+ groupIdentifierToLineItemDescription (lineItem ^. #group)
427+ <> if lineItem ^. #includedInPlan then " (included in plan)" else ""
428+ createInvoiceItem customerId invoiceId groupName (unitAmountFromCost $ lineItem ^. #cost) 1
429+ where
430+ billOverage :: Text -> UnitAmount -> Duration -> Duration -> M ()
431+ billOverage name unitAmount basePlanTime currentUsage = do
432+ log Notice $ "Plan " <> name <> " time has " <> show (toMinutes basePlanTime) <> " min. Current usage is " <> show (toMinutes currentUsage) <> " min"
433+ let usedExtraMinutes :: Int64 = floor $ toMinutes $ currentUsage `subtractDuration` basePlanTime
434+ when (usedExtraMinutes > 0) $ do
435+ log Notice $ "Adding " <> name <> " invoice item of " <> show usedExtraMinutes <> " extra minutes"
436+ createInvoiceItem customerId invoiceId ("Extra " <> name <> " above plan") unitAmount usedExtraMinutes
437+
438+data GetTokensResponseBody = GetTokensResponseBody
439+ { _getTokensResponseBodyTokens :: [AccessTokenMetadata]
440+ }
441+ deriving stock (Eq, Show, Generic)
442+
443+instance ToJSON GetTokensResponseBody where
444+ toJSON = ourToJSON
445+
446+data CreateTokenRequestBody = CreateTokenRequestBody
447+ { _createTokenRequestBodyName :: Text,
448+ _createTokenRequestBodyScopes :: Maybe AccessTokenScopes
449+ }
450+ deriving stock (Eq, Show, Generic)
451+
452+instance FromJSON CreateTokenRequestBody where
453+ parseJSON = ourParseJSON
454+
455+data CreateTokenResponseBody = CreateTokenResponseBody
456+ { _createTokenResponseBodyToken :: AccessToken
457+ }
458+ deriving stock (Eq, Show, Generic)
459+
460+instance ToJSON CreateTokenResponseBody where
461+ toJSON = ourToJSON
462+
463+getAccessTokens :: AuthResult AuthJwtPayload -> M GetTokensResponseBody
464+getAccessTokens (Authenticated ((^. #user) -> user)) = GetTokensResponseBody <$> DB.getAccessTokensForUser (user ^. id)
465+getAccessTokens _ = throw Unauthorized
466+
467+-- For backwards compatability during the first deploy, if `scopes` is not provided, we default to just a cache scope.
468+fallbackAccessTokenScopes :: AccessTokenScopes
469+fallbackAccessTokenScopes = AccessTokenScopes {api = False, cache = True}
470+
471+createAccessToken :: AuthResult AuthJwtPayload -> CreateTokenRequestBody -> M CreateTokenResponseBody
472+createAccessToken (Authenticated (WebSession user _)) (CreateTokenRequestBody name (fromMaybe fallbackAccessTokenScopes -> scopes)) = do
473+ when (scopes == AccessTokenScopes {api = False, cache = False}) $ do
474+ throw $ BadRequest "no scopes enabled"
475+ accessToken <- generateToken (user ^. id) name scopes
476+ pure $ CreateTokenResponseBody accessToken
477+createAccessToken (Authenticated (ApiSession _)) _ = throw $ ForbiddenWithMessage "This endpoint is not available through the programmatic api."
478+createAccessToken _ _ = throw Unauthorized
479+
480+revokeAccessToken :: AuthResult AuthJwtPayload -> Int64 -> M NoContent
481+revokeAccessToken (Authenticated ((^. #user) -> user)) tokenId = do
482+ DB.deleteAccessTokenForUser (user ^. id) tokenId
483+ pure NoContent
484+revokeAccessToken _ _ = throw Unauthorized
485+
486+enabledReposOf :: AuthResult AuthJwtPayload -> M EnabledRepos
487+enabledReposOf (Authenticated (WebSession _ ghToken)) = do
488+ installationIds <- getInstallations ghToken
489+ repos <- forConcurrently installationIds $ \id ->
490+ getReposInInstallationAccessibleTo id ghToken
491+ return
492+ $ EnabledRepos
493+ $ mconcat repos
494+enabledReposOf _ = throw Unauthorized
495+
496+data EnabledRepos = EnabledRepos {_enabledReposRepos :: [Text]}
497+ deriving stock (Eq, Show, Generic)
498+
499+instance ToJSON EnabledRepos where
500+ toJSON = ourToJSON
501+
502+makeFields ''UpgradeOption
backend/src/Garnix/API/Auth.hsnew file+335
@@ -0,0 +1,335 @@
1+module Garnix.API.Auth where
2+
3+import Control.Lens
4+import Garnix.AccessToken
5+import Garnix.AccessToken.Types
6+import Garnix.DB qualified as DB
7+import Garnix.Monad
8+import Garnix.ParseHttpBasicAuth
9+import Garnix.Prelude
10+import Garnix.Types hiding (login)
11+import GitHub qualified as GH
12+import Network.OAuth2 qualified as OA
13+import Servant.Auth.Server
14+ ( Auth,
15+ AuthResult (Authenticated),
16+ Cookie,
17+ JWT,
18+ acceptLogin,
19+ clearSession,
20+ )
21+import Servant.Auth.Server.Internal.JWT (makeJWT)
22+import Web.Cookie
23+
24+data UserDto = UserDto
25+ { _userDtoUsername :: GhLogin,
26+ _userDtoEmail :: Email,
27+ _userDtoIsAdmin :: Bool
28+ }
29+ deriving stock (Generic)
30+
31+instance ToJSON UserDto where
32+ toEncoding = ourToEncoding
33+ toJSON = ourToJSON
34+
35+whoAmIAPI :: AuthResult AuthJwtPayload -> M (Maybe UserDto)
36+whoAmIAPI (Authenticated ((^. #user) -> user)) = do
37+ pure
38+ $ Just
39+ $ UserDto
40+ (user ^. githubLogin)
41+ (user ^. email)
42+ (user ^. subscriptionType == Admin)
43+whoAmIAPI _ = pure Nothing
44+
45+data AuthJwtAPI route = AuthJwtAPI
46+ { jwt :: route :- Header "Authorization" Text :> Auth '[JWT, Cookie] AuthJwtPayload :> Post '[JSON] AuthJwtDto
47+ }
48+ deriving stock (Generic)
49+
50+data AuthJwtDto = AuthJwtDto
51+ { token :: Text,
52+ expiresAt :: UTCTime
53+ }
54+ deriving stock (Generic)
55+ deriving anyclass (ToJSON)
56+
57+authJwtAPI :: AuthJwtAPI (AsServerT M)
58+authJwtAPI =
59+ AuthJwtAPI
60+ { jwt = getJwt
61+ }
62+
63+getJwt :: Maybe Text -> AuthResult AuthJwtPayload -> M AuthJwtDto
64+getJwt mAuthHeader authResult = do
65+ authHeader <- case (mAuthHeader, authResult) of
66+ (_, Authenticated _) -> throw $ ForbiddenWithMessage "Creating JWTs is only allowed with the api access tokens."
67+ (Nothing, _) -> throw $ UnauthorizedWithMessage "Missing Authorization header"
68+ (Just authHeader, _) -> pure authHeader
69+ (username, password) <- case parseBasicAuth authHeader of
70+ Left err -> throw $ BadRequest $ cs err
71+ Right creds -> pure creds
72+ user <-
73+ withError
74+ ( errLens %~ \case
75+ NoSuchUser _ -> Unauthorized
76+ err -> err
77+ )
78+ $ DB.getUser
79+ $ GhLogin username
80+ isValid <- isAccessTokenValid (user ^. id) (AccessToken password) (^. #api)
81+ when (not isValid) $ do
82+ throw Unauthorized
83+ jwtSettings' <- view #jwtSettings
84+ expiresAt <-
85+ liftIO getCurrentTime
86+ <&> addUTCTime 3600
87+ jwt <- liftIO $ makeJWT (ApiSession user) jwtSettings' (Just expiresAt)
88+ jwt <- case jwt of
89+ Left err -> throw $ OtherError $ "Failed to create JWT: " <> show err
90+ Right jwt -> pure jwt
91+ pure
92+ $ AuthJwtDto
93+ { token = cs jwt,
94+ expiresAt
95+ }
96+
97+data LoginAPI route = LoginAPI
98+ { _loginAPILogin :: route :- Get '[JSON] LoginLinks,
99+ _loginAPILogout ::
100+ route
101+ :- Delete
102+ '[JSON]
103+ ( Headers
104+ '[Header "Set-Cookie" SetCookie, Header "Set-Cookie" SetCookie]
105+ ()
106+ ),
107+ _loginAPILoginCallback ::
108+ route
109+ :- "cb"
110+ :> QueryParam "code" OAuthCode
111+ :> Get
112+ '[JSON]
113+ ( Headers
114+ '[ Header "Set-Cookie" SetCookie,
115+ Header "Set-Cookie" SetCookie
116+ ]
117+ GhLogin
118+ )
119+ }
120+ deriving (Generic)
121+
122+data SignupAPI route = SignupAPI
123+ { _signupAPISignup :: route :- Get '[JSON] SignupLinks,
124+ _signupAPISignupCallback ::
125+ route
126+ :- "fill"
127+ :> QueryParam "code" OAuthCode
128+ :> Get
129+ '[JSON]
130+ ( Headers
131+ '[ Header "Set-Cookie" SetCookie,
132+ Header "Set-Cookie" SetCookie
133+ ]
134+ (CreatingUser ())
135+ ),
136+ _signupAPIFinishSignup ::
137+ route
138+ :- Auth '[Cookie] (CreatingUser GhToken)
139+ :> ReqBody '[JSON] CreateUser
140+ :> Post
141+ '[JSON]
142+ ( Headers
143+ '[ Header "Set-Cookie" SetCookie,
144+ Header "Set-Cookie" SetCookie
145+ ]
146+ GhLogin
147+ )
148+ }
149+ deriving (Generic)
150+
151+loginAPI :: LoginAPI (AsServerT M)
152+loginAPI =
153+ LoginAPI
154+ { _loginAPILogin = login,
155+ _loginAPILogout = logout,
156+ _loginAPILoginCallback = loginCallback
157+ }
158+
159+signupAPI :: SignupAPI (AsServerT M)
160+signupAPI =
161+ SignupAPI
162+ { _signupAPISignup = signup,
163+ _signupAPISignupCallback = signupCallback,
164+ _signupAPIFinishSignup = finishSignup
165+ }
166+
167+login :: M LoginLinks
168+login = do
169+ oaState <- OA.newOAuthState
170+ ghOauth <- githubOauthLogin
171+ githubLink <- OA.getAuthorize oaState ghOauth "foo"
172+ return $ LoginLinks {_loginLinksGithub = githubLink}
173+
174+logout ::
175+ M
176+ ( Headers
177+ '[Header "Set-Cookie" SetCookie, Header "Set-Cookie" SetCookie]
178+ ()
179+ )
180+logout = do
181+ cookieSettings' <- view #cookieSettings
182+ return $ clearSession cookieSettings' ()
183+
184+signup :: M SignupLinks
185+signup = do
186+ oaState <- OA.newOAuthState
187+ ghOauth <- githubOauthSignup
188+ githubLink <- OA.getAuthorize oaState ghOauth "foo"
189+ return
190+ $ SignupLinks
191+ { _signupLinksGithub = githubLink
192+ }
193+
194+loginCallback ::
195+ Maybe OAuthCode ->
196+ M
197+ ( Headers
198+ '[Header "Set-Cookie" SetCookie, Header "Set-Cookie" SetCookie]
199+ GhLogin
200+ )
201+loginCallback code = do
202+ (login', _, token) <- callbackHelper githubOauthLogin code
203+ cookieSettings' <- view #cookieSettings
204+ jwtSettings' <- view #jwtSettings
205+ user <- DB.getUser login' <?> "calling getUser"
206+ mApplyCookies <-
207+ liftIO (acceptLogin cookieSettings' jwtSettings' (WebSession user token))
208+ <?> "calling acceptLogin"
209+ case mApplyCookies of
210+ Nothing -> throw Unauthorized
211+ Just applyCookies ->
212+ return
213+ $ applyCookies
214+ $ user
215+ ^. githubLogin
216+
217+signupCallback ::
218+ Maybe OAuthCode ->
219+ M
220+ ( Headers
221+ '[Header "Set-Cookie" SetCookie, Header "Set-Cookie" SetCookie]
222+ (CreatingUser ())
223+ )
224+signupCallback code = do
225+ (login', email', token) <- callbackHelper githubOauthSignup code
226+ eUser <- try $ DB.getUser login' <?> "calling getUser"
227+ creatingUser <- case eUser of
228+ Right _ ->
229+ pure
230+ $ CreatingUser
231+ { _creatingUserExists = True,
232+ _creatingUserGithubLogin = login',
233+ _creatingUserEmail = email',
234+ _creatingUserGithubToken = token
235+ }
236+ Left ErrorWithContext {err = NoSuchUser {}} ->
237+ pure
238+ $ CreatingUser
239+ { _creatingUserExists = False,
240+ _creatingUserGithubLogin = login',
241+ _creatingUserEmail = email',
242+ _creatingUserGithubToken = token
243+ }
244+ Left e -> throwError e
245+ cookieSettings' <- view #cookieSettings
246+ jwtSettings' <- view #jwtSettings
247+ mApplyCookies <- liftIO $ case eUser of
248+ Right user -> acceptLogin cookieSettings' jwtSettings' (WebSession user token)
249+ _ -> acceptLogin cookieSettings' jwtSettings' creatingUser
250+ case mApplyCookies of
251+ Nothing -> throw Unauthorized
252+ Just applyCookies -> return $ applyCookies (void creatingUser)
253+
254+callbackHelper :: M OA.OAuth2 -> Maybe OAuthCode -> M (GhLogin, Email, GhToken)
255+callbackHelper _ Nothing = throw $ OtherError "'code' param missing"
256+callbackHelper githubOauth (Just (OAuthCode code)) = do
257+ oaState <- OA.newOAuthState <?> "Creating new oauth state"
258+ ghOauth <- githubOauth
259+ mToken <-
260+ liftIO (OA.getAuthorized ghOauth oaState (Just code) Nothing)
261+ <?> "calling OA.getAuthorized"
262+ case mToken of
263+ Nothing -> throw GithubDidntGiveUsAToken
264+ Just (token, _) -> do
265+ let auth = GH.OAuth $ cs token
266+ eGhUser <- liftIO (GH.github auth GH.userInfoCurrentR) <?> "calling userInfoCurrentR"
267+ case eGhUser of
268+ Left e -> throw $ OtherError $ show e
269+ Right ghUser -> do
270+ e <- getEmail auth ghUser <?> "calling getEmail"
271+ pure
272+ ( GhLogin . GH.untagName $ GH.userLogin ghUser,
273+ e,
274+ GhToken $ cs token
275+ )
276+ where
277+ getEmail auth ghUser = case GH.userEmail ghUser of
278+ Just e -> pure $ Email e
279+ Nothing -> do
280+ emails <-
281+ liftIO (GH.github auth $ GH.currentUserEmailsR GH.FetchAll)
282+ <?> "calling currentUserEmailsR"
283+ case find GH.emailPrimary <$> emails of
284+ Right (Just e') -> pure $ Email $ GH.emailAddress e'
285+ _ -> throw $ OtherError "No email address"
286+
287+finishSignup ::
288+ AuthResult (CreatingUser GhToken) ->
289+ CreateUser ->
290+ M (Headers '[Header "Set-Cookie" SetCookie, Header "Set-Cookie" SetCookie] GhLogin)
291+finishSignup (Authenticated cUser) addenda = do
292+ -- The things in AuthResult we can trust, because we put them there
293+ user <-
294+ DB.newUser
295+ (cUser ^. githubLogin)
296+ (addenda ^. email)
297+ (addenda ^. subscriptionType)
298+ (addenda ^. agreeToEmails)
299+ cookieSettings' <- view #cookieSettings
300+ jwtSettings' <- view #jwtSettings
301+ mApplyCookies <- liftIO $ acceptLogin cookieSettings' jwtSettings' (WebSession user (cUser ^. githubToken))
302+ case mApplyCookies of
303+ Nothing -> throw Unauthorized
304+ Just applyCookies -> return $ applyCookies $ user ^. githubLogin
305+finishSignup _ _ = throw $ OtherError "Did not receive expected user info"
306+
307+githubOauthLogin :: M OA.OAuth2
308+githubOauthLogin = do
309+ clientId <- view #githubClientId
310+ ghClientSecret <- view #githubClientSecret
311+ fromRelativeUrl <- relativeUrlConverter
312+ pure
313+ $ OA.OAuth2
314+ { oauthClientId = clientId,
315+ oauthClientSecret = ghClientSecret,
316+ oauthOAuthorizeEndpoint = "https://github.com/login/oauth/authorize",
317+ oauthAccessTokenEndpoint = "https://github.com/login/oauth/access_token",
318+ oauthCallback = fromRelativeUrl "login/cb",
319+ oauthScopes = []
320+ }
321+
322+githubOauthSignup :: M OA.OAuth2
323+githubOauthSignup = do
324+ clientId <- view #githubClientId
325+ ghClientSecret <- view #githubClientSecret
326+ fromRelativeUrl <- relativeUrlConverter
327+ pure
328+ $ OA.OAuth2
329+ { oauthClientId = clientId,
330+ oauthClientSecret = ghClientSecret,
331+ oauthOAuthorizeEndpoint = "https://github.com/login/oauth/authorize",
332+ oauthAccessTokenEndpoint = "https://github.com/login/oauth/access_token",
333+ oauthCallback = fromRelativeUrl "signup/fill",
334+ oauthScopes = []
335+ }
backend/src/Garnix/API/Badges.hsnew file+62
@@ -0,0 +1,62 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+
3+module Garnix.API.Badges (Badge (..), badgesAPI) where
4+
5+import Data.Aeson qualified as Aeson
6+import Data.FileEmbed
7+import Garnix.DB qualified as DB
8+import Garnix.Monad
9+import Garnix.Prelude
10+import Garnix.Types
11+
12+badgesAPI :: GhRepoOwner -> GhRepoName -> Maybe Branch -> M Badge
13+badgesAPI owner repo mGivenBranch = do
14+ let branchSummary :: Branch -> M Badge
15+ branchSummary branch = buildSummary <$> DB.getLatestBuildsForBranch owner repo branch
16+ case mGivenBranch of
17+ Just givenBranch -> branchSummary givenBranch
18+ Nothing -> do
19+ eitherRepo <- getDefaultBranch Nothing owner repo
20+ case eitherRepo of
21+ Nothing -> pure $ Badge "build status unknown"
22+ Just defaultBranch -> branchSummary defaultBranch
23+
24+buildSummary :: [Build] -> Badge
25+buildSummary builds =
26+ let successes = length $ filter ((== Just Success) . view status) builds
27+ failures = length $ filter ((`elem` [Just Failure, Just Cancelled]) . view status) builds
28+ inProgress = length $ filter ((== Nothing) . view status) builds
29+ in Badge $ case (successes, failures, inProgress) of
30+ (0, 0, 0) -> "build status unknown"
31+ (0, 0, _) -> "build in progress"
32+ (1, 0, 0) -> "1 build succeeded"
33+ (1, 0, 1) -> "1 build succeeded, 1 build in progress"
34+ (1, 0, pending) -> "1 build succeeded, " <> show pending <> " builds in progress"
35+ (success, 0, 0) -> show success <> " builds succeeded"
36+ (success, 0, 1) -> show success <> " builds succeeded, 1 build in progress"
37+ (success, 0, pending) -> show success <> " builds succeeded, " <> show pending <> " builds in progress"
38+ (0, _failure, 0) -> "all builds failed"
39+ (1, failure, 0) -> "1 build succeeded out of " <> show (1 + failure)
40+ (success, failure, 0) -> show success <> " builds succeeded out of " <> show (success + failure)
41+ (success, failure, pending) -> show success <> " succeeded, " <> show failure <> " failed, and " <> show pending <> " in progress"
42+
43+data Badge = Badge {badgeMessage :: Text}
44+ deriving stock (Show)
45+
46+instance ToJSON Badge where
47+ toJSON :: Badge -> Aeson.Value
48+ toJSON Badge {..} =
49+ Aeson.object
50+ [ "label" Aeson..= (" " :: Text),
51+ "labelColor" Aeson..= ("white" :: Text),
52+ "color" Aeson..= ("black" :: Text),
53+ "message" Aeson..= badgeMessage,
54+ "logoSvg" Aeson..= faviconSvg
55+ ]
56+ where
57+ faviconSvg :: String
58+ faviconSvg =
59+ $( do
60+ badgeFile <- makeRelativeToProject "data/badge_favicon.svg"
61+ embedStringFile badgeFile
62+ )
backend/src/Garnix/API/Builds.hsnew file+170
@@ -0,0 +1,170 @@
1+module Garnix.API.Builds where
2+
3+import Data.List.Extra (unsnoc)
4+import Data.Maybe (maybeToList)
5+import Data.Text qualified as T
6+import Garnix.API.Builds.Types
7+import Garnix.API.Commits (GetCommit, ListCommits, getCommitsForUser, getSingleCommit)
8+import Garnix.Access (Access (..), getBuildWithAccess)
9+import Garnix.DB qualified as DB
10+import Garnix.Monad
11+import Garnix.Orchestrator qualified as Orchestrator
12+import Garnix.Prelude
13+import Garnix.Reporters.OpenSearchReporter (openSearchReporter)
14+import Garnix.Types
15+import Garnix.UserLogs (getLogLines)
16+import GitHub.Data.Id (Id (Id))
17+import Servant.API (Put)
18+import Servant.API.ContentTypes
19+import Servant.API.Stream
20+import Servant.Auth.Server
21+import Servant.Types.SourceT
22+
23+data BuildAPI route = BuildAPI
24+ { _buildAPIgetLogs :: route :- Capture "buildId" BuildId :> "logs" :> QueryParam "after" UTCTime :> Get '[JSON] BuildLogs,
25+ _buildAPIgetLogsRaw :: route :- Capture "buildId" BuildId :> "logs" :> "raw" :> StreamGet NewlineFraming PlainText (SourceT IO Text),
26+ _buildAPIgetBuild :: route :- Capture "buildId" BuildId :> Get '[JSON] BuildResponse,
27+ _buildAPIupdateBuild :: route :- Capture "buildId" BuildId :> ReqBody '[JSON] BuildUpdate :> Put '[JSON] (),
28+ _buildAPIlistCommits :: route :- "commits" :> Get '[JSON] ListCommits,
29+ _buildAPIgetCommit :: route :- "commit" :> Capture "commit" CommitHash :> Get '[JSON] GetCommit,
30+ _buildAPIsubmitTestBuild :: route :- "submit" :> ReqBody '[JSON] SubmitTestBuild :> Post '[JSON] ()
31+ }
32+ deriving (Generic)
33+
34+buildAPI :: AuthResult AuthJwtPayload -> BuildAPI (AsServerT M)
35+buildAPI (Authenticated ((^. #user) -> user')) =
36+ BuildAPI
37+ { _buildAPIgetLogs = getBuildLogs (Just user'),
38+ _buildAPIgetLogsRaw = getLogsRaw (Just user'),
39+ _buildAPIgetBuild = getBuild' (Just user'),
40+ _buildAPIupdateBuild = updateBuild user',
41+ _buildAPIlistCommits = getCommitsForUser user',
42+ _buildAPIgetCommit = getSingleCommit (Just user'),
43+ _buildAPIsubmitTestBuild = submitTestBuild
44+ }
45+buildAPI _ =
46+ BuildAPI
47+ { _buildAPIgetLogs = getBuildLogs Nothing,
48+ _buildAPIgetLogsRaw = getLogsRaw Nothing,
49+ _buildAPIgetBuild = getBuild' Nothing,
50+ _buildAPIupdateBuild = \_ _ -> throw Unauthorized,
51+ _buildAPIlistCommits = throw Unauthorized,
52+ _buildAPIgetCommit = getSingleCommit Nothing,
53+ _buildAPIsubmitTestBuild = submitTestBuild
54+ }
55+
56+data SubmitTestBuild = SubmitTestBuild
57+ { owner :: GhRepoOwner,
58+ repo :: GhRepoName,
59+ testCommit :: CommitHash
60+ }
61+ deriving (Generic)
62+
63+instance FromJSON SubmitTestBuild
64+
65+submitTestBuild :: SubmitTestBuild -> M ()
66+submitTestBuild SubmitTestBuild {owner, repo, testCommit} = do
67+ installationId <- getGarnixInstallationId owner repo
68+ case installationId of
69+ Nothing -> throw NotFound
70+ Just id -> do
71+ iAuth <- getInstallation (Id $ fromInteger id)
72+ tok <- getAccessToken iAuth
73+ let commitInfo =
74+ CommitInfo
75+ { _commitInfoReqUser = "garnix-io",
76+ _commitInfoRepoPublicity = RepoIsPublic False,
77+ _commitInfoRepoInfo = RepoInfo iAuth tok owner repo,
78+ _commitInfoBranch = Nothing,
79+ _commitInfoPrFromFork = Just $ PrFromFork $ getGhLogin (getGhRepoOwner owner) <> "/" <> getGhRepoName repo,
80+ _commitInfoCommit = testCommit
81+ }
82+ void $ Orchestrator.handleCommit openSearchReporter True commitInfo
83+
84+getBuildLogs :: (HasCallStack) => Maybe User -> BuildId -> Maybe UTCTime -> M BuildLogs
85+getBuildLogs user' buildId mAfter = do
86+ build <- getBuildWithAccess Read user' buildId
87+ let maxResults = 4096
88+ logs <- getLogLines build maxResults mAfter
89+ let finished = isJust (build ^. status) && length logs < maxResults
90+ pure $ BuildLogs finished maxResults logs
91+
92+getLogsRaw :: (HasCallStack) => Maybe User -> BuildId -> M (SourceT IO Text)
93+getLogsRaw user buildId = fromStepT <$> getNextStep Nothing
94+ where
95+ getNextStep :: Maybe UTCTime -> M (StepT IO Text)
96+ getNextStep mAfter = do
97+ env :: Env <- ask
98+ BuildLogs finished _ logs <- getBuildLogs user buildId mAfter
99+ let logsText = T.intercalate "\n" $ map formatLogLine logs
100+ lastLogLine = snd <$> unsnoc logs
101+ nextStep = case (finished, lastLogLine) of
102+ (True, _) -> Stop
103+ (False, Nothing) -> Stop
104+ (False, Just lastLogLine) -> Effect $ do
105+ result <- runM env $ getNextStep $ Just $ lastLogLine ^. timestamp
106+ pure $ case result of
107+ Left err -> Servant.Types.SourceT.Error $ cs $ userMessage $ toErrorDetails err
108+ Right x -> x
109+ pure $ Yield logsText nextStep
110+ formatLogLine :: OpenSearchMessage -> Text
111+ formatLogLine (OpenSearchMessage timestamp package phase message) =
112+ show timestamp
113+ <> maybe "" (\p -> " " <> getPackageName p) package
114+ <> maybe "" (\p -> " (" <> p <> ")") phase
115+ <> "> "
116+ <> message
117+
118+getBuild' :: Maybe User -> BuildId -> M BuildResponse
119+getBuild' user' buildId = do
120+ b <- getBuildWithAccess Read user' buildId
121+ originalBuild <- case b ^. drvPath of
122+ Just drv | b ^. alreadyBuilt == Just True -> DB.getOriginalBuildForDrvPath user' drv
123+ _ -> pure Nothing
124+ pure
125+ $ BuildResponse
126+ { _buildResponseId = b ^. id,
127+ _buildResponseRepoUser = b ^. repoUser,
128+ _buildResponseRepoName = b ^. repoName,
129+ _buildResponseGitCommit = b ^. gitCommit,
130+ _buildResponseBranch = b ^. branch,
131+ _buildResponsePackage = b ^. package,
132+ _buildResponsePackageType = b ^. packageType,
133+ _buildResponseSystem = b ^. system,
134+ _buildResponseReqUser = b ^. reqUser,
135+ _buildResponseStatus = b ^. status,
136+ _buildResponseStartTime = b ^. startTime,
137+ _buildResponseEndTime = b ^. endTime,
138+ _buildResponseGithubRunId = b ^. githubRunId,
139+ _buildResponseOriginalBuild = originalBuild,
140+ _buildResponseRelatedBuilds = maybeToList originalBuild
141+ }
142+
143+updateBuild :: User -> BuildId -> BuildUpdate -> M ()
144+updateBuild user' buildId buildUpdate = do
145+ case buildUpdate ^. status of
146+ Just Cancelled -> do
147+ b <- getBuildWithAccess Cancel (Just user') buildId
148+ if isJust $ b ^. status
149+ then throw (BuildAlreadyStopped buildId)
150+ else do
151+ buildEnd <- liftIO getCurrentTime
152+ let build =
153+ b
154+ & status
155+ ?~ Cancelled
156+ & endTime
157+ ?~ buildEnd
158+ DB.reportBuildResultDB build
159+ Just _ -> throw (InvalidBuildUpdate buildUpdate)
160+ Nothing -> pure ()
161+
162+data NewLogsEvent = NewLogsEvent
163+ { _newLogsEventPackage :: Maybe Text,
164+ _newLogsEventContent :: Text
165+ }
166+ deriving stock (Eq, Show, Generic)
167+
168+instance ToJSON NewLogsEvent where
169+ toEncoding = ourToEncoding
170+ toJSON = ourToJSON
backend/src/Garnix/API/Builds/Types.hsnew file+15
@@ -0,0 +1,15 @@
1+module Garnix.API.Builds.Types where
2+
3+import Garnix.Prelude
4+import Garnix.Types
5+
6+data BuildLogs = BuildLogs
7+ { _buildLogsFinished :: Bool,
8+ _buildLogsMaxPageSize :: Int,
9+ _buildLogsLogs :: [OpenSearchMessage]
10+ }
11+ deriving stock (Eq, Show, Generic)
12+
13+instance ToJSON BuildLogs where
14+ toEncoding = ourToEncoding
15+ toJSON = ourToJSON
backend/src/Garnix/API/Cache.hsnew file+173
@@ -0,0 +1,173 @@
1+module Garnix.API.Cache
2+ ( CacheAPI (..),
3+ cacheAPI,
4+
5+ -- * exported for tests
6+ XForwardedFor (..),
7+ isInternal,
8+ )
9+where
10+
11+import Amazonka qualified
12+import Amazonka.S3 qualified as Amazonka
13+import Data.String.Interpolate (i)
14+import Data.String.Interpolate.Util (unindent)
15+import Data.Text qualified as T
16+import Garnix.API.Cache.Auth (getStoreHashPermission)
17+import Garnix.API.Cache.Permissions (Permission (..))
18+import Garnix.API.Cache.Types
19+import Garnix.DB qualified as DB
20+import Garnix.Duration
21+import Garnix.Monad
22+import Garnix.Monad.Metrics (incrementEvent)
23+import Garnix.Nix.Types (StorePath (..))
24+import Garnix.Prelude
25+import Garnix.S3Cache (toNarFilePath)
26+import Garnix.Types
27+import Servant
28+
29+data CacheAPI route = CacheAPI
30+ { _cacheAPInixCacheInfo :: route :- "nix-cache-info" :> Get '[PlainText] Text,
31+ _cacheAPInarInfo ::
32+ route
33+ :- Header "X-Forwarded-For" XForwardedFor
34+ :> Capture "narinfo file" NarInfoFileName
35+ :> Header "Authorization" Text
36+ :> Get '[NarInfo] NarInfo
37+ }
38+ deriving (Generic)
39+
40+newtype XForwardedFor = XForwardedFor Text
41+ deriving stock (Show)
42+
43+instance FromHttpApiData XForwardedFor where
44+ parseUrlPiece :: Text -> Either Text XForwardedFor
45+ parseUrlPiece = Right . XForwardedFor
46+
47+cacheAPI :: CacheAPI (AsServerT M)
48+cacheAPI =
49+ CacheAPI
50+ { _cacheAPInixCacheInfo = pure nixCacheInfo,
51+ _cacheAPInarInfo = serveNarInfo
52+ }
53+
54+nixCacheInfo :: Text
55+nixCacheInfo =
56+ cs
57+ $ unindent
58+ [i|
59+ StoreDir: /nix/store
60+ WantMassQuery: 1
61+ Priority: 50
62+ |]
63+
64+serveNarInfo :: Maybe XForwardedFor -> NarInfoFileName -> Maybe Text -> M NarInfo
65+serveNarInfo xForwardedFor (NarInfoFileName hash) authorization = do
66+ cacheStoreHash <- DB.getS3CacheStoreHash hash
67+ case cacheStoreHash of
68+ Nothing -> shortcut NotFound
69+ Just
70+ ( DB.S3CacheStoreHash
71+ { hash,
72+ packageName,
73+ narHash,
74+ narSize,
75+ public,
76+ sig,
77+ references,
78+ fileSize,
79+ fileHash
80+ }
81+ ) -> do
82+ let storePath = StorePath hash packageName
83+ let compression = XZ
84+ url <- do
85+ if public
86+ then
87+ if isInternal xForwardedFor
88+ then
89+ publicInternalS3Url storePath compression
90+ else
91+ publicS3Url storePath compression
92+ else do
93+ permission <- getStoreHashPermission (getHash storePath) authorization
94+ case permission of
95+ Disallowed -> shortcut NotFound
96+ Allowed -> privateS3Url storePath compression
97+ incrementEvent #s3CacheNarfilesServed
98+ pure
99+ $ NarInfo
100+ { storePath,
101+ narHash,
102+ narSize,
103+ url,
104+ sig,
105+ references,
106+ compression,
107+ fileSize,
108+ fileHash
109+ }
110+
111+publicS3Url :: StorePath -> Compression -> M Text
112+publicS3Url storePath compression = do
113+ publicBaseUrl <- view $ #s3CacheEnv . #publicBaseUrl
114+ pure $ publicBaseUrl <> toNarFilePath storePath compression
115+
116+isInternal :: Maybe XForwardedFor -> Bool
117+isInternal = \case
118+ Nothing -> False
119+ Just (XForwardedFor header) ->
120+ any
121+ (`elem` map T.strip (T.splitOn "," header))
122+ internalServers
123+ where
124+ internalServers =
125+ [ "65.108.28.108",
126+ "2a01:4f9:3071:29ce::2",
127+ "65.108.28.106",
128+ "2a01:4f9:3071:29cf::2",
129+ "65.108.28.107",
130+ "2a01:4f9:3071:29cd::2",
131+ "88.99.75.150",
132+ "2a01:4f8:10a:130d::2",
133+ "157.90.140.190",
134+ "2a01:4f8:2220:37c4::2",
135+ "65.21.80.216",
136+ "2a01:4f9:3a:47cf::1",
137+ "23.88.85.24",
138+ "2a01:4f8:e0:204e::2",
139+ "65.109.75.126",
140+ "2a01:4f9:3071:3220::2",
141+ "91.107.205.127",
142+ "142.132.141.88",
143+ "142.132.141.89",
144+ "2a01:4f8:c012:7fe0::1"
145+ ]
146+
147+publicInternalS3Url :: StorePath -> Compression -> M Text
148+publicInternalS3Url storePath compression = do
149+ log Informational "serving with publicInternalS3Url"
150+ publicBucket <- view $ #s3CacheEnv . #publicBucket
151+ let request =
152+ Amazonka.newGetObject
153+ publicBucket
154+ (Amazonka.ObjectKey (toNarFilePath storePath compression))
155+ env <- view $ #s3CacheEnv . #amazonkaEnv
156+ now <- liftIO getCurrentTime
157+ expiration <- view $ #s3CacheEnv . #expiration
158+ cs <$> Amazonka.presignURL env now (toAmazonkaSeconds expiration) request
159+
160+privateS3Url :: StorePath -> Compression -> M Text
161+privateS3Url storePath compression = do
162+ privateBucket <- view $ #s3CacheEnv . #privateBucket
163+ let request =
164+ Amazonka.newGetObject
165+ privateBucket
166+ (Amazonka.ObjectKey (toNarFilePath storePath compression))
167+ env <- view $ #s3CacheEnv . #amazonkaEnv
168+ now <- liftIO getCurrentTime
169+ expiration <- view $ #s3CacheEnv . #expiration
170+ cs <$> Amazonka.presignURL env now (toAmazonkaSeconds expiration) request
171+
172+toAmazonkaSeconds :: Duration -> Amazonka.Seconds
173+toAmazonkaSeconds = Amazonka.Seconds . realToFrac . toSeconds
backend/src/Garnix/API/Cache/Auth.hsnew file+68
@@ -0,0 +1,68 @@
1+module Garnix.API.Cache.Auth
2+ ( getStoreHashPermission,
3+ __accessTokenValidCache,
4+ )
5+where
6+
7+import Control.Monad.Extra
8+import Data.Functor
9+import Data.List.Extra
10+import Garnix.API.Cache.Permissions
11+import Garnix.AccessToken
12+import Garnix.AccessToken.Types
13+import Garnix.DB qualified as DB
14+import Garnix.Duration
15+import Garnix.ExpiringCache
16+import Garnix.Monad
17+import Garnix.Nix.Types
18+import Garnix.ParseHttpBasicAuth
19+import Garnix.Prelude
20+import Garnix.Types hiding (getUserId)
21+import System.IO.Unsafe qualified
22+
23+getStoreHashPermission :: StoreHash -> Maybe Text -> M Permission
24+getStoreHashPermission storeHash authorization = do
25+ mGhLogin <- case parseBasicAuth <$> authorization of
26+ Nothing -> pure Nothing
27+ Just (Left err) -> do
28+ throw $ UnauthorizedWithMessage $ "Failed to parse basic auth: " <> show err
29+ Just (Right (user, pass)) -> do
30+ let ghLogin = GhLogin user
31+ isValid <- isAccessTokenValidCached storeHash ghLogin $ AccessToken pass
32+ unless isValid $ throw InvalidAccessToken
33+ pure $ Just ghLogin
34+ withTextSpan ("auth_claim", show mGhLogin) $ do
35+ repos <- DB.getReposForHash storeHash
36+ case repos of
37+ [] -> pure Allowed
38+ repos -> do
39+ permissions <- forM repos $ \(repoOwner, repoName) -> do
40+ getRepoPermissions mGhLogin repoOwner repoName
41+ pure $ if Allowed `elem` permissions then Allowed else Disallowed
42+ where
43+ isAccessTokenValidCached :: StoreHash -> GhLogin -> AccessToken -> M Bool
44+ isAccessTokenValidCached storeHash ghLogin accessToken =
45+ lookupCache __accessTokenValidCache (ghLogin, accessToken) $ do
46+ (InternalCacheToken internalToken) <- DB.getUserInternalToken ghLogin
47+ if getAccessTokenText accessToken == internalToken
48+ then do
49+ log Informational $ "authentication successful for internal token for " <> getStoreHash storeHash
50+ pure True
51+ else do
52+ log Informational "internal token check failed, trying to match against user tokens."
53+ userId <-
54+ DB.getUserId ghLogin `catchError` \err -> do
55+ log Warning $ "Failed to lookup user id: " <> show err
56+ throw InvalidAccessToken
57+ isAccessTokenValid userId accessToken (^. #cache)
58+
59+type AccessTokenValidCache = ExpiringCache (GhLogin, AccessToken) Bool
60+
61+{-# NOINLINE __accessTokenValidCache #-}
62+__accessTokenValidCache :: AccessTokenValidCache
63+__accessTokenValidCache =
64+ System.IO.Unsafe.unsafePerformIO
65+ $ mkCache
66+ Nothing
67+ (fromMinutes @Int 5)
68+ (fromMinutes @Int 5)
backend/src/Garnix/API/Cache/Permissions.hsnew file+74
@@ -0,0 +1,74 @@
1+module Garnix.API.Cache.Permissions
2+ ( Permission (..),
3+ getRepoPermissions,
4+ __getRepoPermissionsCache,
5+ )
6+where
7+
8+import Garnix.Duration
9+import Garnix.ExpiringCache
10+import Garnix.Monad
11+import Garnix.Prelude
12+import Garnix.Types
13+import GitHub.Data.Id (Id (..))
14+import System.IO.Unsafe qualified
15+
16+data Permission
17+ = Allowed
18+ | Disallowed
19+ deriving (Eq, Ord, Show)
20+
21+getRepoPermissions :: (HasCallStack) => Maybe GhLogin -> GhRepoOwner -> GhRepoName -> M Permission
22+getRepoPermissions mUser owner repo =
23+ lookupCache __getRepoPermissionsCache (mUser, owner, repo)
24+ $ withTextSpans
25+ [ ("function", "Garnix.API.Cache.Permissions.getRepoPermission"),
26+ ("repo_perm_mUser", show mUser),
27+ ("repo_perm_owner", show owner),
28+ ("repo_perm_repo", show repo)
29+ ]
30+ $ getGarnixInstallationId owner repo
31+ >>= \case
32+ Nothing -> do
33+ log Warning "Cache.getRepoPermissions: could not get garnixInstallationId"
34+ pure Disallowed
35+ Just id -> do
36+ log Informational "Cache.getRepoPermissions: got garnixInstallationId"
37+ iAuth <- getInstallation (Id $ fromInteger id)
38+ repoPublicity <- try $ getRepoPublicity iAuth owner repo
39+ log Informational $ "repoPublicity: " <> show repoPublicity
40+ case (repoPublicity, mUser) of
41+ (Left err, _) -> do
42+ log Informational $ "Error fetching repo publicity, disallowing access: " <> show err
43+ pure Disallowed
44+ (Right (RepoIsPublic True), _) -> do
45+ log Informational "repo is public, allowing access"
46+ pure Allowed
47+ (Right (RepoIsPublic False), Nothing) -> do
48+ log Informational "repo is private and no authentication claim"
49+ pure Disallowed
50+ (Right (RepoIsPublic False), Just user) -> do
51+ collaborators <- getRepoCollaborators iAuth owner repo
52+ case collaborators of
53+ RepoNotFound -> do
54+ log Warning "Repository not found, denying access"
55+ pure Disallowed
56+ GhCollaborators collaborators ->
57+ if user `elem` collaborators
58+ then do
59+ log Informational "User is a collaborator to the repository, allowing"
60+ pure Allowed
61+ else do
62+ log Notice "Access to disallowed resource. Blocking."
63+ pure Disallowed
64+
65+type GithubPermissionCache = ExpiringCache (Maybe GhLogin, GhRepoOwner, GhRepoName) Permission
66+
67+{-# NOINLINE __getRepoPermissionsCache #-}
68+__getRepoPermissionsCache :: GithubPermissionCache
69+__getRepoPermissionsCache =
70+ System.IO.Unsafe.unsafePerformIO
71+ $ mkCache
72+ (Just "__getRepoPermissionsCache")
73+ (fromHours @Int 1)
74+ (fromMinutes @Int 5)
backend/src/Garnix/API/Cache/Types.hsnew file+73
@@ -0,0 +1,73 @@
1+module Garnix.API.Cache.Types where
2+
3+import Data.String.Conversions (LBS)
4+import Data.Text qualified as T
5+import Garnix.Nix.Types (StoreHash (..))
6+import Garnix.Nix.Types qualified as Nix
7+import Garnix.Prelude
8+import Network.HTTP.Media.MediaType ((//))
9+import Servant
10+
11+newtype NarInfoFileName = NarInfoFileName StoreHash
12+ deriving (Show)
13+
14+instance FromHttpApiData NarInfoFileName where
15+ parseUrlPiece path =
16+ case T.stripSuffix ".narinfo" path of
17+ Just hash
18+ | T.length hash == 32 ->
19+ Right $ NarInfoFileName $ StoreHash hash
20+ _ -> Left "Expected a .narInfo file"
21+
22+data NarInfo = NarInfo
23+ { storePath :: Nix.StorePath,
24+ narHash :: Text,
25+ url :: Text,
26+ narSize :: Int64,
27+ sig :: Text,
28+ references :: Text,
29+ compression :: Compression,
30+ fileSize :: Int64,
31+ fileHash :: Text
32+ }
33+ deriving (Generic, Show)
34+
35+instance Accept NarInfo where
36+ contentType Proxy = "text" // "x-nix-narinfo"
37+
38+instance MimeRender NarInfo NarInfo where
39+ mimeRender :: Proxy NarInfo -> NarInfo -> LBS
40+ mimeRender
41+ Proxy
42+ ( NarInfo
43+ { storePath,
44+ narHash,
45+ url,
46+ narSize,
47+ sig,
48+ references,
49+ compression,
50+ fileSize,
51+ fileHash
52+ }
53+ ) =
54+ cs
55+ $ T.unlines
56+ $ fmap
57+ (\(label, value) -> label <> ": " <> value)
58+ [ ("StorePath", cs storePath),
59+ ("NarHash", "sha256:" <> narHash),
60+ ("URL", url),
61+ ("NarSize", show narSize),
62+ ( "Compression",
63+ case compression of
64+ XZ -> "xz"
65+ ),
66+ ("Sig", sig),
67+ ("References", references),
68+ ("FileSize", show fileSize),
69+ ("FileHash", "sha256:" <> fileHash)
70+ ]
71+
72+data Compression = XZ
73+ deriving (Show)
backend/src/Garnix/API/Commits.hsnew file+81
@@ -0,0 +1,81 @@
1+module Garnix.API.Commits where
2+
3+import Garnix.API.Runs (RunSummary, toRunSummary)
4+import Garnix.Access (hasAccessTo, hasAccessToRepo)
5+import Garnix.DB qualified as DB
6+import Garnix.Monad
7+import Garnix.Prelude
8+import Garnix.Types
9+import GitHub.Data.Id (Id (..))
10+import Servant.Auth.Server
11+
12+data CommitAPI route = CommitAPI
13+ { _commitAPIgetCommitsForRepo :: route :- "repo" :> Capture "owner" GhRepoOwner :> Capture "repo" GhRepoName :> Get '[JSON] ListCommits,
14+ _commitAPIgetCommitsForUser :: route :- Get '[JSON] ListCommits,
15+ _commitAPIgetSingleCommit :: route :- Capture "commit" CommitHash :> Get '[JSON] GetCommit
16+ }
17+ deriving (Generic)
18+
19+commitAPI :: AuthResult AuthJwtPayload -> CommitAPI (AsServerT M)
20+commitAPI (Authenticated ((^. #user) -> user')) =
21+ CommitAPI
22+ { _commitAPIgetCommitsForRepo = getCommitsForRepo (Just user'),
23+ _commitAPIgetCommitsForUser = getCommitsForUser user',
24+ _commitAPIgetSingleCommit = getSingleCommit (Just user')
25+ }
26+commitAPI _ =
27+ CommitAPI
28+ { _commitAPIgetCommitsForRepo = getCommitsForRepo Nothing,
29+ _commitAPIgetCommitsForUser = throw Unauthorized,
30+ _commitAPIgetSingleCommit = getSingleCommit Nothing
31+ }
32+
33+data ListCommits = ListCommits
34+ { _listCommitsCommits :: [CommitSummary]
35+ }
36+ deriving (Eq, Show, Generic)
37+
38+instance ToJSON ListCommits where
39+ toEncoding = ourToEncoding
40+ toJSON = ourToJSON
41+
42+data GetCommit = GetCommit
43+ { _getCommitSummary :: CommitSummary,
44+ _getCommitBuilds :: [Build],
45+ _getCommitRuns :: [RunSummary]
46+ }
47+ deriving (Eq, Show, Generic)
48+
49+instance ToJSON GetCommit where
50+ toEncoding = ourToEncoding
51+ toJSON = ourToJSON
52+
53+getCommitsForRepo :: (HasCallStack) => Maybe User -> GhRepoOwner -> GhRepoName -> M ListCommits
54+getCommitsForRepo user repoOwner repoName = do
55+ installationId <- getGarnixInstallationId repoOwner repoName
56+ iAuth <- case installationId of
57+ Nothing -> throw $ NoSuchRepo {_owner = repoOwner, _name = repoName}
58+ Just id -> getInstallation (Id $ fromInteger id)
59+ repoPublicity <- getRepoPublicity iAuth repoOwner repoName
60+ hasAccess <- hasAccessToRepo user repoPublicity repoOwner repoName
61+ when (not hasAccess) $ throw NoSuchRepo {_owner = repoOwner, _name = repoName}
62+ ListCommits <$> DB.getCommitsByOwnerAndRepo repoOwner repoName
63+
64+getCommitsForUser :: User -> M ListCommits
65+getCommitsForUser user = do
66+ commits <- DB.getCommitsForReqUser user
67+ pure $ ListCommits {_listCommitsCommits = commits}
68+
69+getSingleCommit :: Maybe User -> CommitHash -> M GetCommit
70+getSingleCommit user' commit = do
71+ summary <- DB.getCommitSummary commit
72+ hasAccess <- hasAccessTo user' (summary ^. repoIsPublic) (summary ^. reqUser) (summary ^. repoOwner) (summary ^. repoName)
73+ when (not hasAccess) $ throw (NoSuchCommit commit)
74+ result <- DB.getBuildsAndRunsByCommit (summary ^. repoOwner) (summary ^. repoName) commit
75+ pure $ case result of
76+ CommitEvaluating -> GetCommit summary [] []
77+ CommitEvaluated _ builds runs ->
78+ GetCommit
79+ summary
80+ (filter (\b -> b ^. packageType /= TypeOverall) builds)
81+ (map toRunSummary runs)
backend/src/Garnix/API/ConfigSchema.hsnew file+7
@@ -0,0 +1,7 @@
1+module Garnix.API.ConfigSchema (garnixConfigJsonSchema) where
2+
3+import Autodocodec.Schema (JSONSchema, jsonSchemaViaCodec)
4+import Garnix.YamlConfig (GarnixConfig)
5+
6+garnixConfigJsonSchema :: JSONSchema
7+garnixConfigJsonSchema = jsonSchemaViaCodec @GarnixConfig
backend/src/Garnix/API/Dev.hsnew file+47
@@ -0,0 +1,47 @@
1+module Garnix.API.Dev where
2+
3+import Data.Row (Rec, (.==), type (.==))
4+import Data.Set qualified
5+import Garnix.DB qualified as DB
6+import Garnix.Monad
7+import Garnix.Prelude
8+import Garnix.Types
9+import Servant.Auth.Server (SetCookie, acceptLogin)
10+
11+data DevAPI route = DevAPI
12+ { _devAPILogMeIn :: route :- "log-me-in" :> Get '[JSON] LogMeInResponse
13+ }
14+ deriving (Generic)
15+
16+type LogMeInResponse =
17+ Headers
18+ '[ Header "Set-Cookie" SetCookie,
19+ Header "Set-Cookie" SetCookie
20+ ]
21+ (Rec ("success" .== Bool))
22+
23+devAPI :: M LogMeInResponse
24+devAPI = do
25+ testFeatures <- view #testFeatures
26+ when (not (DevApi `Data.Set.member` testFeatures)) $ do
27+ throw DevModeOnly
28+ user <- getTestUser
29+ cookieSettings' <- view #cookieSettings
30+ jwtSettings' <- view #jwtSettings
31+ mApplyCookies <- liftIO $ acceptLogin cookieSettings' jwtSettings' (WebSession user (GhToken "tok"))
32+ case mApplyCookies of
33+ Nothing -> throw Unauthorized
34+ Just applyCookies -> pure $ applyCookies (#success .== True)
35+
36+getTestUser :: M User
37+getTestUser = do
38+ existing <- try $ DB.getUser (GhLogin "dev-user")
39+ case existing of
40+ Right user -> return user
41+ Left (ErrorWithContext {err = NoSuchUser {}}) -> do
42+ DB.newUser
43+ (GhLogin "dev-user")
44+ (Email "dev-user@example.com")
45+ Admin
46+ True
47+ Left e -> throwError e
backend/src/Garnix/API/GhWebhooks.hsnew file+223
@@ -0,0 +1,223 @@
1+{-# OPTIONS_GHC -fno-warn-orphans #-}
2+
3+module Garnix.API.GhWebhooks where
4+
5+import Data.Text qualified as T
6+import Garnix.Async
7+import Garnix.Monad
8+import Garnix.Monad.Async (emptyPromise, logPromiseErrors)
9+import Garnix.Monad.Concurrency (forkM)
10+import Garnix.Orchestrator
11+import Garnix.Prelude
12+import Garnix.Reporters.GithubReporter (mkGithubReporter)
13+import Garnix.Reporters.OpenSearchReporter (openSearchReporter)
14+import Garnix.Types as Types
15+import GitHub (untagId)
16+import GitHub.App.Auth qualified as GH
17+import GitHub.Data.Id (Id (..))
18+import GitHub.Data.Webhooks.Events
19+import GitHub.Data.Webhooks.Payload
20+import Servant.GitHub.Webhook
21+
22+data GhWebhookAPI route = GhWebhookAPI
23+ { _ghWebhookCheckSuite ::
24+ route
25+ :- GitHubEvent '[ 'WebhookCheckSuiteEvent]
26+ :> GitHubSignedReqBody '[JSON] CheckSuiteEvent
27+ :> Post '[JSON] (),
28+ _ghWebhookCheckRun ::
29+ route
30+ :- GitHubEvent '[ 'WebhookCheckRunEvent]
31+ :> GitHubSignedReqBody '[JSON] CheckRunEvent
32+ :> Post '[JSON] (),
33+ _ghWebhookPullRequest ::
34+ route
35+ :- GitHubEvent '[ 'WebhookPullRequestEvent]
36+ :> GitHubSignedReqBody '[JSON] PullRequestEvent
37+ :> Post '[JSON] (),
38+ _ghWebhookPush ::
39+ route
40+ :- GitHubEvent '[ 'WebhookPushEvent]
41+ :> GitHubSignedReqBody '[JSON] PushEvent
42+ :> Post '[JSON] ()
43+ }
44+ deriving (Generic)
45+
46+ghWebhookAPI :: GhWebhookAPI (AsServerT M)
47+ghWebhookAPI =
48+ GhWebhookAPI
49+ { _ghWebhookCheckSuite = wrap $ \event -> ghWebhookCheckSuite event >>= logPromiseErrors,
50+ _ghWebhookCheckRun = wrap ghWebhookCheckRun,
51+ _ghWebhookPullRequest = wrap $ \event -> ghWebhookPullRequest event >>= logPromiseErrors,
52+ _ghWebhookPush = wrap $ \event -> ghWebhookPush event >>= logPromiseErrors
53+ }
54+ where
55+ wrap :: (Show event) => (event -> M a) -> Servant.GitHub.Webhook.RepoWebhookEvent -> ((), event) -> M a
56+ wrap handler _repoWebhookEvent ((), event) = do
57+ uniqueId <- randomBase64 64
58+ withTextSpan ("event_id", uniqueId) $ do
59+ withTextSpan ("tag", "github webhook event") $ do
60+ log Informational $ show event
61+ handler event
62+
63+-- | Triggers a normal build and server deployment.
64+ghWebhookCheckSuite :: (HasCallStack) => CheckSuiteEvent -> M (Promise ())
65+ghWebhookCheckSuite ev
66+ | evCheckSuiteAction ev
67+ == CheckSuiteEventActionRequested
68+ || evCheckSuiteAction ev
69+ == CheckSuiteEventActionRerequested = do
70+ (owner', repo') <- parseRepoFullname (whRepoFullName $ repoForEvent ev)
71+ (iAuth, tok) <- getAuthAndToken (whChecksInstallationId <$> evCheckSuiteInstallation ev)
72+ let commitInfo =
73+ CommitInfo
74+ { _commitInfoReqUser = GhLogin . whUserLogin $ senderOfEvent ev,
75+ _commitInfoRepoPublicity = RepoIsPublic . not . whRepoIsPrivate $ repoForEvent ev,
76+ _commitInfoRepoInfo = RepoInfo iAuth tok owner' repo',
77+ _commitInfoBranch = branch',
78+ _commitInfoPrFromFork = Nothing,
79+ _commitInfoCommit = commit'
80+ }
81+ clientId <- view #githubAppId
82+ isGarnixApp <- case checkSuite' ^. app of
83+ Nothing ->
84+ throw $ OtherError "Check suite without app. Don't know how to proceed"
85+ Just app -> do
86+ pure (app ^. id == untagId clientId)
87+ let reporter = openSearchReporter <> mkGithubReporter (commitInfo ^. repoInfo) (commitInfo ^. commit)
88+ if isGarnixApp
89+ then handleCommit reporter (evCheckSuiteAction ev == CheckSuiteEventActionRerequested) commitInfo
90+ else do
91+ log Informational "Ignoring check suite event from non-Garnix app"
92+ emptyPromise
93+ where
94+ branch' = Branch <$> whCheckSuiteHeadBranch checkSuite'
95+ checkSuite' = evCheckSuiteCheckSuite ev
96+ commit' = CommitHash $ whCheckSuiteHeadSha checkSuite'
97+ghWebhookCheckSuite _ = emptyPromise
98+
99+-- | If the event is a request for rerunning an individual build, triggers a build. Otherwise does nothing.
100+ghWebhookCheckRun :: (HasCallStack) => CheckRunEvent -> M ()
101+ghWebhookCheckRun ev
102+ | evCheckRunAction ev == CheckRunEventActionRerequested = forkM $ do
103+ let ghRunId = GhRunId . fromIntegral . whCheckRunId $ evCheckRunCheckRun ev
104+ reqUser = GhLogin . whUserLogin $ senderOfEvent ev
105+ -- The trick to getting GitHub to properly display the run as re-running is
106+ -- to create a *new* build with the same name as the old one.
107+ -- See https://github.com/orgs/community/discussions/38288
108+ (iAuth, token) <- getAuthAndToken (whChecksInstallationId <$> evCheckRunInstallation ev)
109+ let rerunEvent =
110+ RerunEvent
111+ { reqUser,
112+ ghRunId,
113+ installAuth = iAuth,
114+ token,
115+ repoIsPublic = RepoIsPublic . not . whRepoIsPrivate $ repoForEvent ev
116+ }
117+ handleRerun rerunEvent
118+ | otherwise = pure ()
119+
120+-- | Triggers two things:
121+--
122+-- - Builds for PRs from *forked* repos,
123+-- - PR-deployments (if the PR is *not* coming from a fork).
124+ghWebhookPullRequest :: (HasCallStack) => PullRequestEvent -> M (Promise ())
125+ghWebhookPullRequest ev = do
126+ case evPullReqAction ev of
127+ -- We explicitly pattern-match rather than providing a wildcard so that we
128+ -- get warnings in case there's a change of constructors (in particular,
129+ -- in case 'synchronize' becomes a constructor)
130+ PullRequestOpenedAction -> handlePrEvent
131+ PullRequestActionOther "synchronize" -> handlePrEvent
132+ PullRequestActionOther _ -> emptyPromise
133+ PullRequestAssignedAction -> emptyPromise
134+ PullRequestUnassignedAction -> emptyPromise
135+ PullRequestReviewRequestedAction -> emptyPromise
136+ PullRequestReviewRequestRemovedAction -> emptyPromise
137+ PullRequestLabeledAction -> emptyPromise
138+ PullRequestUnlabeledAction -> emptyPromise
139+ PullRequestEditedAction -> emptyPromise
140+ PullRequestClosedAction -> emptyPromise
141+ PullRequestReopenedAction -> emptyPromise
142+ where
143+ handlePrEvent :: M (Promise ())
144+ handlePrEvent = do
145+ let commit' = CommitHash $ ev ^. payload . Types.head . sha
146+ (owner', repo') <- parseRepoFullname (repoForEvent ev ^. fullName)
147+ (iAuth, tok) <- getAuthAndToken (evPullReqInstallationId ev)
148+ (fromRepo, toRepo) <- getFromTo ev
149+ let prFromFork =
150+ if whRepoFullName toRepo /= whRepoFullName fromRepo
151+ then Just $ PrFromFork (whRepoFullName fromRepo)
152+ else Nothing
153+ let commitInfo =
154+ CommitInfo
155+ { _commitInfoReqUser = GhLogin . whUserLogin $ senderOfEvent ev,
156+ _commitInfoRepoPublicity = RepoIsPublic . not . whRepoIsPrivate $ repoForEvent ev,
157+ _commitInfoRepoInfo = RepoInfo iAuth tok owner' repo',
158+ _commitInfoBranch = Nothing,
159+ _commitInfoPrFromFork = prFromFork,
160+ _commitInfoCommit = commit'
161+ }
162+ let reporter = openSearchReporter <> mkGithubReporter (commitInfo ^. repoInfo) (commitInfo ^. commit)
163+ handlePullRequest reporter commitInfo (GhPullRequestId $ fromIntegral $ ev ^. number)
164+
165+ getFromTo :: PullRequestEvent -> M (HookRepository, HookRepository)
166+ getFromTo ev = do
167+ from <- case ev ^. (payload . Types.head . repo) of
168+ Nothing -> do
169+ -- I don't know why this would happen, so am logging it as an
170+ -- Error for now.
171+ throw $ OtherError "Pull request target without repo. Don't know how to proceed"
172+ Just repo -> pure repo
173+ pure (from, ev ^. repo)
174+
175+-- | Triggers a new run iff it's not the first time this commit
176+-- has been pushed. (If it is the first time, a check suite event has been
177+-- already created.)
178+ghWebhookPush :: (HasCallStack) => PushEvent -> M (Promise ())
179+ghWebhookPush ev
180+ | evPushDeleted ev = emptyPromise
181+ | otherwise = do
182+ (owner', repo') <- parseRepoFullname (whRepoFullName $ repoForEvent ev)
183+ (iAuth, tok) <- getAuthAndToken (whChecksInstallationId <$> evPushInstallation ev)
184+ commit' <- case evPushHeadSha ev of
185+ Nothing -> throw $ OtherError "Push without head sha"
186+ Just c -> pure $ CommitHash c
187+ reqUser <- case evPushSender ev of
188+ Nothing -> throw $ OtherError "Push without a sender"
189+ Just s -> pure . GhLogin . whUserLogin $ s
190+ let commitInfo =
191+ CommitInfo
192+ { _commitInfoReqUser = reqUser,
193+ _commitInfoRepoPublicity = RepoIsPublic . not . whRepoIsPrivate $ repoForEvent ev,
194+ _commitInfoRepoInfo = RepoInfo iAuth tok owner' repo',
195+ _commitInfoBranch = branch',
196+ _commitInfoPrFromFork = Nothing,
197+ _commitInfoCommit = commit'
198+ }
199+ let reporter = openSearchReporter <> mkGithubReporter (commitInfo ^. repoInfo) (commitInfo ^. commit)
200+ handleCommit reporter False commitInfo
201+ where
202+ branch' = case T.splitOn "/" (evPushRef ev) of
203+ "refs" : "heads" : rest -> Just . Branch $ T.intercalate "/" rest
204+ _ -> Nothing
205+
206+parseRepoFullname :: Text -> M (GhRepoOwner, GhRepoName)
207+parseRepoFullname name =
208+ case T.splitOn "/" name of
209+ [o, r] -> pure (GhRepoOwner (GhLogin o), GhRepoName r)
210+ x ->
211+ throw
212+ . OtherError
213+ $ "Expected full name to split in two. Got: "
214+ <> show x
215+
216+getAuthAndToken :: Maybe Int -> M (GH.InstallationAuth, GhToken)
217+getAuthAndToken =
218+ \case
219+ Nothing -> throw $ OtherError "Installation in check suite was Nothing"
220+ Just i -> do
221+ iAuth <- getInstallation (Id i)
222+ tok <- getAccessToken iAuth
223+ pure (iAuth, tok)
backend/src/Garnix/API/Health.hsnew file+26
@@ -0,0 +1,26 @@
1+module Garnix.API.Health where
2+
3+import Garnix.DB qualified as DB
4+import Garnix.Duration
5+import Garnix.Monad
6+import Garnix.Prelude
7+
8+data HealthAPI route = HealthAPI
9+ { _healthAPIHealth :: route :- "check" :> Get '[JSON] (),
10+ _healthAPITriggerCritical :: route :- "some-hard-to-guess-endpoint-ieTh5bi1" :> Get '[JSON] (),
11+ _healthAPISlowAnswer :: route :- "slow-answer-umuS9ain" :> Get '[JSON] ()
12+ }
13+ deriving (Generic)
14+
15+healthAPI :: HealthAPI (AsServerT M)
16+healthAPI =
17+ HealthAPI
18+ { _healthAPIHealth = healthCheck,
19+ _healthAPITriggerCritical = log Critical "critical test message",
20+ _healthAPISlowAnswer = do
21+ liftIO $ threadDelay (fromSeconds @Int 4)
22+ pure ()
23+ }
24+
25+healthCheck :: M ()
26+healthCheck = DB.checkHealth
backend/src/Garnix/API/Hosts.hsnew file+198
@@ -0,0 +1,198 @@
1+module Garnix.API.Hosts
2+ ( getHostsForTraefik,
3+ postHostsHeartbeat,
4+ hostsAPI,
5+ HostsAPI,
6+ getHosts,
7+ HostList (..),
8+ )
9+where
10+
11+import Data.Functor ((<&>))
12+import Data.Map (Map)
13+import Data.Map qualified as Map
14+import Data.Maybe (mapMaybe)
15+import Data.Text qualified as T
16+import Garnix.DB qualified as DB
17+import Garnix.GithubInterface.Types
18+import Garnix.Hosting.Deploy (stopServer)
19+import Garnix.Hosting.Helpers
20+import Garnix.Monad
21+import Garnix.Prelude
22+import Garnix.Types
23+import Servant.Auth.Server
24+
25+data HostsAPI route = HostsAPI
26+ { _hostsAPIGetHostsForTraefik :: route :- "traefik" :> Get '[JSON] HostList,
27+ _hostsAPIHeartbeat :: route :- "heartbeat" :> ReqBody '[JSON] [Text] :> Post '[JSON] NoContent,
28+ _hostsAPIGetIPsForDns :: route :- "dns" :> Get '[JSON] DnsHosts,
29+ _hostsAPIGetDomainsForOnDemandResolver :: route :- "on-demand-resolver" :> Get '[JSON] OnDemandResolverDomainNames,
30+ _hostsAPIGetHosts :: route :- Auth '[JWT, Cookie] AuthJwtPayload :> Get '[JSON] [RunningServer],
31+ _hostsAPIDeleteHost :: route :- Auth '[JWT, Cookie] AuthJwtPayload :> Capture "serverId" ServerId :> Delete '[JSON] ()
32+ }
33+ deriving (Generic)
34+
35+hostsAPI :: HostsAPI (AsServerT M)
36+hostsAPI =
37+ HostsAPI
38+ { _hostsAPIGetHostsForTraefik = getHostsForTraefik,
39+ _hostsAPIHeartbeat = postHostsHeartbeat,
40+ _hostsAPIGetIPsForDns = getHostsForDns,
41+ _hostsAPIGetDomainsForOnDemandResolver = getDomainsForOnDemandResolver,
42+ _hostsAPIGetHosts = getHosts,
43+ _hostsAPIDeleteHost = deleteHost
44+ }
45+
46+data HostList = HostList
47+ { hostList :: [Host],
48+ hostBaseUrl :: Text
49+ }
50+ deriving stock (Eq, Show, Generic)
51+
52+instance ToJSON HostList where
53+ toJSON (HostList hosts baseUrl) =
54+ let routerMapPair serviceDomain ruleDomain =
55+ ( ruleDomain,
56+ [aesonQQ| {
57+ service: #{serviceDomain},
58+ rule: #{"Host(`" <> ruleDomain <> ".garnix.me`)"},
59+ middlewares: ["heartbeatmiddleware"]
60+ }
61+ |]
62+ )
63+
64+ httpRouters =
65+ Map.fromList
66+ $ concatMap
67+ ( \h ->
68+ [routerMapPair (hostToDomainName h) (hostToDomainName h)]
69+ <> if h ^. isPrimary then [routerMapPair (hostToDomainName h) (hostToPrimaryDomainName h)] else []
70+ )
71+ hosts
72+ httpService host =
73+ [aesonQQ|
74+ { loadBalancer:
75+ { servers: [
76+ { url: #{"http://" <> _hostIpV4Addr host}}
77+ ]
78+ }
79+ }
80+ |]
81+ httpServices = Map.fromList $ [(hostToDomainName h, httpService h) | h <- hosts]
82+ in [aesonQQ|
83+ {
84+ http:
85+ {
86+ routers: #{httpRouters},
87+ services: #{httpServices},
88+ middlewares: {
89+ heartbeatmiddleware: {
90+ plugin: {
91+ heartbeatmiddleware: {
92+ reportEndpoint: #{baseUrl <> "/api/hosts/heartbeat"}
93+ }
94+ }
95+ }
96+ }
97+ }
98+ }
99+ |]
100+
101+getHostsForTraefik :: M HostList
102+getHostsForTraefik = do
103+ baseUrl <- view #baseUrl
104+ hosts <-
105+ DB.getAllRunningHosts
106+ <&> filter
107+ ( \host ->
108+ isValidSubdomainString (host ^. repoOwner . to getGhRepoOwner . to getGhLogin)
109+ && isValidSubdomainString (host ^. repoName . to getGhRepoName)
110+ && (isValidSubdomainString (host ^. branch . to getBranch) || isJust (host ^. pullRequest))
111+ && isValidSubdomainString (host ^. packageName . to getPackageName)
112+ )
113+ pure $ HostList hosts baseUrl
114+
115+postHostsHeartbeat :: [Text] -> M NoContent
116+postHostsHeartbeat hosts = NoContent <$ DB.upsertHeartbeat hosts
117+
118+data DnsHosts = DnsHosts
119+ { byHash :: Map Text HostIPs,
120+ byName :: Map Text HostIPs
121+ }
122+ deriving (Generic, ToJSON)
123+
124+data HostIPs = HostIPs {ipv4 :: Text, ipv6 :: Text}
125+ deriving (Eq, Show, Generic, ToJSON)
126+
127+getHostsForDns :: M DnsHosts
128+getHostsForDns = do
129+ runningHosts <- DB.getAllRunningHosts
130+ let mapRunningHosts :: (Host -> Maybe Text) -> Map Text HostIPs
131+ mapRunningHosts getName =
132+ Map.fromList
133+ $ mapMaybe
134+ ( \host -> do
135+ name <- getName host
136+ pure
137+ ( name,
138+ HostIPs
139+ { ipv4 = host ^. ipV4Addr,
140+ ipv6 = host ^. ipV6Addr
141+ }
142+ )
143+ )
144+ runningHosts
145+ let byHash = mapRunningHosts $ \host -> do
146+ drvPath <- host ^. drvPath
147+ T.take 32 <$> T.stripPrefix "/nix/store/" (cs drvPath)
148+ let byName = mapRunningHosts $ Just . hostToDomainName
149+ pure $ DnsHosts {byHash, byName}
150+
151+getHosts :: AuthResult AuthJwtPayload -> M [RunningServer]
152+getHosts (Authenticated (WebSession user ghToken)) = do
153+ getRunningAndRecentServersForOwners
154+ . (GhRepoOwner (user ^. githubLogin) :)
155+ . map organizationName
156+ =<< getInstalledOrgs ghToken
157+getHosts _ = throw Unauthorized
158+
159+deleteHost :: AuthResult AuthJwtPayload -> ServerId -> M ()
160+deleteHost (Authenticated (WebSession user ghToken)) serverId = do
161+ orgs <-
162+ (GhRepoOwner (user ^. githubLogin) :)
163+ . map organizationName
164+ <$> getInstalledOrgs ghToken
165+ hetznerServerIds <- do
166+ DB.getHetznerServerById orgs serverId >>= \case
167+ Nothing -> pure []
168+ Just serverId -> do
169+ pure [serverId]
170+ case hetznerServerIds of
171+ [hetznerServerId] -> do stopServer serverId hetznerServerId
172+ _ -> throw NotFound
173+deleteHost _ _ = throw Unauthorized
174+
175+data OnDemandResolverDomainNames = OnDemandResolverDomainNames
176+ { domains :: [Text]
177+ }
178+ deriving (Generic, ToJSON)
179+
180+getDomainsForOnDemandResolver :: M OnDemandResolverDomainNames
181+getDomainsForOnDemandResolver = do
182+ runningHosts <- DB.getAllRunningHosts
183+ pure
184+ $ OnDemandResolverDomainNames
185+ { domains =
186+ concatMap
187+ ( \host ->
188+ [hostToDomainName host <> ".garnix.me"]
189+ <> if host ^. isPrimary then [hostToPrimaryDomainName host <> ".garnix.me"] else []
190+ )
191+ runningHosts
192+ }
193+
194+hostToPrimaryDomainName :: Host -> Text
195+hostToPrimaryDomainName host =
196+ getGhRepoName (_hostRepoName host)
197+ <> "."
198+ <> getGhLogin (getGhRepoOwner (_hostRepoOwner host))
backend/src/Garnix/API/Keys.hsnew file+51
@@ -0,0 +1,51 @@
1+module Garnix.API.Keys where
2+
3+import Data.Text qualified as T
4+import Garnix.DB qualified as DB
5+import Garnix.Monad
6+import Garnix.Monad.SubProcess.Deprecated qualified as Deprecated
7+import Garnix.Prelude
8+import Garnix.Types
9+
10+getRepoPublicKey :: GhRepoOwner -> GhRepoName -> M PublicKey
11+getRepoPublicKey owner name = fst <$> getRepoKeys owner name
12+
13+getActionPublicKey :: GhRepoOwner -> GhRepoName -> PackageName -> M PublicKey
14+getActionPublicKey owner name action = fst <$> getActionKeys owner name action
15+
16+getRepoKeys :: GhRepoOwner -> GhRepoName -> M (PublicKey, PrivateKey)
17+getRepoKeys owner name = do
18+ mkey <- DB.getRepoKeyDB owner name
19+ case mkey of
20+ Nothing -> do
21+ (candidatePubKey, candidatePrivKey) <- generateKeys
22+ DB.setRepoKeyDB owner name candidatePubKey candidatePrivKey
23+ Just key -> pure key
24+
25+getActionKeys :: GhRepoOwner -> GhRepoName -> PackageName -> M (PublicKey, PrivateKey)
26+getActionKeys owner name action = do
27+ mkey <- DB.getActionKeyDB owner name action
28+ case mkey of
29+ Nothing -> do
30+ (candidatePubKey, candidatePrivKey) <- generateKeys
31+ DB.setActionKeyDB owner name action candidatePubKey candidatePrivKey
32+ Just key -> pure key
33+
34+generateKeys :: M (Candidate PublicKey, Candidate PrivateKey)
35+generateKeys = do
36+ output <- Deprecated.runProc "age-keygen" [] []
37+ case T.lines output of
38+ [_createdAt, pubKeyLine, privKeyLine] -> do
39+ unless ("# public key: " `T.isPrefixOf` pubKeyLine)
40+ $ throw
41+ $ OtherError "age-keygen responded with unexpected format"
42+ repoSecretsPubKey <- view #repoSecretsEncryptionPubKey
43+ privKey <-
44+ liftIO (makePrivateKey (cs privKeyLine) repoSecretsPubKey) >>= \case
45+ Left e -> throw $ OtherError e
46+ Right v -> pure v
47+ pure
48+ ( Candidate (PublicKey $ T.drop (T.length "# public key: ") pubKeyLine),
49+ Candidate privKey
50+ )
51+ _ -> throw $ OtherError "age-keygen responded with unexpected format"
backend/src/Garnix/API/Modules.hsnew file+146
@@ -0,0 +1,146 @@
1+module Garnix.API.Modules
2+ ( ModulesAPI,
3+ modulesAPI,
4+ )
5+where
6+
7+import Control.Lens
8+import Data.ByteString (ByteString)
9+import Data.Row
10+import Garnix.Build qualified as Build
11+import Garnix.Build.Checkout qualified as Checkout
12+import Garnix.Build.Module qualified as Build.Module
13+import Garnix.DB.ModuleValues qualified as ModuleValues
14+import Garnix.Monad
15+import Garnix.Monad.SubProcess qualified as SubProcess
16+import Garnix.Prelude
17+import Garnix.Types
18+import Servant.API (OctetStream, Put)
19+import Servant.Auth.Server
20+
21+data BuildInfo = BuildInfo
22+ { _buildInfoCommit :: CommitHash,
23+ _buildInfoBranch :: Maybe Branch
24+ }
25+ deriving stock (Generic)
26+
27+instance ToJSON BuildInfo where
28+ toEncoding = ourToEncoding
29+ toJSON = ourToJSON
30+
31+data ModulesAPI route = ModulesAPI
32+ { _modulesAPIgetValues ::
33+ route
34+ :- Get '[JSON] ModuleValues.GetRepoAndModuleValues,
35+ _modulesAPIupdateValues ::
36+ route
37+ :- ReqBody '[JSON] ModuleValues.UpdateRepoModuleValues
38+ :> Put '[JSON] NoContent,
39+ _modulesAPIgetAvailableModules ::
40+ route
41+ :- "available"
42+ :> Get '[JSON] (Rec ("modules" .== [ModuleValues.Module])),
43+ _modulesAPIrunBuild ::
44+ route
45+ :- "run"
46+ :> Post '[JSON] BuildInfo,
47+ _modulesAPIcreatePullRequest ::
48+ route
49+ :- "pull-request"
50+ :> Post '[JSON] PullRequestResult,
51+ _modulesAPIgetFlake ::
52+ route
53+ :- "reset"
54+ :> Get '[OctetStream] (Headers '[Header "Content-Disposition" Text] ByteString),
55+ _modulesAPIReset :: route :- "reset" :> Post '[JSON] NoContent
56+ }
57+ deriving (Generic)
58+
59+modulesAPI :: AuthResult AuthJwtPayload -> ModulesAPI (AsServerT M)
60+modulesAPI = \case
61+ Authenticated ((^. #user) -> user) ->
62+ ModulesAPI
63+ { _modulesAPIgetValues = getValues user,
64+ _modulesAPIupdateValues = updateValues user,
65+ _modulesAPIgetAvailableModules = getAvailableModules,
66+ _modulesAPIrunBuild = runBuild user,
67+ _modulesAPIcreatePullRequest = createPullRequest user,
68+ _modulesAPIgetFlake = getFlake user,
69+ _modulesAPIReset = reset user
70+ }
71+ _ ->
72+ ModulesAPI
73+ { _modulesAPIgetValues = throw Unauthorized,
74+ _modulesAPIupdateValues = const $ throw Unauthorized,
75+ _modulesAPIgetAvailableModules = getAvailableModules,
76+ _modulesAPIrunBuild = throw Unauthorized,
77+ _modulesAPIcreatePullRequest = throw Unauthorized,
78+ _modulesAPIgetFlake = throw Unauthorized,
79+ _modulesAPIReset = throw Unauthorized
80+ }
81+
82+getValues :: User -> M ModuleValues.GetRepoAndModuleValues
83+getValues = maybe (throw NotFound) pure <=< ModuleValues.get . _userGithubLogin
84+
85+updateValues :: User -> ModuleValues.UpdateRepoModuleValues -> M NoContent
86+updateValues user values = ModuleValues.update (user ^. githubLogin) values $> NoContent
87+
88+getAvailableModules :: M (Rec ("modules" .== [ModuleValues.Module]))
89+getAvailableModules = (#modules .==) <$> ModuleValues.getAvailableModules
90+
91+runBuild :: User -> M BuildInfo
92+runBuild user = do
93+ repoAndModuleValues <- getValues user
94+ commitInfo <- Build.buildModule (user ^. githubLogin) repoAndModuleValues
95+ pure $ BuildInfo (commitInfo ^. commit) (commitInfo ^. branch)
96+
97+createPullRequest :: User -> M PullRequestResult
98+createPullRequest user = do
99+ ModuleValues.get (user ^. githubLogin) >>= \case
100+ Nothing -> throw NotFound
101+ Just repoAndModuleValues -> do
102+ commitInfo <- Build.Module.getCommitInfo (user ^. githubLogin) repoAndModuleValues
103+ withSpan commitInfo $ do
104+ let baseBranch = maybe (Branch "main") identity $ commitInfo ^. branch
105+ newBranch <- Branch . ("garnix-modules-" <>) <$> randomBase64 8
106+
107+ pushNewBranch repoAndModuleValues commitInfo baseBranch newBranch
108+
109+ openPullRequest commitInfo baseBranch newBranch
110+ where
111+ pushNewBranch :: ModuleValues.GetRepoAndModuleValues -> CommitInfo -> Branch -> Branch -> M ()
112+ pushNewBranch repoAndModuleValues commitInfo baseBranch newBranch = do
113+ let remote = Build.Module.remoteWithFlake baseBranch repoAndModuleValues Checkout.remoteWithConfig
114+ remoteUrl <- getRemote commitInfo
115+ Checkout.runWithCheckout remote commitInfo $ \_garnixConfig -> do
116+ SubProcess.runGitProcess ["checkout", "-b", getBranch newBranch]
117+ SubProcess.runGitProcess ["commit", "-am", "Add garnix modules."]
118+ SubProcess.runGitProcess ["push", realRemoteUrl remoteUrl, getBranch newBranch]
119+
120+ openPullRequest :: CommitInfo -> Branch -> Branch -> M PullRequestResult
121+ openPullRequest commitInfo baseBranch newBranch =
122+ openGithubPullRequest
123+ (commitInfo ^. repoInfo . ghRepoOwner)
124+ (commitInfo ^. repoInfo . ghRepoName)
125+ PullRequest
126+ { _pullRequestTitle = "Enable garnix modules",
127+ _pullRequestBody = "This is an automated pull request created using [garnix modules](https://garnix.io/modules).\n\nCreate or edit your existing modules [here](https://garnix.io/modules/configure).",
128+ _pullRequestHeadBranch = newBranch,
129+ _pullRequestBaseBranch = baseBranch
130+ }
131+
132+getFlake :: User -> M (Headers '[Header "Content-Disposition" Text] ByteString)
133+getFlake user = do
134+ ModuleValues.get (user ^. githubLogin) >>= \case
135+ Nothing -> throw NotFound
136+ Just repoAndModuleValues -> do
137+ commitInfo <- Build.Module.getCommitInfo (user ^. githubLogin) repoAndModuleValues
138+ let defaultBranch = maybe (Branch "main") identity $ commitInfo ^. branch
139+ contents <- Build.Module.generateFlakeNix defaultBranch repoAndModuleValues
140+ pure $ addHeader "attachment; filename=\"flake.nix\"" $ cs contents
141+
142+reset :: User -> M NoContent
143+reset user =
144+ ModuleValues.get (user ^. githubLogin) >>= \case
145+ Nothing -> throw NotFound
146+ Just _ -> ModuleValues.delete (user ^. githubLogin) $> NoContent
backend/src/Garnix/API/Runs.hsnew file+71
@@ -0,0 +1,71 @@
1+module Garnix.API.Runs where
2+
3+import Garnix.API.Builds.Types
4+import Garnix.Access (Access (..), getRunWithAccess)
5+import Garnix.Monad
6+import Garnix.Prelude
7+import Garnix.Types
8+import Garnix.UserLogs (getRunLogLines)
9+import Servant.Auth.Server (AuthResult (..))
10+
11+data RunAPI route = RunAPI
12+ { _runAPIGetRun :: route :- Capture "runId" RunId :> Get '[JSON] RunSummary,
13+ _runAPIGetLogs :: route :- Capture "runId" RunId :> "logs" :> QueryParam "after" UTCTime :> Get '[JSON] BuildLogs
14+ }
15+ deriving stock (Generic)
16+
17+runAPI :: AuthResult AuthJwtPayload -> RunAPI (AsServerT M)
18+runAPI (Authenticated ((^. #user) -> user)) =
19+ RunAPI
20+ { _runAPIGetRun = getRun (Just user),
21+ _runAPIGetLogs = getRunLogs (Just user)
22+ }
23+runAPI _ =
24+ RunAPI
25+ { _runAPIGetRun = getRun Nothing,
26+ _runAPIGetLogs = getRunLogs Nothing
27+ }
28+
29+getRun :: Maybe User -> RunId -> M RunSummary
30+getRun mUser runId = do
31+ run <- getRunWithAccess Read mUser runId
32+ pure $ toRunSummary run
33+
34+getRunLogs :: Maybe User -> RunId -> Maybe UTCTime -> M BuildLogs
35+getRunLogs mUser runId mAfter = do
36+ run <- getRunWithAccess Read mUser runId
37+ let maxResults = 4096
38+ logs <- getRunLogLines run maxResults mAfter
39+ let finished = isJust (run ^. status) && length logs < maxResults
40+ pure $ BuildLogs finished maxResults logs
41+
42+data RunSummary = RunSummary
43+ { _runSummaryId :: Text,
44+ _runSummaryName :: Text,
45+ _runSummaryRepoUser :: GhRepoOwner,
46+ _runSummaryRepoName :: GhRepoName,
47+ _runSummaryGitCommit :: CommitHash,
48+ _runSummaryBranch :: Maybe Branch,
49+ _runSummaryStatus :: Maybe Status,
50+ _runSummaryStartTime :: UTCTime,
51+ _runSummaryEndTime :: Maybe UTCTime
52+ }
53+ deriving (Eq, Show, Generic)
54+
55+toRunSummary :: Run -> RunSummary
56+toRunSummary run@(Run {_runId, _runName, _runStatus, _runStartTime, _runEndTime}) =
57+ RunSummary
58+ { _runSummaryId = getRunId _runId ^. re hashIdText,
59+ _runSummaryName = _runName,
60+ _runSummaryRepoUser = run ^. repoUser,
61+ _runSummaryRepoName = run ^. repoName,
62+ _runSummaryGitCommit = run ^. gitCommit,
63+ _runSummaryBranch = run ^. branch,
64+ _runSummaryStatus = _runStatus,
65+ _runSummaryStartTime = _runStartTime,
66+ _runSummaryEndTime = _runEndTime
67+ }
68+
69+instance ToJSON RunSummary where
70+ toEncoding = ourToEncoding
71+ toJSON = ourToJSON
backend/src/Garnix/API/Stripe.hsnew file+60
@@ -0,0 +1,60 @@
1+module Garnix.API.Stripe where
2+
3+import Data.Aeson (Value)
4+import Data.ByteString.Lazy qualified
5+import Garnix.API.Account qualified as Account
6+import Garnix.Monad
7+import Garnix.Prelude
8+import Garnix.StripeLib qualified as StripeLib
9+import Network.HTTP.Types (hContentType)
10+import Network.Wai qualified as Wai
11+import Servant
12+import Servant.API.ContentTypes (canHandleCTypeH)
13+import Servant.Server.Internal.Delayed (Delayed, addBodyCheck)
14+import Servant.Server.Internal.DelayedIO (DelayedIO, delayedFail, delayedFailFatal, withRequest)
15+import Servant.Server.Internal.Router (Router)
16+import Stripe.Concepts qualified
17+import Stripe.Signature (isSigValid, parseSig)
18+
19+type StripeWebhookAPI = StripeWebhookEvent :> Post '[JSON] ()
20+
21+stripeWebhookAPI :: ServerT StripeWebhookAPI M
22+stripeWebhookAPI body = do
23+ parsedEvent <- StripeLib.fromWebhookRequest body
24+ case parsedEvent of
25+ Just (StripeLib.SubscriptionCreatedOrUpdated event) -> Account.handleSubscriptionAdded event
26+ Just (StripeLib.InvoiceCreated event) -> Account.handleInvoiceCreated event
27+ Nothing -> return ()
28+
29+data StripeWebhookEvent
30+
31+instance
32+ (HasServer api context, HasContextEntry context Stripe.Concepts.WebhookSecretKey) =>
33+ HasServer (StripeWebhookEvent :> api) context
34+ where
35+ type ServerT (StripeWebhookEvent :> api) m = Value -> ServerT api m
36+
37+ hoistServerWithContext Proxy pc nt s = hoistServerWithContext (Proxy :: Proxy api) pc nt . s
38+
39+ route :: Proxy (StripeWebhookEvent :> api) -> Context context -> Delayed env (Server (StripeWebhookEvent :> api)) -> Router env
40+ route Proxy context subServer =
41+ route (Proxy :: Proxy api) context
42+ $ addBodyCheck subServer (withRequest contentTypeCheck) (\f -> withRequest $ bodyCheck f)
43+ where
44+ contentTypeCheck request =
45+ let contentType = fromMaybe "application/octet-stream" $ lookup hContentType $ Wai.requestHeaders request
46+ in case canHandleCTypeH (Proxy :: Proxy '[JSON]) (cs contentType) of
47+ Nothing -> delayedFail err415
48+ Just f -> return f
49+ bodyCheck :: (Data.ByteString.Lazy.ByteString -> Either String Value) -> Wai.Request -> DelayedIO Value
50+ bodyCheck f request = do
51+ case lookup "Stripe-Signature" (Wai.requestHeaders request) >>= parseSig . cs of
52+ Nothing -> delayedFailFatal err401
53+ Just stripeSignature -> do
54+ body <- liftIO $ Wai.lazyRequestBody request
55+ let webhookSecret :: Stripe.Concepts.WebhookSecretKey = getContextEntry context
56+ if isSigValid stripeSignature webhookSecret (cs body)
57+ then case f body of
58+ Left err -> delayedFailFatal err400 {errBody = cs err}
59+ Right value -> return value
60+ else delayedFailFatal err401
backend/src/Garnix/Access.hsnew file+82
@@ -0,0 +1,82 @@
1+module Garnix.Access
2+ ( Access (..),
3+ getBuildWithAccess,
4+ getRunWithAccess,
5+ hasAccessTo,
6+ hasAccessToRepo,
7+ )
8+where
9+
10+import Garnix.DB qualified as DB
11+import Garnix.Monad
12+import Garnix.Prelude
13+import Garnix.Types as Types
14+import GitHub.Data.Id (Id (Id))
15+
16+data Access = Read | Cancel
17+
18+getRunWithAccess :: Access -> Maybe User -> RunId -> M Run
19+getRunWithAccess access user' runId = do
20+ let accessCheck = case access of
21+ Read -> hasAccessTo
22+ Cancel -> canCancelBuild
23+ run' <- DB.getRun runId
24+ run <- case run' of
25+ Just run -> pure run
26+ Nothing -> throw (NoSuchRun runId)
27+ installationId <- getGarnixInstallationId (run ^. repoUser) (run ^. repoName)
28+ iAuth <- case installationId of
29+ Nothing -> throw $ OtherError "Failed to look up installation auth"
30+ Just id -> getInstallation (Id $ fromInteger id)
31+ repoPublicity <- getRepoPublicity iAuth (run ^. repoUser) (run ^. repoName)
32+ hasAccess <- accessCheck user' repoPublicity (run ^. reqUser) (run ^. repoUser) (run ^. repoName)
33+ when (not hasAccess) $ throw (NoSuchRun runId)
34+ pure run
35+
36+getBuildWithAccess :: Access -> Maybe User -> BuildId -> M Build
37+getBuildWithAccess access user' buildId = do
38+ let accessCheck = case access of
39+ Read -> hasAccessTo
40+ Cancel -> canCancelBuild
41+ build <- DB.getBuild buildId
42+ hasAccess <- accessCheck user' (build ^. repoIsPublic) (build ^. reqUser) (build ^. repoUser) (build ^. repoName)
43+ when (not hasAccess) $ throw (NoSuchBuild buildId)
44+ pure build
45+
46+hasAccessTo :: Maybe User -> RepoPublicity -> GhLogin -> GhRepoOwner -> GhRepoName -> M Bool
47+hasAccessTo user' repoIsPublic reqUser owner name
48+ | user' ^? _Just . githubLogin == Just reqUser = pure True
49+ | otherwise = hasAccessToRepo user' repoIsPublic owner name
50+
51+hasAccessToRepo :: Maybe User -> RepoPublicity -> GhRepoOwner -> GhRepoName -> M Bool
52+hasAccessToRepo user' repoIsPublic owner name
53+ | isRepoPublic repoIsPublic = pure True
54+ | user' ^? _Just . subscriptionType == Just Admin = pure True
55+ | otherwise = case user' of
56+ Nothing -> pure False
57+ Just user -> do
58+ collaborators <- getCollaborators owner name
59+ case collaborators of
60+ RepoNotFound -> pure False
61+ GhCollaborators collaborators' -> pure $ (user ^. githubLogin) `elem` collaborators'
62+
63+getCollaborators :: GhRepoOwner -> GhRepoName -> M GhCollaborators
64+getCollaborators owner repo = do
65+ installationId <- getGarnixInstallationId owner repo
66+ case installationId of
67+ Nothing -> pure RepoNotFound
68+ Just id -> do
69+ iAuth <- getInstallation (Id $ fromInteger id)
70+ getRepoCollaborators iAuth owner repo
71+
72+canCancelBuild :: Maybe User -> RepoPublicity -> GhLogin -> GhRepoOwner -> GhRepoName -> M Bool
73+canCancelBuild user' _ reqUser owner name
74+ | user' ^? _Just . subscriptionType == Just Admin = pure True
75+ | user' ^? _Just . githubLogin == Just reqUser = pure True
76+ | otherwise = case user' of
77+ Nothing -> pure False
78+ Just user -> do
79+ collaborators <- getCollaborators owner name
80+ case collaborators of
81+ RepoNotFound -> pure False
82+ GhCollaborators collaborators' -> pure $ (user ^. githubLogin) `elem` collaborators'
backend/src/Garnix/AccessToken.hsnew file+28
@@ -0,0 +1,28 @@
1+module Garnix.AccessToken
2+ ( isAccessTokenValid,
3+ generateToken,
4+ )
5+where
6+
7+import Control.Monad.Extra
8+import Garnix.AccessToken.Types
9+import Garnix.DB qualified as DB
10+import Garnix.Monad
11+import Garnix.Password
12+import Garnix.Prelude
13+import Garnix.Types
14+
15+generateToken :: UserId -> Text -> AccessTokenScopes -> M AccessToken
16+generateToken userId name scopes = do
17+ token <- AccessToken <$> randomBase64 30
18+ hashedToken <- hashPassword $ getAccessTokenText token
19+ DB.insertAccessTokenForUser userId name scopes hashedToken
20+ pure token
21+
22+isAccessTokenValid :: UserId -> AccessToken -> (AccessTokenScopes -> Bool) -> M Bool
23+isAccessTokenValid userId token checkScopes = do
24+ accessTokenHashes <- DB.getAccessTokenHashesForUser userId
25+ flip anyM accessTokenHashes $ \(tokenId, hash, scopes) -> do
26+ let isValid = checkScopes scopes && validatePassword hash (getAccessTokenText token)
27+ when isValid $ DB.markAccessTokenUsed userId tokenId
28+ pure isValid
backend/src/Garnix/AccessToken/Types.hsnew file+45
@@ -0,0 +1,45 @@
1+module Garnix.AccessToken.Types
2+ ( AccessToken (..),
3+ AccessTokenMetadata (..),
4+ AccessTokenScopes (..),
5+ )
6+where
7+
8+import Control.Lens
9+import Data.Aeson qualified as Aeson
10+import Garnix.Prelude
11+
12+newtype AccessToken = AccessToken {getAccessTokenText :: Text}
13+ deriving stock (Eq, Show, Generic, Ord)
14+
15+instance ToJSON AccessToken where
16+ toJSON x = Aeson.String $ getAccessTokenText x
17+
18+data AccessTokenMetadata = AccessTokenMetadata
19+ { _accessTokenMetadataId :: Int64,
20+ _accessTokenMetadataName :: Text,
21+ _accessTokenMetadataCreated :: UTCTime,
22+ _accessTokenMetadataLastUsed :: Maybe UTCTime,
23+ _accessTokenMetadataScopes :: AccessTokenScopes
24+ }
25+ deriving stock (Eq, Show, Generic)
26+
27+instance ToJSON AccessTokenMetadata where
28+ toJSON = ourToJSON
29+
30+data AccessTokenScopes = AccessTokenScopes
31+ { cache :: Bool,
32+ api :: Bool
33+ }
34+ deriving stock (Eq, Show, Generic)
35+ deriving anyclass (ToJSON)
36+
37+instance FromJSON AccessTokenScopes where
38+ parseJSON = Aeson.withObject "AccessTokenScopes" $ \o -> do
39+ cache <-
40+ o Aeson..:? "cache"
41+ <&> fromMaybe False
42+ api <-
43+ o Aeson..:? "api"
44+ <&> fromMaybe False
45+ pure $ AccessTokenScopes {cache, api}
backend/src/Garnix/Async.hsnew file+54
@@ -0,0 +1,54 @@
1+module Garnix.Async where
2+
3+import Control.Concurrent hiding (threadDelay)
4+import Control.Concurrent.Async.Lifted (race)
5+import Control.Exception
6+import Garnix.Duration
7+import Garnix.Prelude
8+import Garnix.Types
9+
10+-- | Replacement for `Async`. The main differences are:
11+--
12+-- 1. `Promise`s also deal with monadic errors, not just runtime exceptions.
13+--
14+-- 2. The default behavior when running multiple `Promise`s concurrently is
15+-- that an error or exception in one of the threads does *not* cancel the
16+-- others. (See for example `Garnix.Monad.Async.doAllConcurrently`.)
17+newtype Promise a = Promise {getPromise :: MVar (PromiseResult a)}
18+
19+data PromiseResult a
20+ = PromiseException SomeException
21+ | PromiseThrew ErrorWithContext
22+ | PromiseSucceeded a
23+ deriving stock (Functor, Generic, Show)
24+
25+instance Applicative PromiseResult where
26+ pure = PromiseSucceeded
27+
28+ liftA2 :: (a -> b -> c) -> PromiseResult a -> PromiseResult b -> PromiseResult c
29+ liftA2 f ra rb = case ra of
30+ PromiseSucceeded a -> case rb of
31+ PromiseSucceeded b -> PromiseSucceeded $ f a b
32+ PromiseException e -> PromiseException e
33+ PromiseThrew e -> PromiseThrew e
34+ PromiseException e -> PromiseException e
35+ PromiseThrew e -> PromiseThrew e
36+
37+instance Monad PromiseResult where
38+ (>>=) :: PromiseResult a -> (a -> PromiseResult b) -> PromiseResult b
39+ ra >>= f = case ra of
40+ PromiseSucceeded a -> f a
41+ PromiseException e -> PromiseException e
42+ PromiseThrew e -> PromiseThrew e
43+
44+resolveIO :: Promise a -> IO (PromiseResult a)
45+resolveIO (Promise mvar) = readMVar mvar
46+
47+timeout :: (MonadBaseControl IO m, MonadIO m) => Duration -> m a -> m (Maybe a)
48+timeout delay action =
49+ race (liftIO $ threadDelay delay) action >>= \case
50+ Left _ -> pure Nothing
51+ Right v -> pure $ Just v
52+
53+isResolved :: Promise a -> IO Bool
54+isResolved (Promise mvar) = isJust <$> tryTakeMVar mvar
backend/src/Garnix/Attribute.hsnew file+274
@@ -0,0 +1,274 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+
3+module Garnix.Attribute where
4+
5+import Data.Text qualified as T
6+import Garnix.Monad
7+import Garnix.Prelude
8+import Garnix.Types
9+import Garnix.YamlConfig
10+
11+-- | An attribute is something like '.#nixosConfigurations.container'.
12+data Attribute = Attribute
13+ { _attributePackageType :: PackageType,
14+ _attributeSystem :: Maybe System,
15+ _attributePackageName :: Maybe PackageName,
16+ _attributeExtension :: Maybe Text
17+ }
18+ deriving stock (Eq, Show, Generic)
19+
20+makeFields ''Attribute
21+
22+-- In time the '.' component should also be part of the `Attribute`. But
23+-- for now we don't use anything other than '.'
24+localAttr :: FlakeDir -> Attribute -> M Text
25+localAttr flakeDir attr = do
26+ flakeDir' <- safeGetAbsoluteFlakeDir flakeDir
27+ pure $ cs flakeDir' <> "#" <> review asAttribute attr
28+
29+(<.>) :: Text -> Text -> Text
30+a <.> b = a <> "." <> b
31+
32+-- Attrs that we want to build the subAttrs of
33+allParentAttrs :: [Attribute]
34+allParentAttrs =
35+ [ homeConfigurationsAttr,
36+ darwinConfigurationsAttr,
37+ nixosConfigurationsAttr
38+ ]
39+ <> [ typ sys
40+ | typ <- [packagesAttr, checksAttr, devShellsAttr],
41+ sys <- supportedSystems
42+ ]
43+
44+-- Attrs that we want to build directly. They may, however, not exist
45+allDirectAttrs :: [Attribute]
46+allDirectAttrs =
47+ [typ sys | typ <- [defaultDevShellAttr, defaultPackageAttr], sys <- supportedSystems]
48+
49+addNixosExtension :: Attribute -> Attribute
50+addNixosExtension attr
51+ | attr ^. packageType == TypeHomeConfiguration =
52+ attr & extension ?~ "activationPackage"
53+ | attr ^. packageType == TypeDarwinConfiguration =
54+ attr & extension ?~ "config.system.build.toplevel"
55+ | attr ^. packageType == TypeNixosConfiguration =
56+ attr & extension ?~ "config.system.build.toplevel"
57+ | otherwise = attr
58+
59+homeConfigurationsAttr :: Attribute
60+homeConfigurationsAttr = attrPT TypeHomeConfiguration
61+
62+darwinConfigurationsAttr :: Attribute
63+darwinConfigurationsAttr = attrPT TypeDarwinConfiguration
64+
65+nixosConfigurationsAttr :: Attribute
66+nixosConfigurationsAttr = attrPT TypeNixosConfiguration
67+
68+packageAttr :: System -> PackageName -> Attribute
69+packageAttr sys n = packagesAttr sys & packageName ?~ n
70+
71+packagesAttr :: System -> Attribute
72+packagesAttr sys = attrPT TypePackage & system ?~ sys
73+
74+checkAttr :: System -> PackageName -> Attribute
75+checkAttr sys n = checksAttr sys & packageName ?~ n
76+
77+checksAttr :: System -> Attribute
78+checksAttr sys = attrPT TypeCheck & system ?~ sys
79+
80+devShellAttr :: System -> PackageName -> Attribute
81+devShellAttr sys n = devShellsAttr sys & packageName ?~ n
82+
83+devShellsAttr :: System -> Attribute
84+devShellsAttr sys = attrPT TypeDevShell & system ?~ sys
85+
86+defaultDevShellAttr :: System -> Attribute
87+defaultDevShellAttr sys =
88+ attrPT TypeDefaultDevShell
89+ & system
90+ ?~ sys
91+ -- See Note [Nothing vs "" in packageName]
92+ & packageName
93+ ?~ ""
94+
95+defaultPackageAttr :: System -> Attribute
96+defaultPackageAttr sys =
97+ attrPT TypeDefaultPackage
98+ & system
99+ ?~ sys
100+ -- See Note [Nothing vs "" in packageName]
101+ & packageName
102+ ?~ ""
103+
104+overallAttr :: Attribute
105+overallAttr = attrPT TypeOverall
106+
107+attribute :: Build -> Attribute
108+attribute build =
109+ Attribute
110+ (build ^. packageType)
111+ (build ^. system . maybeSystemIso)
112+ (Just $ build ^. package)
113+ Nothing
114+
115+addSubAttr :: Attribute -> Text -> Maybe Attribute
116+addSubAttr attr sub = (review asAttribute attr <.> sub) ^? asAttribute
117+
118+asAttribute :: Prism' Text Attribute
119+asAttribute = prism there back
120+ where
121+ there (Attribute TypeOverall Nothing Nothing Nothing) = ""
122+ there (Attribute typ sys name' ext) =
123+ let addName att = case name' of
124+ Nothing -> att
125+ Just (PackageName n) -> att ++ [n]
126+ addExt att = case ext of
127+ Nothing -> att
128+ Just e -> att ++ [e]
129+ in T.intercalate "." . filter (not . T.null) . addExt . addName $ case (typ, sys) of
130+ (TypeHomeConfiguration, Just _) -> error "Can't convert home package with system"
131+ (TypeHomeConfiguration, Nothing) ->
132+ ["homeConfigurations"]
133+ (TypeDarwinConfiguration, Just _) -> error "Can't convert darwin package with system"
134+ (TypeDarwinConfiguration, Nothing) ->
135+ ["darwinConfigurations"]
136+ (TypeNixosConfiguration, Just _) -> error "Can't convert nixos package with system"
137+ (TypeNixosConfiguration, Nothing) ->
138+ ["nixosConfigurations"]
139+ (TypePackage, Nothing) -> error "Can't convert package with no system"
140+ (TypePackage, Just system') ->
141+ ["packages", system' ^. systemTextIso]
142+ (TypeCheck, Nothing) -> error "Can't convert check with no system"
143+ (TypeCheck, Just system') ->
144+ ["checks", system' ^. systemTextIso]
145+ (TypeDevShell, Nothing) -> error "Can't convert devShell with no system"
146+ (TypeDevShell, Just system') ->
147+ ["devShells", system' ^. systemTextIso]
148+ (TypeDefaultDevShell, Nothing) -> error "Can't convert devShell with no system"
149+ (TypeDefaultDevShell, Just system') ->
150+ ["devShell", system' ^. systemTextIso]
151+ (TypeDefaultPackage, Nothing) -> error "Can't convert devShell with no system"
152+ (TypeDefaultPackage, Just system') ->
153+ ["defaultPackage", system' ^. systemTextIso]
154+ (TypeApp, Nothing) -> error "Can't convert app with no system"
155+ (TypeApp, Just system') ->
156+ ["apps", system' ^. systemTextIso]
157+ (TypeOverall, _) -> error "Can't convert overall package"
158+ back i =
159+ let nameAndExt a x = case x of
160+ [] -> a
161+ [name'] -> a & packageName ?~ PackageName name'
162+ name' : r ->
163+ a
164+ & packageName
165+ ?~ PackageName name'
166+ & extension
167+ ?~ T.intercalate "." r
168+ ext a r = case r of
169+ [] -> a
170+ _ -> a & extension ?~ T.intercalate "." r
171+ in case T.splitOn "." i of
172+ [] -> Right overallAttr
173+ "homeConfigurations" : t ->
174+ Right $ homeConfigurationsAttr `nameAndExt` t
175+ "darwinConfigurations" : t ->
176+ Right $ darwinConfigurationsAttr `nameAndExt` t
177+ "nixosConfigurations" : t ->
178+ Right $ nixosConfigurationsAttr `nameAndExt` t
179+ "packages" : system' : t ->
180+ Right $ packagesAttr (system' ^. from systemTextIso) `nameAndExt` t
181+ "checks" : system' : t ->
182+ Right $ checksAttr (system' ^. from systemTextIso) `nameAndExt` t
183+ "devShells" : system' : t ->
184+ Right $ devShellsAttr (system' ^. from systemTextIso) `nameAndExt` t
185+ "defaultPackage" : system' : t ->
186+ Right $ defaultPackageAttr (system' ^. from systemTextIso) `ext` t
187+ "devShell" : system' : t ->
188+ Right $ defaultDevShellAttr (system' ^. from systemTextIso) `ext` t
189+ _ -> Left i
190+
191+-- * Matching
192+
193+matchesConfig :: Attribute -> GarnixConfig -> Maybe Branch -> Bool
194+matchesConfig attr cfg branch =
195+ any matchesBuildSection $ cfg ^. buildSections
196+ where
197+ matchesBuildSection buildSection =
198+ branchMatches buildSection
199+ && anyMatchAttr (buildSection ^. includeSection)
200+ && not (anyMatchAttr (buildSection ^. excludeSection))
201+ anyMatchAttr = (any . matches) attr
202+ branchMatches s = case (s ^. branchSection, branch) of
203+ (Nothing, _) -> True
204+ (a, b) -> a == b
205+
206+-- | Check whether a child of the attribute *could* be matched by the config
207+--
208+-- Note that this doesn't take into account the excludes section.
209+mightMatchConfig :: Attribute -> GarnixConfig -> Bool
210+mightMatchConfig attr cfg =
211+ let includes = concatMap (^. includeSection) (cfg ^. buildSections)
212+ in any (\i -> attr `mightMatch` i) includes
213+
214+matches :: Attribute -> AttributeMatcher -> Bool
215+matches attr matcher = case T.splitOn "." $ review asAttribute attr of
216+ [a, b, c] -> case matcher ^. thirdPart of
217+ Nothing -> False
218+ Just c' ->
219+ a
220+ `partMatches` (matcher ^. firstPart . to cs)
221+ && b
222+ `partMatches` (matcher ^. secondPart . to cs)
223+ && c
224+ `partMatches` cs c'
225+ [a, b] -> case matcher ^. thirdPart of
226+ Nothing ->
227+ a
228+ `partMatches` (matcher ^. firstPart . to cs)
229+ && b
230+ `partMatches` (matcher ^. secondPart . to cs)
231+ Just _ -> False
232+ _ -> False
233+ where
234+ partMatches :: Text -> Text -> Bool
235+ partMatches _ "*" = True
236+ partMatches attPart matchPart = attPart == matchPart
237+
238+-- | Checks whether a attribute could have a child matched by a matchers
239+mightMatch :: Attribute -> AttributeMatcher -> Bool
240+mightMatch attr matcher = case T.splitOn "." $ review asAttribute attr of
241+ [a, b] -> case matcher ^. thirdPart of
242+ Nothing -> False
243+ Just _ ->
244+ a
245+ `partMatches` (matcher ^. firstPart . to cs)
246+ && b
247+ `partMatches` (matcher ^. secondPart . to cs)
248+ [a] -> case matcher ^. thirdPart of
249+ Nothing ->
250+ a `partMatches` (matcher ^. firstPart . to cs)
251+ Just _ -> False
252+ _ -> False
253+ where
254+ partMatches :: Text -> Text -> Bool
255+ partMatches _ "*" = True
256+ partMatches attPart matchPart = attPart == matchPart
257+
258+-- * Helpers
259+
260+attrPT :: PackageType -> Attribute
261+attrPT pt = Attribute pt Nothing Nothing Nothing
262+
263+{- Note [Nothing vs "" in packageName]
264+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
265+ We distinguish between Nothing and "" in packageName. The difference is
266+ that Nothing indicates that we don't know what the package name is. "" on
267+ the other hand indicates that there *is* no package name. So:
268+
269+ defaultPackage.x86_64-linux ==> packageName is Just ""
270+ packages.x86_64-linux ==> packageName is Nothing
271+
272+ This isn't perfect, since encoding it like this allows both forms to
273+ exist in the Attribute form, but not the Text form.
274+-}
backend/src/Garnix/Build.hsnew file+81
@@ -0,0 +1,81 @@
1+module Garnix.Build
2+ ( buildFlake,
3+ rerunBuild,
4+ buildModule,
5+ )
6+where
7+
8+import Control.Concurrent.Async.Lifted
9+import Cradle hiding (ExitCode)
10+import Garnix.Async
11+import Garnix.Build.Checkout (withAuthorization)
12+import Garnix.Build.Checkout qualified as Checkout
13+import Garnix.Build.Flake
14+import Garnix.Build.FodCheck qualified as FodCheck
15+import Garnix.Build.Helpers
16+import Garnix.Build.MetaCheck qualified as MetaCheck
17+import Garnix.Build.Module qualified as Module
18+import Garnix.Build.Package (doBuild)
19+import Garnix.Build.Reporting
20+import Garnix.DB qualified as DB
21+import Garnix.DB.ModuleValues qualified as ModuleValues
22+import Garnix.Entitlements qualified as Entitlements
23+import Garnix.Monad
24+import Garnix.Monad.Async (emptyPromise, spawn)
25+import Garnix.Monad.Concurrency
26+import Garnix.Prelude
27+import Garnix.Reporters.GithubReporter (mkGithubReporter)
28+import Garnix.Reporters.OpenSearchReporter (openSearchReporter)
29+import Garnix.Types as Types
30+import Garnix.YamlConfig (flakeDir)
31+
32+buildModule :: GhLogin -> ModuleValues.GetRepoAndModuleValues -> M CommitInfo
33+buildModule reqUser modules = do
34+ commitInfo <- Module.getCommitInfo reqUser modules
35+ let reporter = openSearchReporter <> mkGithubReporter (commitInfo ^. repoInfo) (commitInfo ^. commit)
36+ ensureNoFlakeOnCommit commitInfo
37+ let defaultBranch = maybe (Branch "main") identity $ commitInfo ^. branch
38+ remote = Module.remoteWithFlake defaultBranch modules Checkout.remoteWithConfig
39+ forkM $ runBuildFlake reporter ModulePreview commitInfo remote
40+ pure commitInfo
41+ where
42+ ensureNoFlakeOnCommit :: CommitInfo -> M ()
43+ ensureNoFlakeOnCommit commitInfo =
44+ doesRepoFileExist commitInfo "flake.nix" >>= \case
45+ FileExists -> do
46+ log Informational "flake.nix already exists - skipping."
47+ throw ModuleErrorFlakeExists
48+ _ -> pure ()
49+
50+-- | Build the flake, sending updates to github and the DB as you go along.
51+--
52+-- N.B.: This function should return as quickly as possible so the connection
53+-- can be closed.
54+buildFlake :: (HasCallStack) => Reporter -> CommitInfo -> M (Promise ())
55+buildFlake = curry $ mockable #buildFlakeMock $ \(reporter, commitInfo) -> do
56+ filesExist <-
57+ concurrently
58+ (doesRepoFileExist commitInfo "flake.nix")
59+ (doesRepoFileExist commitInfo "garnix.yaml")
60+ case filesExist of
61+ (FileDoesntExist, FileDoesntExist) -> do
62+ log Informational "No flake.nix or garnix.yaml - skipping."
63+ emptyPromise
64+ _ -> spawn $ runBuildFlake reporter Webhook commitInfo Checkout.remoteWithConfig
65+
66+rerunBuild :: Reporter -> Build -> CommitInfo -> M ()
67+rerunBuild reporter build commitInfo = do
68+ MetaCheck.update reporter commitInfo
69+ runReporter <- createNewRun reporter $ ReportBuild (reportNameForBuild build) build
70+ let build' = build & githubRunId .~ Garnix.Monad.ghRunId runReporter
71+ DB.reportBuildResultDB build' <?> "Adding build github ID to DB"
72+ reportOnError runReporter build' commitInfo $ do
73+ repoConfig <- DB.getRepoConfig (commitInfo ^. repoInfo . ghRepoOwner) (commitInfo ^. repoInfo . ghRepoName)
74+ plan <- Entitlements.getPlan (build ^. repoUser)
75+ Checkout.runWithCheckout Checkout.remoteWithConfig commitInfo $ \config -> do
76+ withAuthorization (config ^. flakeDir) repoConfig commitInfo $ do
77+ reportBuildResult runReporter build'
78+ void $ withInternalCacheToken (commitInfo ^. reqUser) $ do
79+ FodCheck.withFodChecker reporter commitInfo plan $ \fodChecker -> do
80+ doBuild fodChecker runReporter Webhook (config ^. flakeDir) repoConfig plan build'
81+ MetaCheck.update reporter commitInfo
backend/src/Garnix/Build/Action.hsnew file+357
@@ -0,0 +1,357 @@
1+{-# LANGUAGE OverloadedLists #-}
2+
3+module Garnix.Build.Action
4+ ( getActionAppAttributes,
5+ run,
6+ )
7+where
8+
9+import Control.Lens
10+import Cradle (ProcessConfiguration)
11+import Cradle qualified
12+import Data.Aeson ((.:))
13+import Data.Aeson qualified as JSON
14+import Data.Aeson.Types qualified as JSON
15+import Data.ByteString qualified as BS
16+import Data.Map qualified as Map
17+import Data.Text qualified as T
18+import Data.Text.IO qualified as T
19+import Garnix.API.Keys qualified as Keys
20+import Garnix.Async qualified as Async
21+import Garnix.Attribute
22+import Garnix.DB qualified as DB
23+import Garnix.Duration
24+import Garnix.Monad
25+import Garnix.Nix.StorePath (withStorePath)
26+import Garnix.Nix.Types qualified as Nix
27+import Garnix.NixConfig (addNixConfigEnvironment)
28+import Garnix.NixConfig qualified as NixConfig
29+import Garnix.Prelude
30+import Garnix.Reporters.Utils (runWithRunReporter, withRunReporter)
31+import Garnix.Request
32+import Garnix.SafeUnix (safeCreatePipe)
33+import Garnix.Sandbox
34+import Garnix.Types
35+import Garnix.YamlConfig (Action (..), ActionSandboxType (..), sandboxType)
36+import Garnix.YamlConfig qualified as Config
37+import System.Directory qualified as IO
38+import System.IO (hClose, hFlush)
39+
40+getActionAppAttributes :: Config.GarnixConfig -> [(Attribute, Action)]
41+getActionAppAttributes config = go <$> config ^. Config.actions
42+ where
43+ go :: Config.Action -> (Attribute, Action)
44+ go a@(Config.Action package _ _ _) =
45+ ( Attribute
46+ { _attributePackageType = TypeApp,
47+ _attributeSystem = Just X8664Linux,
48+ _attributePackageName = Just package,
49+ _attributeExtension = Nothing
50+ },
51+ a
52+ )
53+
54+allowedSharedResourcesUsers :: [Text]
55+allowedSharedResourcesUsers =
56+ T.toLower
57+ <$> [ "garnix-io",
58+ "bellroy",
59+ "emanueljg",
60+ "psychefoundation",
61+ "imiron-io",
62+ "oliverlee",
63+ "jameshaydon",
64+ "ilyakooo0",
65+ "montelot",
66+ "soenkehahn"
67+ ]
68+
69+run :: FlakeDir -> RepoConfig -> Reporter -> CommitInfo -> Attribute -> Action -> Build -> M ()
70+run flakeDir repoConfig reporter commitInfo attr actionConfig build =
71+ withTextSpan ("action_run", review asAttribute attr) $ do
72+ ensureBuildIsApp build
73+ derivation <- getBuildDerivation build
74+ command <- evaluateAppExecPath flakeDir repoConfig build
75+ let name = "action " <> maybe "Unknown" cs (attr ^. packageName)
76+ let environmentVars =
77+ Map.fromList
78+ [ ("GARNIX_CI", "true"),
79+ ("GARNIX_BRANCH", maybe "" getBranch $ build ^. branch),
80+ ("GARNIX_COMMIT_SHA", build ^. gitCommit . to getCommitHash)
81+ ]
82+ run <- DB.newRun name commitInfo
83+ withRunReporter reporter (ReportRun run) $ \runReporter -> withStorePath build "out" $ \_ -> do
84+ ensureAllowedSandboxType (build ^. repoUser) actionConfig
85+ validateApplication command
86+ copyClosure derivation <?> "Action: copy closure"
87+ privKey <- case build ^. prFromFork of
88+ Nothing -> do
89+ Just . snd <$> Keys.getActionKeys (build ^. repoUser) (build ^. repoName) (build ^. package)
90+ Just _fork -> pure Nothing
91+ withTimeout $ runAction runReporter actionConfig command privKey environmentVars <?> "Action: execute"
92+ where
93+ withTimeout :: M a -> M a
94+ withTimeout inner = do
95+ outerActionTimeout <-
96+ view (#action . #timeoutDuration)
97+ <&> addDuration (fromMinutes @Int 10)
98+ Async.timeout outerActionTimeout inner
99+ >>= \case
100+ Nothing -> throw ActionExecutionTimeout
101+ Just r -> pure r
102+
103+ ensureBuildIsApp :: Build -> M ()
104+ ensureBuildIsApp build =
105+ case build ^. packageType of
106+ TypeApp -> pure ()
107+ pkgType -> do
108+ log Critical $ "Broken invariant: should not try to run something that's not an action: " <> show attr <> " (with type " <> show pkgType <> ")"
109+ throw $ OtherError "Internal error: attempted to run an action on an attribute that's not an action."
110+
111+ getBuildDerivation :: Build -> M Nix.DrvPath
112+ getBuildDerivation build = do
113+ let drv = case build ^. drvPath of
114+ Nothing -> Left "build has no drvPath"
115+ Just drv' -> Nix.DrvPath <$> Nix.parseStorePath drv'
116+ case drv of
117+ Left err -> do
118+ log Warning $ "Unexpected: could not find the app derivation: " <> err
119+ throw $ OtherError "Cannot run action: could not find the derivation needed to run the application. Did you make sure the 'program' is an executable path that's part of a derivation?"
120+ Right derivation -> pure derivation
121+
122+validateApplication :: Nix.AppExecPath -> M ()
123+validateApplication (Nix.AppExecPath path) = do
124+ when (not $ "/nix/store/" `T.isPrefixOf` path)
125+ $ throw
126+ $ ActionPreconditionNixStore path
127+ pathExists <- liftIO $ IO.doesPathExist (cs path)
128+ when (not pathExists)
129+ $ throw
130+ $ ActionPreconditionFileExists path
131+
132+ensureAllowedSandboxType :: GhRepoOwner -> Action -> M ()
133+ensureAllowedSandboxType owner actionConfig = do
134+ when
135+ ( T.toLower (getGhLogin $ getGhRepoOwner owner)
136+ `notElem` allowedSharedResourcesUsers
137+ && actionConfig
138+ ^. sandboxType == SharedResources
139+ )
140+ $ throw
141+ $ ActionSandboxTypeNotAllowed "shared-resources"
142+
143+copyClosure :: Nix.DrvPath -> M ()
144+copyClosure path = do
145+ actionTimeout <- view $ #action . #timeoutDuration
146+ retryingFor actionTimeout $ do
147+ actionHost <- view $ #action . #runnerHost
148+ sshKey <- view $ #action . #runnerSshKey
149+ (ip, sshArgs) <- sshArgsFor actionHost sshKey
150+ nixConfig <- view #userNixConfig
151+ let args = ["copy", "--no-check-sigs", "--to", "ssh-ng://action-runner@" <> cs ip, cs path <> "^*"]
152+ result <-
153+ Cradle.run
154+ $ Cradle.cmd "nix"
155+ & Cradle.addArgs args
156+ & NixConfig.addNixConfigEnvironment nixConfig
157+ & Cradle.modifyEnvVar "NIX_SSHOPTS" (const $ Just $ cs $ T.intercalate " " sshArgs)
158+ & Cradle.silenceStderr
159+ case result of
160+ (Cradle.ExitFailure code, Cradle.StdoutRaw out, Cradle.StderrRaw err) -> do
161+ log Warning
162+ $ "copyClosure error stdout ("
163+ <> cs out
164+ <> ") stderr ("
165+ <> cs err
166+ <> ")"
167+ throw RunProcessError {command = "nix", arguments = args, stdErr = cs err, stdOut = cs out, exitCode = code}
168+ (Cradle.ExitSuccess, Cradle.StdoutRaw out, Cradle.StderrRaw err) -> do
169+ log Informational
170+ $ "copyClosure success ("
171+ <> cs out
172+ <> ") stderr ("
173+ <> cs err
174+ <> ")"
175+
176+runAction :: RunReporter -> Action -> Nix.AppExecPath -> Maybe PrivateKey -> Map.Map Text Text -> M ()
177+runAction runReporter actionConfig path actionPrivateKey environmentVars = do
178+ repoSecretsKey <- view #repoSecretsEncryptionKeyPath
179+ privKey <- case actionPrivateKey of
180+ Nothing -> pure "none"
181+ Just key -> do
182+ liftIO (unsafeDecryptPrivateKey key repoSecretsKey) >>= \case
183+ Left _ -> throw ActionKeyDecryptionFailure
184+ Right k -> pure k
185+ proc <- runSshOnActionRunner actionConfig (Nix.getAppExecPath path) privKey environmentVars
186+ exitCode <- runWithRunReporter runReporter proc
187+ when
188+ ( exitCode
189+ == Cradle.ExitFailure 124
190+ && actionConfig
191+ ^. sandboxType == SharedResources
192+ )
193+ $ do
194+ throw ActionExecutionTimeout
195+ when
196+ ( exitCode
197+ == Cradle.ExitFailure 255
198+ && actionConfig
199+ ^. sandboxType == FastStartup
200+ )
201+ $ do
202+ throw ActionExecutionTimeout
203+ let status = case exitCode of
204+ Cradle.ExitSuccess -> RunReportStatusSuccess
205+ _ -> RunReportStatusFailure
206+ reportComplete runReporter status
207+
208+copyRepoToActionRunner :: M FilePath
209+copyRepoToActionRunner = do
210+ repo <- view #workingDir
211+ actionHost <- view $ #action . #runnerHost
212+ sshKey <- view $ #action . #runnerSshKey
213+ unlessM (liftIO $ IO.doesDirectoryExist (repo </> ".git"))
214+ $ throw
215+ $ OtherError
216+ $ "Expected directory "
217+ <> cs repo
218+ <> "to be the repo root"
219+ (ip, sshArgs) <- sshArgsFor actionHost sshKey
220+ (_, port) <- splitPortFromIP actionHost
221+ (sshExitCode, Cradle.StdoutTrimmed tempDir, Cradle.StderrRaw sshErr) <-
222+ Cradle.run $ Cradle.cmd "ssh"
223+ & Cradle.addArgs (["action-runner@" <> ip] <> sshArgs <> ["--", "mktemp", "-d"])
224+ unless (sshExitCode == Cradle.ExitSuccess)
225+ $ throw
226+ $ OtherError
227+ $ "ssh failed. Error: <> "
228+ <> show sshErr
229+ (rsyncExitCode, Cradle.StderrRaw rsyncErr) <-
230+ Cradle.run $ Cradle.cmd "rsync"
231+ & Cradle.addArgs ["-a", "-e", "ssh -p " <> port <> " -i " <> sshKey, cs (repo <> "/"), "action-runner@[" <> ip <> "]:" <> tempDir]
232+ unless (rsyncExitCode == Cradle.ExitSuccess)
233+ $ throw
234+ $ OtherError
235+ $ "rsync failed. Error: <> "
236+ <> show rsyncErr
237+ return (cs tempDir)
238+
239+runSshOnActionRunner :: Action -> Text -> Text -> Map.Map Text Text -> M Cradle.ProcessConfiguration
240+runSshOnActionRunner actionConfig command stdin environmentVars = do
241+ actionHost <- view $ #action . #runnerHost
242+ sshKey <- view $ #action . #runnerSshKey
243+ actionTimeout <- view $ #action . #timeoutDuration
244+ (ip, sshArgs) <- sshArgsFor actionHost sshKey
245+ repoDirEnvVar <-
246+ if actionConfig ^. Config.withRepoContents
247+ then do
248+ log Informational "Copying repo contents"
249+ dir <- copyRepoToActionRunner
250+ pure $ "ACTION_REPO_DIR=" <> dir
251+ else do
252+ log Informational "Running without repo contents"
253+ pure ""
254+ let actionRunnerCmd = case actionConfig ^. sandboxType of
255+ FastStartup -> "action-runner"
256+ SharedResources -> "bwrap-action-runner"
257+ let args =
258+ ["action-runner@" <> ip]
259+ <> sshArgs
260+ <> ["--"]
261+ <> [ T.unwords
262+ $ ( ( environmentVars
263+ & Map.toList
264+ & fmap (\(k, v) -> k <> "=" <> shellEscape v)
265+ )
266+ <> [cs repoDirEnvVar]
267+ )
268+ <> [ actionRunnerCmd,
269+ shellEscape command,
270+ shellEscape $ show @Int $ floor $ toSeconds actionTimeout
271+ ]
272+ ]
273+ (readEnd, writeEnd) <- liftIO safeCreatePipe
274+ void . fork . liftIO $ do
275+ T.hPutStr writeEnd stdin
276+ hFlush writeEnd
277+ hClose writeEnd
278+ pure
279+ $ Cradle.cmd "ssh"
280+ & Cradle.addArgs args
281+ & Cradle.setStdinHandle readEnd
282+ where
283+ shellEscape :: Text -> Text
284+ shellEscape t = "'" <> T.replace "'" "'\"'\"'" t <> "'"
285+
286+sshArgsFor :: Text -> Text -> M (Text, [Text])
287+sshArgsFor server key = do
288+ (i, port) <- splitPortFromIP server
289+ pure
290+ ( i,
291+ [ "-q",
292+ "-o",
293+ "BatchMode=yes",
294+ "-p",
295+ port,
296+ "-i",
297+ cs key
298+ ]
299+ )
300+
301+splitPortFromIP :: Text -> M (Text, Text)
302+splitPortFromIP ip = case T.splitOn ":" ip of
303+ [_] -> pure (ip, "22")
304+ [ip', p] -> pure (ip', p)
305+ _ -> throw $ OtherError ("Action host IP not valid: " <> ip)
306+
307+evaluateAppExecPath :: FlakeDir -> RepoConfig -> Build -> M Nix.AppExecPath
308+evaluateAppExecPath flakeDir repoConfig build = do
309+ nixConfig <- view #userNixConfig
310+ workingDir <- view #workingDir
311+ cacheDir <- getNixXdgCacheDir
312+ let appDerivation = "a: [{ run = a.program; }]"
313+ attr <- localAttr flakeDir . addNixosExtension . attribute $ build
314+ (exitCode, stdout, Cradle.StderrRaw stderr) <-
315+ (>>= Cradle.run)
316+ $ Cradle.cmd "comment"
317+ & Cradle.addArgs
318+ [ buildComment build,
319+ "--",
320+ "prlimit",
321+ "--as=" <> show (toBytes $ repoConfig ^. maxEvalMemory),
322+ "nix",
323+ "eval",
324+ attr,
325+ "--apply",
326+ appDerivation,
327+ "--json"
328+ ]
329+ & addNixConfigEnvironment nixConfig
330+ & Cradle.setWorkingDir workingDir
331+ & pure
332+ & inNixSandbox [] (Just cacheDir)
333+ case exitCode of
334+ Cradle.ExitFailure _ -> throw $ ActionEvaluationFailure $ cs stderr
335+ Cradle.ExitSuccess -> errorOnParseFailure $ parseNixEvalOutput stdout
336+ where
337+ errorOnParseFailure :: Maybe Nix.AppExecPath -> M Nix.AppExecPath
338+ errorOnParseFailure =
339+ \case
340+ Nothing -> do
341+ log Warning "Unexpected: could not find the action path to run."
342+ throw $ OtherError "Cannot run action: could not find the application's path. Did you correctly supply the app 'program'?"
343+ Just app -> pure app
344+
345+ parseNixEvalOutput :: Cradle.StdoutRaw -> Maybe Nix.AppExecPath
346+ parseNixEvalOutput (Cradle.StdoutRaw stdout) =
347+ JSON.decode (BS.fromStrict stdout) >>= singletonList >>= JSON.parseMaybe go
348+ where
349+ singletonList :: JSON.Value -> Maybe JSON.Value
350+ singletonList =
351+ \case
352+ JSON.Array [single] -> Just single
353+ _ -> Nothing
354+
355+ go :: JSON.Value -> JSON.Parser Nix.AppExecPath
356+ go = JSON.withObject "app" $ \o ->
357+ Nix.AppExecPath <$> o .: "run"
backend/src/Garnix/Build/Checkout.hsnew file+63
@@ -0,0 +1,63 @@
1+module Garnix.Build.Checkout
2+ ( withCheckout,
3+ remoteWithConfig,
4+ Remote,
5+ runWithCheckout,
6+ withAuthorization,
7+ withBeforeAction,
8+ cleanRemote,
9+ )
10+where
11+
12+import Control.Lens qualified as Lens
13+import Control.Lens.Regex.Text qualified as Regex
14+import Garnix.Build.Helpers qualified as Internal
15+import Garnix.FlakeInputAuthorization (checkAuthorization)
16+import Garnix.Monad
17+import Garnix.Monad.ForkT (safeSystemTempDirectory)
18+import Garnix.Monad.Metrics
19+import Garnix.Monad.SubProcess.Deprecated qualified as Deprecated
20+import Garnix.Prelude
21+import Garnix.Types as Types
22+import Garnix.YamlConfig (GarnixConfig, getConfig)
23+
24+withCheckout :: CommitInfo -> M a -> M a
25+withCheckout commitInfo action = do
26+ remote <- getRemote commitInfo
27+ inRunsDirectory . Internal.withPrivateNixXdgCache $ do
28+ timingAs #gitCloneTime $ void $ Deprecated.runProc "git" ["clone", "--filter=tree:0", realRemoteUrl remote, "."] []
29+ void $ Deprecated.runProc "git" ["checkout", getCommitHash (commitInfo ^. commit)] []
30+ cleanRemote remote
31+ action
32+ where
33+ inRunsDirectory :: (HasCallStack) => M a -> M a
34+ inRunsDirectory action = do
35+ tmp <- safeSystemTempDirectory "garnix-runs"
36+ local (#workingDir .~ tmp) action
37+
38+-- Done so the token never leaks
39+cleanRemote :: RemoteUrl -> M ()
40+cleanRemote remote = do
41+ let cleanRemote = realRemoteUrl remote & [Regex.regex|x-access-token:.*@|] . Regex.match .~ ""
42+ void $ Deprecated.runProc "git" ["remote", "set-url", "origin", cleanRemote] []
43+
44+newtype Remote = Remote
45+ { runWithCheckout :: forall a. CommitInfo -> (GarnixConfig -> M a) -> M a
46+ }
47+
48+withBeforeAction :: M () -> Remote -> Remote
49+withBeforeAction before remote = Remote $ \commitInfo action -> do
50+ runWithCheckout remote commitInfo $ \garnixConfig -> do
51+ before
52+ action garnixConfig
53+
54+remoteWithConfig :: (HasCallStack) => Remote
55+remoteWithConfig = Remote $ \commitInfo action -> do
56+ withCheckout commitInfo $ do
57+ config <- getConfig
58+ action config
59+
60+withAuthorization :: FlakeDir -> RepoConfig -> CommitInfo -> M a -> M a
61+withAuthorization flakeDir repoConfig commitInfo action = do
62+ authConfig <- checkAuthorization flakeDir repoConfig commitInfo
63+ Lens.locally #userNixConfig (authConfig <>) action
backend/src/Garnix/Build/Evaluation.hsnew file+216
@@ -0,0 +1,216 @@
1+{-# LANGUAGE OverloadedLists #-}
2+{-# LANGUAGE TemplateHaskell #-}
3+
4+module Garnix.Build.Evaluation
5+ ( evaluateAttribute,
6+ Stdout (..),
7+ Stderr (..),
8+ RanCommand (..),
9+ ParsingError (..),
10+ NumberOfParsedResults (..),
11+ EvaluationResult (..),
12+ EvaluateError (..),
13+ _TimeoutReached,
14+ _NixEvaluationError,
15+ _ParseError,
16+ _UnexpectedNumberOfParsedResults,
17+ )
18+where
19+
20+import Cradle (ExitCode)
21+import Cradle hiding (ExitCode)
22+import Data.Aeson ((.:))
23+import Data.Aeson qualified as JSON
24+import Data.Aeson.Types qualified as JSON
25+import Data.ByteString (ByteString)
26+import Data.Either.Extra (mapLeft)
27+import Garnix.Async
28+import Garnix.Build.Types (EvaluationResult (..))
29+import Garnix.Duration
30+import Garnix.Incremental
31+import Garnix.Monad
32+import Garnix.Monad.Bubbling
33+import Garnix.Monad.Metrics
34+import Garnix.Monad.Pool (withPoolM)
35+import Garnix.Nix.Plan qualified as Nix
36+import Garnix.Nix.Types qualified as Nix
37+import Garnix.NixConfig (addNixConfigEnvironment)
38+import Garnix.Prelude
39+import Garnix.Sandbox
40+import Garnix.Types as Types
41+
42+newtype Stdout = Stdout ByteString
43+ deriving newtype (Show)
44+
45+newtype Stderr = Stderr ByteString
46+ deriving newtype (Show)
47+
48+newtype ParsingError = ParsingError String
49+ deriving newtype (Show)
50+
51+newtype NumberOfParsedResults = NumberOfParsedResults Int
52+ deriving newtype (Show)
53+
54+newtype RanCommand = RanCommand String
55+ deriving newtype (Show)
56+
57+data EvaluateError
58+ = TimeoutReached
59+ | NixEvaluationError Stderr RanCommand
60+ | ParseError ParsingError Stdout Stderr
61+ | AppEvalParseError Stdout Stderr
62+ | UnexpectedNumberOfParsedResults NumberOfParsedResults Stdout
63+ | AttributeIsSourceOutput Text
64+ deriving stock (Show)
65+
66+makePrisms ''EvaluateError
67+
68+data NixBuildPackage
69+ = SourceOutput Text
70+ | Derivation Nix.DrvPath Nix.BuildOutputs
71+
72+instance FromJSON NixBuildPackage where
73+ parseJSON =
74+ \case
75+ JSON.String s -> pure $ SourceOutput $ cs s
76+ json -> flip (JSON.withObject "nix evaluate attribute") json $ \o -> do
77+ drvPath <- o .: "drvPath" >>= parseStorePath
78+ outputs <- o .: "outputs" >>= mapM parseStorePath
79+ pure $ Derivation (Nix.DrvPath drvPath) (Nix.BuildOutputs outputs)
80+ where
81+ parseStorePath :: JSON.Value -> JSON.Parser Nix.StorePath
82+ parseStorePath v = do
83+ text :: Text <- parseJSON v
84+ case Nix.parseStorePath text of
85+ Left err -> JSON.parseFail $ cs err
86+ Right path -> pure path
87+
88+evaluateAttribute ::
89+ RepoConfig ->
90+ ProductPlan ->
91+ String ->
92+ FilePath ->
93+ Build ->
94+ Text ->
95+ M (Either EvaluateError EvaluationResult)
96+evaluateAttribute repoConfig plan cacheDir workingDir build attr = withBubbling $ \bubble -> do
97+ evalRes <-
98+ timingAs #evalDrvPathTime
99+ $ withPoolM nixEvalPool (build ^. repoUser)
100+ $ logDuration "nix eval for drvPath"
101+ $ withTextSpan ("phase", "eval")
102+ $ timeout (fromMinutes $ plan ^. packageEvaluationTimeout)
103+ $ do
104+ mDrvPath <-
105+ if build ^. packageType == TypeApp
106+ then bubble =<< mapLeft (uncurry AppEvalParseError) <$> getAppDrvPath repoConfig build workingDir cacheDir attr
107+ else pure Nothing
108+ result <- runNixEval (maybe attr cs mDrvPath)
109+ case result of
110+ (ExitFailure _, _, StderrRaw stderr) ->
111+ pure
112+ $ Left
113+ $ NixEvaluationError
114+ (Stderr stderr)
115+ (RanCommand $ cs $ "prlimit --as=" <> show (toBytes maxEvalMem) <> " nix build " <> attr <> " --dry-run --json")
116+ (ExitSuccess, StdoutRaw stdout, StderrRaw stderr) ->
117+ case mDrvPath of
118+ Nothing ->
119+ case JSON.eitherDecodeStrict' @[NixBuildPackage] stdout of
120+ Left err -> pure $ Left $ ParseError (ParsingError err) (Stdout stdout) (Stderr stderr)
121+ Right [result] -> case result of
122+ SourceOutput src -> pure $ Left $ AttributeIsSourceOutput src
123+ Derivation drvPath outputs -> do
124+ plan <- Nix.getPlanOf stderr
125+ pure $ Right $ EvaluationResult drvPath (Nix.planOutputs plan) outputs
126+ Right xs -> pure $ Left $ UnexpectedNumberOfParsedResults (NumberOfParsedResults $ length xs) (Stdout stdout)
127+ Just drvPath -> do
128+ plan <- Nix.getPlanOf stderr
129+ pure $ Right $ EvaluationResult drvPath (Nix.planOutputs plan) (Nix.BuildOutputs mempty)
130+ bubble $ fromMaybe (Left TimeoutReached) evalRes
131+ where
132+ maxEvalMem :: Memory
133+ maxEvalMem = repoConfig ^. maxEvalMemory
134+
135+ runNixEval :: Text -> M (ExitCode, StdoutRaw, StderrRaw)
136+ runNixEval attr' = do
137+ let withIncrementalExtraArgs action
138+ | build ^. wantsIncrementalism = withIntermediatesFlake build $ \case
139+ Nothing -> action [] []
140+ Just intermediates ->
141+ action
142+ [ "--override-input",
143+ "garnix-incrementalize",
144+ "path:" <> cs intermediates
145+ ]
146+ [(intermediates, ReadOnly)]
147+ | otherwise = action [] []
148+ nixConfig <- view #userNixConfig
149+ withIncrementalExtraArgs $ \incArgs extraSandboxPaths ->
150+ (>>= run)
151+ $ cmd "comment"
152+ & addArgs
153+ ( [ buildComment build,
154+ "--",
155+ "prlimit",
156+ "--as=" <> show (toBytes maxEvalMem),
157+ "nix",
158+ "build",
159+ attr' <> "^*",
160+ "--dry-run",
161+ "--extra-experimental-features",
162+ "fetch-closure",
163+ "--json"
164+ ]
165+ <> incArgs
166+ )
167+ & addNixConfigEnvironment nixConfig
168+ & setWorkingDir workingDir
169+ & pure
170+ & inNixSandbox extraSandboxPaths (Just cacheDir)
171+
172+getAppDrvPath :: RepoConfig -> Build -> FilePath -> FilePath -> Text -> M (Either (Stdout, Stderr) (Maybe Nix.DrvPath))
173+getAppDrvPath repoConfig build workingDir cacheDir attr = do
174+ let appDerivation = "a: [{ context = builtins.attrNames (builtins.getContext a.program); }]"
175+ nixConfig <- view #userNixConfig
176+ (exitCode, StdoutRaw stdout, StderrRaw stderr) <-
177+ (>>= run)
178+ $ cmd "comment"
179+ & addArgs
180+ [ buildComment build,
181+ "--",
182+ "prlimit",
183+ "--as=" <> show (toBytes $ repoConfig ^. maxEvalMemory),
184+ "nix",
185+ "eval",
186+ attr,
187+ "--apply",
188+ appDerivation,
189+ "--json"
190+ ]
191+ & addNixConfigEnvironment nixConfig
192+ & setWorkingDir workingDir
193+ & pure
194+ & inNixSandbox [] (Just cacheDir)
195+ pure $ case exitCode of
196+ ExitFailure _ -> Left (Stdout stdout, Stderr stderr)
197+ ExitSuccess -> Right $ parseNixEvalOutput stdout
198+ where
199+ parseNixEvalOutput :: ByteString -> Maybe Nix.DrvPath
200+ parseNixEvalOutput stdout =
201+ JSON.decode (cs stdout) >>= singletonList >>= JSON.parseMaybe go
202+ where
203+ singletonList :: JSON.Value -> Maybe JSON.Value
204+ singletonList =
205+ \case
206+ JSON.Array [single] -> Just single
207+ _ -> Nothing
208+ go :: JSON.Value -> JSON.Parser Nix.DrvPath
209+ go json = flip (JSON.withObject "app") json $ \o -> do
210+ textDrvPath :: [Text] <- o .: "context"
211+ path <- case textDrvPath of
212+ [singlePath] -> pure singlePath
213+ _ -> JSON.parseFail "AppPackage: expecting single context derivation"
214+ case Nix.parseStorePath path of
215+ Left err -> JSON.parseFail $ cs err
216+ Right path -> pure $ Nix.DrvPath path
backend/src/Garnix/Build/Flake.hsnew file+173
@@ -0,0 +1,173 @@
1+module Garnix.Build.Flake
2+ ( runBuildFlake,
3+ )
4+where
5+
6+import Control.Lens
7+import Garnix.Attribute
8+import Garnix.Build.Action qualified as Action
9+import Garnix.Build.Checkout (Remote, runWithCheckout, withAuthorization)
10+import Garnix.Build.FodCheck qualified as FodCheck
11+import Garnix.Build.Helpers
12+import Garnix.Build.MetaCheck qualified as MetaCheck
13+import Garnix.Build.Package (doBuild)
14+import Garnix.Build.Reporting
15+import Garnix.DB qualified as DB
16+import Garnix.Entitlements (addDefaultEntitlements, getPlan, hasRemainingCiTime)
17+import Garnix.GetAttributes
18+import Garnix.Hosting.Deploy (rolloutNewServerVersion)
19+import Garnix.Modules qualified as Modules
20+import Garnix.Monad
21+import Garnix.Monad.Async (joinAll, joinAll_, resolve, spawn)
22+import Garnix.Prelude
23+import Garnix.Types as Types
24+import Garnix.YamlConfig (Action, ExcludeBranches (..), GarnixConfig, IncrementalizeBuildsSection (..), flakeDir, incrementalizeBuildsSection)
25+
26+runBuildFlake :: (HasCallStack) => Reporter -> BuildKind -> CommitInfo -> Remote -> M ()
27+runBuildFlake reporter buildKind commitInfo withCheckout = do
28+ let repoOwner = commitInfo ^. repoInfo . ghRepoOwner
29+ addDefaultEntitlements repoOwner
30+ (startingBuild, startingBuildRunReporter) <- newBuild reporter commitInfo (PackageInfo TypeOverall NoSystem buildStarting) False
31+ withInternalCacheToken (commitInfo ^. reqUser) $ do
32+ metaCheckRun <- MetaCheck.newReport reporter commitInfo
33+ flip catchEither (\err -> MetaCheck.updateFail commitInfo metaCheckRun (Just err) >> rethrowEither err) $ do
34+ reportOnError startingBuildRunReporter startingBuild commitInfo $ do
35+ hasCiTime <- hasRemainingCiTime repoOwner
36+ when (not hasCiTime) $ do
37+ log Notice $ show (commitInfo ^. repoInfo . ghRepoOwner) <> " ran out of CI time."
38+ throw $ EntitlementError "You have exhausted your monthly CI quota"
39+ repoConfig <- DB.getRepoConfig (commitInfo ^. repoInfo . ghRepoOwner) (commitInfo ^. repoInfo . ghRepoName)
40+ runWithCheckout withCheckout commitInfo $ \config -> do
41+ withAuthorization (config ^. flakeDir) repoConfig commitInfo $ do
42+ plan <- getPlan repoOwner
43+ initialBuilds <- setupBuilds reporter commitInfo config plan
44+ initialActions <- setupActions reporter commitInfo config
45+ updatedBuild <-
46+ liftIO getCurrentTime <&> \now ->
47+ startingBuild
48+ & status ?~ Success
49+ & endTime ?~ now
50+ DB.setCommitStatus (commitInfo ^. repoInfo . ghRepoOwner) (commitInfo ^. repoInfo . ghRepoName) (commitInfo ^. commit) Evaluated
51+ reportBuildResult startingBuildRunReporter updatedBuild
52+
53+ FodCheck.withFodChecker reporter commitInfo plan $ \fodChecker -> do
54+ buildPromises <- forM initialBuilds $ \(initialBuild, runReporter) -> do
55+ spawn $ doBuild fodChecker runReporter buildKind (config ^. flakeDir) repoConfig plan initialBuild
56+ actionPromises <- forM initialActions $ \(initialBuild, runReporter, actionConfig) -> do
57+ spawn
58+ $ buildAndRunAction
59+ reporter
60+ fodChecker
61+ runReporter
62+ commitInfo
63+ buildKind
64+ plan
65+ (config ^. flakeDir)
66+ repoConfig
67+ initialBuild
68+ actionConfig
69+ builds <- joinAll buildPromises >>= resolve
70+ joinAll_ actionPromises >>= resolve
71+
72+ let allBuildsSucceeded = all (\build -> build ^. status == Just Success) builds
73+
74+ when allBuildsSucceeded $ do
75+ Modules.publish reporter config commitInfo
76+
77+ deployments <- case (commitInfo ^. prFromFork, commitInfo ^. branch) of
78+ (Nothing, Nothing) -> do
79+ log Critical "Both branch and fork info are missing. Expected exactly one to be present."
80+ pure []
81+ (Just _, Just _) -> do
82+ log Critical "Both branch and fork info are present. Expected exactly one to be present."
83+ pure []
84+ (Nothing, Just (branch :: Branch)) -> do
85+ if allBuildsSucceeded
86+ then do
87+ rolloutNewServerVersion reporter commitInfo (BranchDeployment branch)
88+ else pure []
89+ (Just (_ :: PrFromFork), Nothing) -> do
90+ log Notice "PR is from fork. Not deploying servers"
91+ pure []
92+
93+ let allDeploymentsSucceeded =
94+ all
95+ ( \serverInfo ->
96+ isJust (serverInfo ^. readyAt)
97+ && isNothing (serverInfo ^. endedAt)
98+ )
99+ deployments
100+ if allBuildsSucceeded && allDeploymentsSucceeded
101+ then MetaCheck.updateSuccess commitInfo metaCheckRun
102+ else MetaCheck.updateFail commitInfo metaCheckRun Nothing
103+
104+setupBuilds :: Reporter -> CommitInfo -> GarnixConfig -> ProductPlan -> M [(Build, RunReporter)]
105+setupBuilds reporter commitInfo config plan = do
106+ toBuild <- do
107+ attributes <- getAttributesToBuild commitInfo config
108+ when (length attributes > fromIntegral (plan ^. maximumPackagesPerFlake)) $ do
109+ throw
110+ $ OtherError
111+ $ "Number of packages too large. Maximum is "
112+ <> show (plan ^. maximumPackagesPerFlake)
113+ <> ", you have "
114+ <> show (length attributes)
115+ pure attributes
116+ log Informational $ "Will build the following attributes: " <> show toBuild
117+ forM toBuild $ \attr -> do
118+ setupBuild reporter config commitInfo attr
119+
120+setupActions :: Reporter -> CommitInfo -> GarnixConfig -> M [(Build, RunReporter, Action)]
121+setupActions reporter commitInfo config = do
122+ log Informational $ "Will run the following actions: " <> show (Action.getActionAppAttributes config)
123+ forM (Action.getActionAppAttributes config) $ \(attr, actionConfig) -> do
124+ (build, reporter) <- setupBuild reporter config commitInfo attr
125+ return (build, reporter, actionConfig)
126+
127+buildAndRunAction ::
128+ Reporter ->
129+ Maybe FodChecker ->
130+ RunReporter ->
131+ CommitInfo ->
132+ BuildKind ->
133+ ProductPlan ->
134+ FlakeDir ->
135+ RepoConfig ->
136+ Build ->
137+ Action ->
138+ M ()
139+buildAndRunAction reporter fodChecker runReporter commitInfo buildKind plan flakeDir repoConfig initialBuild actionConfig = do
140+ build <- doBuild fodChecker runReporter buildKind flakeDir repoConfig plan initialBuild
141+ Action.run flakeDir repoConfig reporter commitInfo (attribute build) actionConfig build
142+
143+newBuild :: Reporter -> CommitInfo -> PackageInfo -> Bool -> M (Build, RunReporter)
144+newBuild reporter commitInfo packageInfo wantsIncrementalism = withSpan packageInfo $ do
145+ hostname <- view #hostname
146+ initialBuild <-
147+ DB.newBuildDB commitInfo packageInfo hostname wantsIncrementalism
148+ <?> "Creating a build in the DB"
149+ withSpan (initialBuild ^. id) $ do
150+ runReporter <- createNewRun reporter $ ReportBuild (reportNameForBuild initialBuild) initialBuild
151+ log Informational $ "My GH run id is: " <> show (Garnix.Monad.ghRunId runReporter)
152+ let build = initialBuild & githubRunId .~ Garnix.Monad.ghRunId runReporter
153+ DB.reportBuildResultDB build <?> "Adding build github ID to DB"
154+ pure (build, runReporter)
155+
156+setupBuild :: Reporter -> GarnixConfig -> CommitInfo -> Attribute -> M (Build, RunReporter)
157+setupBuild reporter config commitInfo attr = case attr ^. packageName of
158+ Nothing -> do
159+ throw $ OtherError "Tried to build, but no package name available"
160+ Just pkgName -> do
161+ let wantsIncrementalism' = case config ^. incrementalizeBuildsSection of
162+ IncrementalizeBuilds True -> True
163+ IncrementalizeBuilds False -> False
164+ IncrementalBuildsExcludeBranches (ExcludeBranches brs) -> case commitInfo ^. branch of
165+ Nothing -> False
166+ Just br -> br `notElem` brs
167+ (build, runReporter) <-
168+ newBuild
169+ reporter
170+ commitInfo
171+ (PackageInfo (attr ^. packageType) (attr ^. system . from maybeSystemIso) pkgName)
172+ wantsIncrementalism'
173+ pure (build, runReporter)
backend/src/Garnix/Build/FodCheck.hsnew file+301
@@ -0,0 +1,301 @@
1+module Garnix.Build.FodCheck
2+ ( withFodChecker,
3+ fodCheck,
4+ -- exported for testing:
5+ __findAllFodsRecursively,
6+ __pickRemoteBuilderUrlFromMachinesFile,
7+ __parseMachinesFile,
8+ )
9+where
10+
11+import Control.Concurrent.Lifted (modifyMVar, modifyMVar_, newMVar, readMVar, swapMVar)
12+import Control.Lens ((<&>))
13+import Control.Monad.Extra (mapMaybeM)
14+import Cradle
15+import Data.Attoparsec.Text hiding ((<?>))
16+import Data.Either.Extra (mapLeft)
17+import Data.Map.Strict (Map)
18+import Data.Map.Strict qualified as Map
19+import Data.Maybe
20+import Data.Row (Rec, type (.+), type (.==))
21+import Data.Set qualified as Set
22+import Data.Text.IO qualified as T
23+import Garnix.BuildLogs.Types (LogLine (LogLine), mkLogLine)
24+import Garnix.DB qualified as DB
25+import Garnix.DB.FeatureFlags qualified as FeatureFlags
26+import Garnix.DB.FeatureFlags.Types qualified as FeatureFlags
27+import Garnix.Monad
28+import Garnix.Monad.Async (joinAll, resolve, spawn)
29+import Garnix.Monad.Bubbling
30+import Garnix.Monad.Metrics (timingAs)
31+import Garnix.Monad.Pool (withPoolM)
32+import Garnix.Monad.SubProcess (runSubProcess)
33+import Garnix.Nix.Types qualified as Nix
34+import Garnix.NixConfig (addNixConfigEnvironment)
35+import Garnix.Prelude hiding (Alternative)
36+import Garnix.Types
37+import Garnix.YamlConfig qualified as YamlConfig
38+import System.Metrics.Prometheus.Metric.Histogram qualified as Prometheus
39+
40+withFodChecker :: Reporter -> CommitInfo -> ProductPlan -> (Maybe FodChecker -> M a) -> M a
41+withFodChecker reporter commitInfo plan action = do
42+ fodChecker <- getFodChecker reporter commitInfo plan
43+ let reportToSummary = case fodChecker of
44+ Just fodChecker -> \logLine -> reportLogs (fodChecker ^. #runReporter) $ LogLine (Just $ PackageName "FOD Summary") Nothing logLine
45+ Nothing -> const $ pure ()
46+ reportToSummary "Checking fixed output derivations..."
47+ a <- action fodChecker
48+ case fodChecker of
49+ Nothing -> pure ()
50+ Just fodChecker -> do
51+ promises <-
52+ swapMVar (fodChecker ^. #promises) Nothing
53+ <&> fromMaybe (error "impossible: should still be Just")
54+ errors <- mapLeft join . collectErrors <$> (resolve =<< joinAll promises)
55+ skipped <- readMVar $ fodChecker ^. #totalSkipped
56+ verified <- readMVar $ fodChecker ^. #totalVerified
57+ case skipped of
58+ 0 -> pure ()
59+ 1 -> reportToSummary "1 FOD was skipped because it was verified in previous garnix builds."
60+ skipped -> reportToSummary $ show skipped <> " FODs were skipped because they were verified in previous garnix builds."
61+ case verified of
62+ 0 -> pure ()
63+ 1 -> reportToSummary "1 FOD was verified."
64+ verified -> reportToSummary $ show verified <> " FODs were verified."
65+ case errors of
66+ Right () -> reportComplete (fodChecker ^. #runReporter) RunReportStatusSuccess
67+ Left (errors :: [(Nix.DrvPath, Text)]) -> do
68+ reportToSummary $ show (length errors) <> " FOD checks failed."
69+ reportComplete (fodChecker ^. #runReporter) RunReportStatusFailure
70+ pure a
71+
72+getFodChecker :: Reporter -> CommitInfo -> ProductPlan -> M (Maybe FodChecker)
73+getFodChecker reporter commitInfo plan = do
74+ garnixConfig <- YamlConfig.getConfig
75+ if garnixConfig ^. YamlConfig.fodChecks
76+ then do
77+ run <- DB.newRun "FOD checks" commitInfo
78+ runReporter <- createNewRun reporter (ReportRun run)
79+ if plan ^. isPaid || (commitInfo ^. repoInfo . ghRepoOwner == "garnix-io")
80+ then do
81+ Just <$> mkFodChecker runReporter
82+ else do
83+ reportLogs runReporter $ mkLogLine "`fodChecks` are enabled in your garnix.yaml, but that feature requires a paid plan."
84+ reportComplete runReporter RunReportStatusFailure
85+ pure Nothing
86+ else do
87+ randomlyEnabled <- FeatureFlags.isFeatureOn FeatureFlags.FodChecks
88+ if randomlyEnabled
89+ then Just <$> mkFodChecker mempty
90+ else pure Nothing
91+ where
92+ mkFodChecker :: RunReporter -> M FodChecker
93+ mkFodChecker runReporter =
94+ FodChecker runReporter
95+ <$> newMVar 0
96+ <*> newMVar 0
97+ <*> newMVar (Just [])
98+ <*> newMVar mempty
99+
100+fodCheck :: Maybe FodChecker -> Nix.DrvPath -> M ()
101+fodCheck = curry $ mockable #fodCheckMock $ \(fodChecker, drvPath) -> do
102+ withTextSpan ("fod_check_package", cs drvPath) $ do
103+ case fodChecker of
104+ Nothing -> pure ()
105+ Just fodChecker -> do
106+ modifyMVar_ (fodChecker ^. #promises) $ \case
107+ Nothing -> do
108+ log Critical "withFodChecker already exited"
109+ pure Nothing
110+ Just promises -> do
111+ new <- spawn (runFodCheck fodChecker drvPath)
112+ pure $ Just (new : promises)
113+
114+runFodCheck :: FodChecker -> Nix.DrvPath -> M (Either [(Nix.DrvPath, Text)] ())
115+runFodCheck fodChecker drvPath = do
116+ withMessage "running fodCheck" $ do
117+ timingAs #fodCheckTime $ do
118+ logDuration "FOD check" $ do
119+ allFods <- __findAllFodsRecursively drvPath
120+ notStartedOrDone <- getNotStartedOrDone fodChecker allFods
121+ fods <- DB.keepUnverifiedFods notStartedOrDone
122+ modifyMVar_ (fodChecker ^. #totalSkipped) $ pure . (+ (length notStartedOrDone - length fods))
123+ (liftIO . Prometheus.observe (fromIntegral $ length fods))
124+ =<< view (#metrics . #fodCheckBatchSize)
125+ errors <- forConcurrently (toList fods) $ \(drvPath, system) -> do
126+ withPoolM fodCheckPool () $ do
127+ mapLeft (drvPath,) <$> checkFod fodChecker drvPath system
128+ pure $ collectErrors errors
129+
130+getNotStartedOrDone :: FodChecker -> Set (Nix.DrvPath, System) -> M (Set (Nix.DrvPath, System))
131+getNotStartedOrDone fodChecker set = do
132+ modifyMVar (fodChecker ^. #startedOrDone) $ \current -> do
133+ pure
134+ ( current <> Set.map fst set,
135+ set & Set.filter (\(drvPath, _) -> drvPath `Set.notMember` current)
136+ )
137+
138+collectErrors :: [Either e ()] -> Either [e] ()
139+collectErrors eithers = case catMaybes $ map (either Just (const Nothing)) eithers of
140+ [] -> Right ()
141+ errors -> Left errors
142+
143+__findAllFodsRecursively :: Nix.DrvPath -> M (Set (Nix.DrvPath, System))
144+__findAllFodsRecursively drvPath = do
145+ nixConfig <- view #userNixConfig
146+ StdoutRaw stdout <-
147+ runSubProcess
148+ $ cmd "nix"
149+ & addArgs ["derivation", "show", "--recursive", cs drvPath :: Text]
150+ & addNixConfigEnvironment nixConfig
151+ parsed :: NixDerivationShowJson <- aesonDecode "nix derivation show output" parseJSON (cs stdout)
152+ (Set.fromList <$>) $ flip mapMaybeM (Map.toList $ parsed ^. #derivations) $ \(drvPathText, info) -> do
153+ case (info ^. #env . #outputHash, info ^. #env . #system) of
154+ (Nothing, _) -> pure Nothing
155+ (Just _, Nothing) ->
156+ throw
157+ $ DecodeError
158+ { original = cs stdout,
159+ message = "fod doesn't have an `env.system` field."
160+ }
161+ (Just _fodDerivation, Just system) -> do
162+ case Nix.parseDrvPath ("/nix/store/" <> drvPathText) of
163+ Left err -> throw $ DecodeError {original = cs stdout, message = "Failed to parse drv path: " <> err}
164+ Right drvPath -> pure $ Just (drvPath, system ^. re systemTextIso)
165+
166+type NixDerivationShowJson =
167+ ( Rec
168+ ( "derivations"
169+ .== Map.Map
170+ Text
171+ ( Rec
172+ ( "env"
173+ .== NixDerivationShowEnvJson
174+ )
175+ )
176+ )
177+ )
178+
179+data NixDerivationShowEnvJson = NixDerivationShowEnvJson
180+ { outputHash :: Maybe Text,
181+ system :: Maybe Text
182+ }
183+ deriving (Generic, Show)
184+
185+instance FromJSON NixDerivationShowEnvJson
186+
187+checkFod :: FodChecker -> Nix.DrvPath -> System -> M (Either Text ())
188+checkFod fodChecker drvPath system = do
189+ log Informational $ "checking fod: " <> cs drvPath <> ", system: " <> system ^. systemTextIso
190+ result <- rebuildFod system drvPath
191+ case result of
192+ Left stderr -> do
193+ let errorMessage =
194+ "Failure when checking FOD '"
195+ <> cs drvPath
196+ <> "':\n"
197+ <> stderr
198+ log Error errorMessage
199+ reportLogs (fodChecker ^. #runReporter) $ LogLine (Just $ PackageName $ cs drvPath <> " (failed)") Nothing errorMessage
200+ pure $ Left errorMessage
201+ Right stdout -> do
202+ modifyMVar_ (fodChecker ^. #totalVerified) $ pure . (+ 1)
203+ storePath :: Nix.StorePath <- do
204+ parsed <- aesonDecode "nix build output" (parseJSON @NixBuildOutput) stdout
205+ case parsed of
206+ [derivation] -> case Map.elems (derivation ^. #outputs) of
207+ [output] -> case Nix.parseStorePath output of
208+ Right storePath -> pure storePath
209+ Left error -> throw $ DecodeError {original = stdout, message = error}
210+ _ -> throw $ OtherError "impossible: fods cannot have multiple outputs"
211+ _ -> throw $ OtherError "impossible: more than one derivation returned"
212+ DB.addVerifiedFod drvPath storePath
213+ pure $ Right ()
214+
215+type NixBuildOutput = [Rec ("drvPath" .== Text .+ "outputs" .== Map.Map Text Text)]
216+
217+-- Untested.
218+rebuildFod :: System -> Nix.DrvPath -> M (Either Text Text)
219+rebuildFod = do
220+ curry $ mockable #rebuildFodMock $ \(system, drvPath) -> withBubbling $ \bubble -> do
221+ contents <- liftIO $ T.readFile "/etc/nix/machines"
222+ remoteBuilderUrl <- __pickRemoteBuilderUrlFromMachinesFile system contents
223+ (bubble =<< copyClosure drvPath remoteBuilderUrl)
224+ <?> ("rebuildFod: copying closure of " <> cs drvPath <> " to " <> remoteBuilderUrl)
225+ (exitCode, StdoutRaw (cs -> stdout), StderrRaw (cs -> stderr)) <- do
226+ nixConfig <- view #userNixConfig
227+ run
228+ $ cmd "nix"
229+ & addArgs
230+ [ "build",
231+ cs drvPath <> "^*",
232+ "--no-link",
233+ "--json",
234+ "--rebuild",
235+ "--store",
236+ remoteBuilderUrl,
237+ "--eval-store",
238+ "auto"
239+ ]
240+ & modifyEnvVar "NIX_SSHOPTS" (const $ Just $ unwords remoteBuilderSshArgs)
241+ & addNixConfigEnvironment nixConfig
242+ bubble $ case exitCode of
243+ ExitFailure _ -> Left stderr
244+ ExitSuccess -> Right stdout
245+
246+-- Untested.
247+copyClosure :: Nix.DrvPath -> Text -> M (Either Text ())
248+copyClosure drvPath remoteBuilderUrl = do
249+ nixConfig <- view #userNixConfig
250+ (exitCode, StderrRaw (cs -> stderr)) <-
251+ Cradle.run
252+ $ cmd "nix"
253+ & Cradle.addArgs
254+ [ "copy",
255+ "--no-check-sigs",
256+ "--to",
257+ remoteBuilderUrl,
258+ cs drvPath <> "^*"
259+ ]
260+ & addNixConfigEnvironment nixConfig
261+ & Cradle.modifyEnvVar "NIX_SSHOPTS" (const $ Just $ unwords remoteBuilderSshArgs)
262+ & Cradle.silenceStderr
263+ case exitCode of
264+ ExitFailure _ -> pure $ Left stderr
265+ ExitSuccess -> pure $ Right ()
266+
267+remoteBuilderSshArgs :: [String]
268+remoteBuilderSshArgs = ["-i", "/run/secrets/garnix_server_remote_builder_ssh_garnix"]
269+
270+__pickRemoteBuilderUrlFromMachinesFile :: System -> Text -> M Text
271+__pickRemoteBuilderUrlFromMachinesFile (replaceBuiltinSystem -> system) machinesFile =
272+ case __parseMachinesFile machinesFile of
273+ Left error -> throw $ OtherError $ "cannot parse /etc/nix/machines: " <> error
274+ Right machines -> do
275+ machinesForSystem <- case Map.lookup system machines of
276+ Nothing -> throw $ OtherError $ "Cannot find remote builder for " <> system ^. systemTextIso
277+ Just ms -> pure ms
278+ randomElement machinesForSystem
279+
280+replaceBuiltinSystem :: System -> System
281+replaceBuiltinSystem = \case
282+ OtherSystem "builtin" -> X8664Linux
283+ other -> other
284+
285+__parseMachinesFile :: Text -> Either Text (Map.Map System [Text])
286+__parseMachinesFile machinesFile =
287+ mapLeft cs $ parseOnly machines machinesFile
288+ where
289+ machines :: Parser (Map System [Text])
290+ machines = do
291+ _ <- skipSpace
292+ lines <- sepBy line (many1 endOfLine) <* skipSpace <* endOfInput
293+ pure $ Map.fromListWith (<>) $ join $ map (\(builder, systems) -> map (,[builder]) systems) lines
294+
295+ line :: Parser (Text, [System])
296+ line = do
297+ _ <- many (char ' ')
298+ machine <- cs <$> manyTill anyChar (many1 (char ' '))
299+ systems <- sepBy (cs <$> many (satisfy (`notElem` [' ', ',']))) (char ',') <* char ' '
300+ _ <- many (satisfy (/= '\n'))
301+ pure (machine, fmap (^. re systemTextIso) systems)
backend/src/Garnix/Build/Helpers.hsnew file+36
@@ -0,0 +1,36 @@
1+module Garnix.Build.Helpers
2+ ( withPrivateNixXdgCache,
3+ withInternalCacheToken,
4+ )
5+where
6+
7+import Garnix.DB qualified as DB
8+import Garnix.Monad
9+import Garnix.Monad.ForkT (safeSystemTempDirectory, safeSystemTempFile)
10+import Garnix.NixConfig qualified as NixConfig
11+import Garnix.Prelude
12+import Garnix.Types as Types
13+import System.IO (hClose, hPutStrLn)
14+
15+-- We need to make sure each build has its own cache in order
16+-- to avoid leaking private repositories between organisations.
17+withPrivateNixXdgCache :: M a -> M a
18+withPrivateNixXdgCache action = do
19+ tempDir <- safeSystemTempDirectory "garnix-cache"
20+ local (#nixXdgCacheDir ?~ tempDir) $ do
21+ action <?> "running action with private nix xdg cache"
22+
23+withInternalCacheToken :: GhLogin -> M a -> M a
24+withInternalCacheToken reqUser cont = do
25+ token <- DB.getUserInternalToken reqUser
26+ (path, handle) <- safeSystemTempFile "garnix-netrc"
27+ liftIO $ do
28+ hPutStrLn handle
29+ . unlines
30+ $ [ "machine cache.garnix.io",
31+ "login " <> cs (getGhLogin reqUser),
32+ "password " <> cs (getInternalCacheToken token)
33+ ]
34+ hClose handle
35+ local (#userNixConfig %~ ((NixConfig.fromNetRcFile . NetRcFile $ path) <>)) $ do
36+ withTextSpan ("internal_token", show reqUser) cont
backend/src/Garnix/Build/MetaCheck.hsnew file+98
@@ -0,0 +1,98 @@
1+module Garnix.Build.MetaCheck
2+ ( update,
3+ newReport,
4+ updateFail,
5+ updateSuccess,
6+ )
7+where
8+
9+import Garnix.DB qualified as DB
10+import Garnix.Monad
11+import Garnix.Prelude
12+import Garnix.Types as Types
13+
14+update :: Reporter -> CommitInfo -> M ()
15+update reporter commitInfo = do
16+ DB.getBuildsAndRunsByCommit (commitInfo ^. repoInfo . ghRepoOwner) (commitInfo ^. repoInfo . ghRepoName) (commitInfo ^. commit)
17+ >>= \case
18+ CommitEvaluating -> pure ()
19+ CommitEvaluated commitState builds _ ->
20+ setCheckTo
21+ commitState
22+ (foldl' mergeMetaCheckStatus CheckSuccess (map (^. status . to mapToMetaCheckStatus) builds))
23+ where
24+ mapToMetaCheckStatus :: Maybe Status -> CheckStatus
25+ mapToMetaCheckStatus = \case
26+ Nothing -> CheckPending
27+ Just Success -> CheckSuccess
28+ Just Failure -> CheckFail
29+ Just Timeout -> CheckFail
30+ Just Cancelled -> CheckFail
31+
32+ mergeMetaCheckStatus :: CheckStatus -> CheckStatus -> CheckStatus
33+ mergeMetaCheckStatus a b = case (a, b) of
34+ (CheckFail, _) -> CheckFail
35+ (_, CheckFail) -> CheckFail
36+ (CheckPending, _) -> CheckPending
37+ (_, CheckPending) -> CheckPending
38+ (CheckSuccess, CheckSuccess) -> CheckSuccess
39+
40+ setCheckTo :: Commit -> CheckStatus -> M ()
41+ setCheckTo commit' newStatus = do
42+ updatedCheck <-
43+ DB.setMetaCheck
44+ (commitInfo ^. repoInfo . ghRepoOwner)
45+ (commitInfo ^. repoInfo . ghRepoName)
46+ (commitInfo ^. commit)
47+ ( DB.CheckStatusUpdate
48+ { _checkStatusUpdateFrom = commit' ^. metaCheck,
49+ _checkStatusUpdateTo = newStatus
50+ }
51+ )
52+ when updatedCheck $ do
53+ let st = case newStatus of
54+ CheckPending -> Nothing
55+ CheckFail -> Just RunReportStatusFailure
56+ CheckSuccess -> Just RunReportStatusSuccess
57+ case st of
58+ Just status -> do
59+ runReporter <- createNewRun reporter MetaCheck
60+ reportComplete runReporter status
61+ Nothing -> pure ()
62+
63+newReport :: Reporter -> CommitInfo -> M RunReporter
64+newReport reporter commitInfo = do
65+ void $ DB.newCommit (commitInfo ^. repoInfo . ghRepoOwner) (commitInfo ^. repoInfo . ghRepoName) (commitInfo ^. commit)
66+ createNewRun reporter MetaCheck
67+
68+updateFail :: CommitInfo -> RunReporter -> Maybe (Either SomeException ErrorWithContext) -> M ()
69+updateFail commitInfo runReporter e = do
70+ case e of
71+ Just err ->
72+ log Warning $ "failMetaCheck: uncaught " <> either (const "IO exception: ") (const "monadic error: ") err <> either show show err
73+ Nothing -> pure ()
74+ updated <-
75+ DB.setMetaCheck
76+ (commitInfo ^. repoInfo . ghRepoOwner)
77+ (commitInfo ^. repoInfo . ghRepoName)
78+ (commitInfo ^. commit)
79+ ( DB.CheckStatusUpdate
80+ { _checkStatusUpdateFrom = CheckPending,
81+ _checkStatusUpdateTo = CheckFail
82+ }
83+ )
84+ when updated $ reportComplete runReporter RunReportStatusFailure
85+
86+updateSuccess :: CommitInfo -> RunReporter -> M ()
87+updateSuccess commitInfo runReporter = do
88+ updated <-
89+ DB.setMetaCheck
90+ (commitInfo ^. repoInfo . ghRepoOwner)
91+ (commitInfo ^. repoInfo . ghRepoName)
92+ (commitInfo ^. commit)
93+ ( DB.CheckStatusUpdate
94+ { _checkStatusUpdateFrom = CheckPending,
95+ _checkStatusUpdateTo = CheckSuccess
96+ }
97+ )
98+ when updated $ reportComplete runReporter RunReportStatusSuccess
backend/src/Garnix/Build/Module.hsnew file+191
@@ -0,0 +1,191 @@
1+module Garnix.Build.Module
2+ ( generateFlakeNix,
3+ getCommitInfo,
4+ remoteWithFlake,
5+ -- exported for testing:
6+ _moduleConfig,
7+ )
8+where
9+
10+import Control.Lens
11+import Cradle qualified
12+import Data.Map.Strict (Map)
13+import Data.Map.Strict qualified as Map
14+import Data.String.Interpolate (i)
15+import Data.String.Interpolate.Util (unindent)
16+import Data.Text qualified as T
17+import Data.Text.IO qualified as Text
18+import Garnix.Build.Checkout qualified as Checkout
19+import Garnix.DB.ModuleValues qualified as ModuleValues
20+import Garnix.Monad
21+import Garnix.Monad.SubProcess qualified as SubProcess
22+import Garnix.NixConfig qualified as NixConfig
23+import Garnix.Prelude
24+import Garnix.Sandbox
25+import Garnix.Types (Branch (..), CommitInfo (..), Error (..), GhLogin, RepoInfo (..), getCommitHash, getGhLogin, getGhRepoName, getGhRepoOwner)
26+import GitHub.Data.Id (Id (Id))
27+
28+getCommitInfo :: GhLogin -> ModuleValues.GetRepoAndModuleValues -> M CommitInfo
29+getCommitInfo reqUser modules = do
30+ case (modules ^. #repo_user, modules ^. #repo_name) of
31+ (Just user, Just repo) -> do
32+ installationId <- getGarnixInstallationId user repo
33+ iAuth <- case installationId of
34+ Nothing -> throw $ NoSuchRepo {_owner = user, _name = repo}
35+ Just id -> getInstallation (Id $ fromInteger id)
36+ repoPublicity <- getRepoPublicity iAuth user repo
37+ getDefaultBranch (Just iAuth) user repo >>= \case
38+ Nothing -> throw $ NoSuchRepo {_owner = user, _name = repo}
39+ Just branch -> do
40+ token <- getAccessToken iAuth
41+ commit <- getHeadCommit token user repo branch
42+ pure
43+ $ CommitInfo
44+ { _commitInfoReqUser = reqUser,
45+ _commitInfoRepoPublicity = repoPublicity,
46+ _commitInfoRepoInfo = RepoInfo iAuth token user repo,
47+ _commitInfoBranch = Just branch,
48+ _commitInfoPrFromFork = Nothing,
49+ _commitInfoCommit = commit
50+ }
51+ _ -> throw $ OtherError "cannot run build without a repository"
52+
53+remoteWithFlake :: Branch -> ModuleValues.GetRepoAndModuleValues -> Checkout.Remote -> Checkout.Remote
54+remoteWithFlake branch values = Checkout.withBeforeAction $ do
55+ flakeContents <- generateFlakeNix branch values
56+ dir <- view #workingDir
57+ liftIO $ Text.writeFile (dir </> "flake.nix") flakeContents
58+ SubProcess.runGitProcess ["add", "flake.nix"]
59+ generateNixFlakeLockFile
60+ SubProcess.runGitProcess ["add", "flake.lock"]
61+ where
62+ generateNixFlakeLockFile :: M ()
63+ generateNixFlakeLockFile = do
64+ let modules = values ^. #modules
65+ let garnixLibInput = ("garnix-lib", "github:garnix-io/garnix-lib?ref=d3f3a98a0baddb3bdc6e0d028d1b58251a1d86f5")
66+ let inputs =
67+ garnixLibInput
68+ : ( ( \m ->
69+ ( m ^. #name,
70+ "github:"
71+ <> getGhLogin (getGhRepoOwner $ m ^. #repo_user)
72+ <> "/"
73+ <> getGhRepoName (m ^. #repo_name)
74+ <> "?ref="
75+ <> getCommitHash (m ^. #git_commit)
76+ )
77+ )
78+ <$> modules
79+ )
80+ args = (\(input, hash) -> ["--override-input", input, hash]) <$> inputs
81+ runNixCommand $ ["flake", "lock"] <> join args
82+
83+ runNixCommand :: [Text] -> M ()
84+ runNixCommand args = do
85+ cacheDir <- getNixXdgCacheDir
86+ nixConfig <- view #userNixConfig
87+ dir <- view #workingDir
88+ result <-
89+ (>>= Cradle.run)
90+ $ Cradle.cmd "nix"
91+ & Cradle.addArgs args
92+ & NixConfig.addNixConfigEnvironment nixConfig
93+ & Cradle.setWorkingDir dir
94+ & Cradle.silenceStderr
95+ & pure
96+ & inNixSandbox [] (Just cacheDir)
97+ case result of
98+ (Cradle.ExitFailure code, Cradle.StdoutRaw out, Cradle.StderrRaw err) -> do
99+ log Warning
100+ $ "runNixCommand error "
101+ <> T.unwords args
102+ <> ": stdout ("
103+ <> cs out
104+ <> ") stderr ("
105+ <> cs err
106+ <> ")"
107+ throw RunProcessError {command = "nix", arguments = args, stdErr = cs err, stdOut = cs out, exitCode = code}
108+ _ -> pure ()
109+
110+generateFlakeNix :: Branch -> ModuleValues.GetRepoAndModuleValues -> M Text
111+generateFlakeNix branch values = do
112+ let modules = Map.fromList . map (\m -> (m ^. #name, m)) $ values ^. #modules
113+ moduleConfigs <- mapM (getModuleConfig modules) (values ^. #user_config)
114+ toFlakeFile branch moduleConfigs
115+ where
116+ getModuleConfig :: Map Text ModuleValues.Module -> ModuleValues.ModuleValue -> M (ModuleValues.Module, ModuleValues.ModuleConfig)
117+ getModuleConfig modules value = case Map.lookup (value ^. #module_name) modules of
118+ Just m -> pure (m, value ^. #values)
119+ Nothing -> throw $ OtherError $ "module not found: " <> value ^. #module_name
120+
121+toFlakeFile :: Branch -> [(ModuleValues.Module, ModuleValues.ModuleConfig)] -> M Text
122+toFlakeFile branch modulesAndValues = do
123+ pure
124+ $ cs
125+ $ unindent
126+ [i|
127+{
128+ inputs = {
129+ garnix-lib.url = "github:garnix-io/garnix-lib";#{foldMap extraInputs (fst <$> modulesAndValues)}
130+ };
131+
132+ nixConfig = {
133+ extra-substituters = [ "https://cache.garnix.io" ];
134+ extra-trusted-public-keys = [ "cache.garnix.io:CTFPyKSLcx5RMJKfLo5EEPUObbA78b0YQ2DTCJXqr9g=" ];
135+ };
136+
137+ outputs = inputs: inputs.garnix-lib.lib.mkModules {
138+ modules = [#{foldMap moduleList (fst <$> modulesAndValues)}
139+ ];
140+
141+ config = { pkgs, ... }: {
142+#{foldMap _moduleConfig (snd <$> modulesAndValues)}
143+ garnix.deployBranch = "#{getBranch branch}";
144+ };
145+ };
146+}
147+ |]
148+ where
149+ extraInputs :: ModuleValues.Module -> Text
150+ extraInputs m =
151+ let moduleName = m ^. #name
152+ repoUser = getGhLogin . getGhRepoOwner $ m ^. #repo_user
153+ repoName = getGhRepoName $ m ^. #repo_name
154+ in cs
155+ [i|
156+ #{moduleName}.url = "github:#{repoUser}/#{repoName}";|]
157+
158+ moduleList :: ModuleValues.Module -> Text
159+ moduleList m =
160+ let moduleName = m ^. #name
161+ in cs
162+ [i|
163+ inputs.#{moduleName}.garnixModules.default|]
164+
165+_moduleConfig :: ModuleValues.ModuleConfig -> Text
166+_moduleConfig (ModuleValues.ModuleConfig identifier value) = mkAttr 3 identifier value
167+ where
168+ go :: Int -> ModuleValues.NixValue -> Text
169+ go indent = \case
170+ ModuleValues.Secret secret -> escapeNixStr $ secret ^. #encryptedValue
171+ ModuleValues.NixString str -> escapeNixStr str
172+ ModuleValues.NixPath path -> path
173+ ModuleValues.NixRaw raw -> raw
174+ ModuleValues.NixBool b -> if b then "true" else "false"
175+ ModuleValues.NixInt n -> show n
176+ ModuleValues.NixNull -> "null"
177+ ModuleValues.NixList l -> "[ " <> T.intercalate " " (go indent <$> l) <> " ]"
178+ ModuleValues.NixSet attrs ->
179+ "{\n"
180+ <> Map.foldMapWithKey (mkAttr (succ indent)) attrs
181+ <> mkIndent indent
182+ <> "}"
183+
184+ escapeNixStr :: Text -> Text
185+ escapeNixStr str = "\"" <> T.replace "\"" "\\\"" (T.replace "\\" "\\\\" str) <> "\""
186+
187+ mkAttr :: Int -> ModuleValues.NixIdentifier -> ModuleValues.NixValue -> Text
188+ mkAttr indent key value = mkIndent indent <> ModuleValues.getNixIdentifier key <> " = " <> go indent value <> ";\n"
189+
190+ mkIndent :: Int -> Text
191+ mkIndent n = cs $ replicate (2 * n) ' '
backend/src/Garnix/Build/Package.hsnew file+266
@@ -0,0 +1,266 @@
1+module Garnix.Build.Package where
2+
3+import Control.Concurrent.Async.Lifted
4+import Control.Lens
5+import Cradle
6+import Data.Aeson qualified as Aeson
7+import Data.Text qualified as T
8+import Data.Text.IO qualified as T
9+import Data.UUID (UUID)
10+import Garnix.Async
11+import Garnix.Attribute
12+import Garnix.Build.Evaluation
13+import Garnix.Build.FodCheck qualified as FodCheck
14+import Garnix.Build.Reporting
15+import Garnix.BuildLogs
16+import Garnix.BuildLogs.Types (LogLine (LogLine), mkLogLine)
17+import Garnix.DB qualified as DB
18+import Garnix.Duration
19+import Garnix.Monad
20+import Garnix.Monad.Concurrency
21+import Garnix.Monad.Metrics
22+import Garnix.Monad.SubProcess
23+import Garnix.Nix.Types (DrvPath)
24+import Garnix.NixConfig (addNixConfigEnvironment)
25+import Garnix.Prelude
26+import Garnix.S3Cache qualified as S3Cache
27+import Garnix.Sandbox
28+import Garnix.Types as Types
29+import System.Random (randomIO)
30+
31+doBuild :: Maybe FodChecker -> RunReporter -> BuildKind -> FlakeDir -> RepoConfig -> ProductPlan -> Build -> M Build
32+doBuild fodChecker runReporter buildKind flakeDir repoConfig plan initialBuild = do
33+ attr <- localAttr flakeDir (attribute initialBuild)
34+ withMessage ("Running build for " <> attr) $ do
35+ withSpan (initialBuild ^. id, initialBuild ^. packageType, initialBuild ^. system, initialBuild ^. package) $ do
36+ -- Catch both IO exceptions and M errors
37+ ( runBuild
38+ `catchError` \_ -> do
39+ returnFailedBuild
40+ )
41+ `catch` \e -> do
42+ log Warning $ "An exception occurred while building the package: " <> show (e :: SomeException)
43+ returnFailedBuild
44+ where
45+ returnFailedBuild :: M Build
46+ returnFailedBuild = do
47+ let build = initialBuild & status ?~ Failure
48+ reportBuildResult runReporter build
49+ pure build
50+
51+ runBuild :: M Build
52+ runBuild = do
53+ build <- buildPkg fodChecker runReporter buildKind flakeDir repoConfig plan initialBuild <?> "Starting build"
54+ persistence <- getPersistenceName flakeDir build
55+ let updatedBuild = build & persistenceName .~ persistence
56+ reportBuildResult runReporter updatedBuild <?> "Reporting final build result"
57+ pure updatedBuild
58+
59+getPersistenceName :: FlakeDir -> Build -> M (Maybe Text)
60+getPersistenceName flakeDir b = do
61+ let isNixos = b ^. packageType == TypeNixosConfiguration
62+ succeeded = b ^. status == Just Success
63+ workingDir <- view #workingDir
64+ cacheDir <- getNixXdgCacheDir
65+ nixConfig <- view #userNixConfig
66+ flakeDir' <- safeGetAbsoluteFlakeDir flakeDir
67+ if isNixos && succeeded
68+ then do
69+ (exit, StdoutRaw s) <-
70+ (>>= run)
71+ $ cmd "nix"
72+ & addArgs
73+ [ "eval",
74+ cs flakeDir' <> "#nixosConfigurations." <> cs (b ^. package),
75+ "--apply",
76+ "c : if c.config.garnix.server.persistence.enable then c.config.garnix.server.persistence.name else null",
77+ "--json" :: Text
78+ ]
79+ & addNixConfigEnvironment nixConfig
80+ & setWorkingDir workingDir
81+ & silenceStderr
82+ & pure
83+ & inNixSandbox [] (Just cacheDir)
84+
85+ pure $ case exit of
86+ ExitFailure _ -> Nothing
87+ ExitSuccess -> case Aeson.decodeStrict @Text s of
88+ Just "" -> Nothing
89+ other -> other
90+ else pure Nothing
91+
92+buildPkg ::
93+ (HasCallStack) =>
94+ Maybe FodChecker ->
95+ RunReporter ->
96+ BuildKind ->
97+ FlakeDir ->
98+ RepoConfig ->
99+ ProductPlan ->
100+ Build ->
101+ M Build
102+buildPkg = curry7
103+ $ mockable #buildPkgMock
104+ $ \(fodChecker, runReporter, buildKind, flakeDir, repoConfig, productPlan, build) -> do
105+ incrementEvent #packageBuildsAttempted
106+ cacheDir <- getNixXdgCacheDir
107+ attr <- localAttr flakeDir . addNixosExtension . attribute $ build
108+ workingDir <- view #workingDir
109+ evalRes <- evaluateAttribute repoConfig productPlan cacheDir workingDir build attr
110+ case evalRes of
111+ Right evaluationResult -> do
112+ let drvPath' = evaluationResult ^. #derivation
113+ FodCheck.fodCheck fodChecker drvPath'
114+ build <-
115+ if null $ evaluationResult ^. #toUpload
116+ then do
117+ log Informational $ "No derivations to upload for " <> cs drvPath'
118+ DB.setBuildUploaded (build ^. id)
119+ pure $ build
120+ & status ?~ Success
121+ & alreadyBuilt ?~ True
122+ else do
123+ let builder = runNixBuild runReporter productPlan cacheDir workingDir build drvPath'
124+ status' <- withAsync builder $ \q -> do
125+ abortOnCancellation build q
126+ log Informational "buildPkg: build finished, checking status"
127+ forkM $ do
128+ S3Cache.upload runReporter (build ^. repoUser) (build ^. repoName) evaluationResult (build ^. repoIsPublic)
129+ DB.setBuildUploaded (build ^. id)
130+ case status' of
131+ Failure -> log Warning "build failed"
132+ Cancelled -> log Notice "build cancelled"
133+ _ -> pure ()
134+ pure $ build
135+ & status ?~ status'
136+ & alreadyBuilt ?~ False
137+ buildEnd <- liftIO getCurrentTime
138+ pure $ build
139+ & drvPath ?~ cs (evaluationResult ^. #derivation)
140+ & outputPaths ?~ BuildOutputsPgColumn (evaluationResult ^. #outputs)
141+ & endTime ?~ buildEnd
142+ Left err -> do
143+ build <- do
144+ evalEnd <- liftIO getCurrentTime
145+ pure $ build
146+ & status ?~ Failure
147+ & endTime ?~ evalEnd
148+ & alreadyBuilt ?~ False
149+ case err of
150+ (AttributeIsSourceOutput src) -> do
151+ log Warning $ "found src derivation: " <> cs src
152+ reportLogs runReporter $ mkLogLine $ "failed output is source path: not supported (" <> cs src <> ")"
153+ pure build
154+ TimeoutReached -> do
155+ let message = "Timed out during nix evaluation of: " <> attr
156+ reportLogs runReporter $ mkLogLine message
157+ pure $ build
158+ & status ?~ Timeout
159+ & endTime .~ Nothing
160+ (NixEvaluationError (Stderr errorMessage) (RanCommand command)) -> do
161+ log Warning $ "package evaluation failed: " <> cs errorMessage
162+ case buildKind of
163+ Webhook -> do
164+ reportLogs runReporter $ LogLine (Just $ build ^. package) Nothing $ "failed running package evaluation. If you have `nix` installed, you can reproduce the error locally by running: " <> cs command
165+ reportLogs runReporter $ LogLine (Just $ build ^. package) Nothing $ cs errorMessage
166+ ModulePreview -> do
167+ log Critical
168+ $ "Module evaluation error for https://github.com/"
169+ <> getGhLogin (getGhRepoOwner $ build ^. repoUser)
170+ <> "/"
171+ <> getGhRepoName (build ^. repoName)
172+ <> "/commit/"
173+ <> getCommitHash (build ^. gitCommit)
174+ <> " error: "
175+ <> cs errorMessage
176+ liftIO (T.readFile (workingDir </> "flake.nix")) >>= log Informational
177+
178+ reportLogs runReporter
179+ $ LogLine (Just $ build ^. package) Nothing
180+ $ T.unlines
181+ [ "Package evaluation failed. The error message is:",
182+ "",
183+ cs errorMessage,
184+ "",
185+ "(This may be caused by a misconfiguration on your part. Go to https://garnix.io/modules/configure to correct this. This could also be a bug in the module. Consider opening an issue on https://github.com/garnix-io/issues/issues."
186+ ]
187+ pure build
188+ (AppEvalParseError (Stdout stdout) (Stderr err)) -> do
189+ log Critical
190+ $ "parsing nix eval for apps failed "
191+ <> cs err
192+ <> " ("
193+ <> cs stdout
194+ <> ")"
195+ reportLogs runReporter $ mkLogLine "failed parsing nix outputs for app: unexpected format"
196+ pure build
197+ (ParseError (ParsingError err) (Stdout json) _stderr) -> do
198+ log Critical
199+ $ "parsing nix build dry-run failed "
200+ <> cs err
201+ <> " ("
202+ <> cs json
203+ <> ")"
204+ reportLogs runReporter $ mkLogLine "failed parsing nix outputs: unexpected format"
205+ pure build
206+ (UnexpectedNumberOfParsedResults (NumberOfParsedResults 0) _stdout) -> do
207+ log Critical "parsing nix build dry-run failed: empty result"
208+ reportLogs runReporter $ mkLogLine "failed parsing nix outputs: empty result"
209+ pure build
210+ (UnexpectedNumberOfParsedResults (NumberOfParsedResults num) (Stdout stdout)) -> do
211+ log Critical $ "parsing nix build dry-run failed: got " <> show num <> " results" <> " (" <> cs stdout <> ")"
212+ reportLogs runReporter $ mkLogLine "failed parsing nix outputs: too many results"
213+ pure build
214+
215+abortOnCancellation :: Build -> Async (Either ErrorWithContext Status) -> M Status
216+abortOnCancellation build builder = do
217+ let go = do
218+ b <- DB.getBuild $ build ^. id
219+ case b ^. status of
220+ Just Cancelled -> pure ()
221+ _ -> threadDelay (fromSeconds @Int 30) >> go
222+ withAsync go $ \isCancelled -> do
223+ result <- waitEither builder isCancelled
224+ pure $ case result of
225+ Left status -> status
226+ Right _ -> Cancelled
227+
228+runNixBuild :: RunReporter -> ProductPlan -> String -> FilePath -> Build -> DrvPath -> M Status
229+runNixBuild runReporter productPlan cacheDir workingDir build drvPath = do
230+ nixConfig <- view #userNixConfig
231+ log Informational $ "runNixBuild: using nixConfig '" <> show nixConfig <> "'"
232+ processor <- buildInternalLogProcessor (reportLogs runReporter) <$> mkInternalLogProcessorState <?> "runNixBuild: buildInternalLogProcessor"
233+ -- This is a unique ID for the outlink (which is stored in the working dir) so
234+ -- that nothing gets garbage collected until the working dir is.
235+ uuid :: UUID <- randomIO
236+ mExitCode <-
237+ withTextSpan ("phase", "build") $ do
238+ withUtf8LinesStream processor $ \logHandle -> do
239+ timeout (fromMinutes $ productPlan ^. packageBuildTimeout)
240+ $ (>>= run)
241+ $ cmd "comment"
242+ & addArgs
243+ [ buildComment build,
244+ "--",
245+ "nix",
246+ "build",
247+ T.append (cs drvPath) "^*",
248+ "--log-lines",
249+ "0",
250+ "--print-build-logs",
251+ "--log-format",
252+ "internal-json",
253+ "--out-link",
254+ show uuid
255+ ]
256+ & addNixConfigEnvironment nixConfig
257+ & addStdoutHandle logHandle
258+ & addStderrHandle logHandle
259+ & setWorkingDir workingDir
260+ & pure
261+ & inNixSandbox [] (Just cacheDir)
262+ log Informational $ "runNixBuild: exit code is " <> show mExitCode
263+ pure $ case mExitCode of
264+ Nothing -> Timeout
265+ Just (ExitFailure _) -> Failure
266+ Just ExitSuccess -> Success
backend/src/Garnix/Build/Reporting.hsnew file+55
@@ -0,0 +1,55 @@
1+module Garnix.Build.Reporting
2+ ( reportOnError,
3+ reportBuildResult,
4+ reportNameForBuild,
5+ )
6+where
7+
8+import Control.Lens
9+import Garnix.BuildLogs.Types (mkLogLine)
10+import Garnix.DB qualified as DB
11+import Garnix.Monad
12+import Garnix.Prelude
13+import Garnix.Types as Types
14+
15+reportNameForBuild :: Build -> Text
16+reportNameForBuild build =
17+ let isBuildStarting = (build ^. package) == buildStarting
18+ sysBrackets = case build ^. system of
19+ NoSystem -> ""
20+ IsSystem sys -> " [" <> sys ^. systemTextIso <> "]"
21+ in if isBuildStarting
22+ then "Evaluate flake.nix"
23+ else case build ^. packageType of
24+ TypeOverall -> "Evaluate flake.nix"
25+ TypeDevShell ->
26+ "devShell " <> cs (build ^. package) <> sysBrackets
27+ TypePackage ->
28+ "package " <> cs (build ^. package) <> sysBrackets
29+ TypeCheck ->
30+ "check " <> cs (build ^. package) <> sysBrackets
31+ TypeHomeConfiguration -> "homeConfig " <> cs (build ^. package)
32+ TypeDarwinConfiguration -> "darwinConfig " <> cs (build ^. package)
33+ TypeNixosConfiguration -> "nixosConfig " <> cs (build ^. package)
34+ TypeDefaultPackage -> "default package" <> sysBrackets
35+ TypeDefaultDevShell -> "default devShell" <> sysBrackets
36+ TypeApp -> "app " <> cs (build ^. package)
37+
38+reportBuildResult :: RunReporter -> Build -> M ()
39+reportBuildResult runReporter build = do
40+ DB.reportBuildResultDB build
41+ let status' = case build ^. status of
42+ Nothing -> RunReportStatusInProgress
43+ Just Success -> RunReportStatusSuccess
44+ Just Failure -> RunReportStatusFailure
45+ Just Timeout -> RunReportStatusTimeout
46+ Just Cancelled -> RunReportStatusCancelled
47+ ignoringAllErrors $ reportComplete runReporter status'
48+
49+reportOnError :: RunReporter -> Build -> CommitInfo -> M a -> M a
50+reportOnError runReporter build commitInfo io = do
51+ io `whenError` \e -> do
52+ DB.setCommitStatus (commitInfo ^. repoInfo . ghRepoOwner) (commitInfo ^. repoInfo . ghRepoName) (commitInfo ^. commit) Evaluated
53+ reportLogs runReporter $ mkLogLine $ showPretty (err e)
54+ reportBuildResult runReporter $ build
55+ & status ?~ Failure
backend/src/Garnix/Build/Types.hsnew file+21
@@ -0,0 +1,21 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+
3+module Garnix.Build.Types where
4+
5+import Garnix.Nix.Types qualified as Nix
6+import Garnix.Prelude
7+import Garnix.Types
8+
9+data AppBuildDetails = AppBuildDetails
10+ { _appBuildDetailsAppExecPath :: Nix.AppExecPath,
11+ _appBuildDetailsDrvPath :: Nix.DrvPath
12+ }
13+
14+makeFields ''AppBuildDetails
15+
16+data EvaluationResult = EvaluationResult
17+ { derivation :: Nix.DrvPath,
18+ toUpload :: [Nix.StorePath],
19+ outputs :: Nix.BuildOutputs
20+ }
21+ deriving stock (Show, Generic, Eq)
backend/src/Garnix/BuildLogs.hsnew file+90
@@ -0,0 +1,90 @@
1+module Garnix.BuildLogs
2+ ( buildInternalLogProcessor,
3+ mkInternalLogProcessorState,
4+ processLogsForGithub,
5+ processMessage,
6+ shouldKeepLn,
7+ )
8+where
9+
10+import Control.Concurrent
11+import Data.Aeson hiding (Error)
12+import Data.ByteString.Lazy qualified as BS
13+import Data.Map (Map)
14+import Data.Map qualified as Map
15+import Data.String.AnsiEscapeCodes.Strip.Text (stripAnsiEscapeCodes)
16+import Data.Text qualified as T
17+import Garnix.BuildLogs.Internal qualified as LogsInternal
18+import Garnix.BuildLogs.Types (LogLine (LogLine), mkLogLine)
19+import Garnix.Monad
20+import Garnix.Prelude
21+import Garnix.Types
22+
23+-- * Common
24+
25+processMessage :: Text -> Text
26+processMessage =
27+ T.unlines
28+ . filter shouldKeepLn
29+ . T.lines
30+ . obfuscateGithubToken
31+ . stripAnsiEscapeCodes
32+
33+shouldKeepLn :: Text -> Bool
34+shouldKeepLn ln =
35+ not ("error (ignored): error: SQLite database" `T.isPrefixOf` ln)
36+ && not ("garnixMacBuilder" `T.isInfixOf` ln)
37+ && not ("macMini" `T.isInfixOf` ln)
38+ && not ("/run/secrets/garnix_server_ssh" `T.isInfixOf` ln)
39+
40+-- * Parsing internal nix logs for opensearch
41+
42+data InternalLogProcessorState = InternalLogProcessorState (MVar (Map LogsInternal.InternalId Text)) (MVar (Map LogsInternal.InternalId PackageName))
43+
44+mkInternalLogProcessorState :: M InternalLogProcessorState
45+mkInternalLogProcessorState = liftIO $ InternalLogProcessorState <$> newMVar mempty <*> newMVar mempty
46+
47+buildInternalLogProcessor :: (LogLine -> M ()) -> InternalLogProcessorState -> (String -> M ())
48+buildInternalLogProcessor consumer (InternalLogProcessorState phaseById pkgsById) internalLogLine = do
49+ let jsonStr = BS.stripPrefix "@nix" (cs internalLogLine)
50+ parsedLine = case jsonStr of
51+ Just line -> eitherDecode' line
52+ Nothing -> Left $ "Expected internal log line from nix but got: " <> internalLogLine
53+ case parsedLine of
54+ Right (LogsInternal.Start id _ (LogsInternal.Build drvPath _)) -> setPackage id $ drvPathToPkgName drvPath
55+ Right (LogsInternal.Result id (LogsInternal.SetPhase phase)) -> setPhase id phase
56+ Right (LogsInternal.Result id (LogsInternal.LogLine log)) -> LogLine <$> getPkgForId id <*> getPhaseForId id <*> pure log >>= consumer
57+ Right (LogsInternal.Msg log) -> consumer $ mkLogLine log
58+ Right _ -> pure ()
59+ Left err -> log Warning $ cs err
60+ where
61+ setPackage :: LogsInternal.InternalId -> PackageName -> M ()
62+ setPackage id pkgName = liftIO $ modifyMVar_ pkgsById $ pure . Map.insert id pkgName
63+
64+ getPkgForId :: LogsInternal.InternalId -> M (Maybe PackageName)
65+ getPkgForId id = liftIO $ Map.lookup id <$> readMVar pkgsById
66+
67+ setPhase :: LogsInternal.InternalId -> Text -> M ()
68+ setPhase id phase = liftIO $ modifyMVar_ phaseById $ pure . Map.insert id phase
69+
70+ getPhaseForId :: LogsInternal.InternalId -> M (Maybe Text)
71+ getPhaseForId id = liftIO $ Map.lookup id <$> readMVar phaseById
72+
73+drvPathToPkgName :: Text -> PackageName
74+drvPathToPkgName = PackageName . T.dropEnd suffixLength . T.drop prefixLength
75+ where
76+ prefixLength = T.length "/nix/store/this-is-a-fictional-hash-.......-"
77+ suffixLength = T.length ".drv"
78+
79+-- * Github-specific
80+
81+processLogsForGithub :: RawLogs -> Text
82+processLogsForGithub (RawLogs l) =
83+ "Last 100 lines of logs:\n"
84+ <> "```\n"
85+ <> T.unlines (reverse $ take 100 $ reverse $ fmap limitLine $ T.lines $ processMessage l)
86+ <> "```\n"
87+ where
88+ limitLine ln
89+ | T.length ln > 650 = T.take 649 ln <> "…"
90+ | otherwise = ln
backend/src/Garnix/BuildLogs/Internal.hsnew file+117
@@ -0,0 +1,117 @@
1+module Garnix.BuildLogs.Internal where
2+
3+import Data.Aeson
4+import Garnix.Prelude
5+
6+newtype InternalId = InternalId Int64
7+ deriving stock (Eq, Show, Generic, Ord)
8+
9+instance FromJSON InternalId where parseJSON = ourParseJSON
10+
11+-- https://github.com/NixOS/nix/blob/bd7a0746361d42a121b2cef1571bda4f7c370c16/src/libutil/error.hh#L37-L46
12+data LogLevel
13+ = LvlError
14+ | LvlWarn
15+ | LvlNotice
16+ | LvlInfo
17+ | LvlTalkative
18+ | LvlChatty
19+ | LvlDebug
20+ | LvlVomit
21+ deriving stock (Eq, Show, Generic)
22+
23+instance FromJSON LogLevel where
24+ parseJSON = withScientific "LogLevel" $ \v -> do
25+ case v of
26+ 0 -> pure LvlError
27+ 1 -> pure LvlWarn
28+ 2 -> pure LvlNotice
29+ 3 -> pure LvlInfo
30+ 4 -> pure LvlTalkative
31+ 5 -> pure LvlChatty
32+ 6 -> pure LvlDebug
33+ 7 -> pure LvlVomit
34+ _ -> fail $ "Unknown log level: " <> cs (show v)
35+
36+data ProgressReport = ProgressReport
37+ { done :: Int64,
38+ expected :: Int64,
39+ running :: Int64,
40+ failed :: Int64
41+ }
42+ deriving stock (Eq, Show, Generic)
43+
44+data InternalBuildLogLineStart
45+ = Build Text Text
46+ | Builds Text
47+ | CopyPaths Text
48+ | Unknown Text
49+ | QueryPathInfo Text
50+ | FileTransfer Text
51+ | Realize Text
52+ | OtherStartType -- Catchall for types we don't care about
53+ deriving stock (Eq, Show, Generic)
54+
55+data InternalBuildLogLineResult
56+ = SetPhase Text
57+ | SetExpected Int Int
58+ | LogLine Text
59+ | Progress ProgressReport
60+ | OtherResultType -- Catchall for types we don't care about
61+ deriving stock (Eq, Show, Generic)
62+
63+data InternalBuildLogLine
64+ = Start InternalId LogLevel InternalBuildLogLineStart
65+ | Stop InternalId
66+ | Result InternalId InternalBuildLogLineResult
67+ | Unhandled
68+ | Msg Text
69+ deriving stock (Eq, Show, Generic)
70+
71+instance FromJSON InternalBuildLogLine where
72+ parseJSON = withObject "InternalBuildLogLine" $ \v -> do
73+ action :: Text <- v .: "action"
74+ case action of
75+ "start" -> do
76+ typ :: Int <- v .: "type"
77+ id :: InternalId <- v .: "id"
78+ level :: LogLevel <- v .: "level"
79+ -- `typ` is `ActivityType` declared here:
80+ -- https://github.com/NixOS/nix/blob/bd7a0746361d42a121b2cef1571bda4f7c370c16/src/libutil/logging.hh#L12-L27
81+ Start id level <$> case typ of
82+ 0 -> Unknown <$> v .: "text"
83+ 101 -> FileTransfer <$> v .: "text"
84+ 102 -> Realize <$> v .: "text"
85+ 103 -> CopyPaths <$> v .: "text"
86+ 104 -> Builds <$> v .: "text"
87+ 105 -> do
88+ text <- v .: "text"
89+ (drvPath, _, _, _) :: (Text, Text, Int, Int) <- v .: "fields"
90+ pure $ Build drvPath text
91+ 109 -> QueryPathInfo <$> v .: "text"
92+ _ -> pure OtherStartType
93+ "stop" -> Stop <$> v .: "id"
94+ "result" -> do
95+ -- `typ` is `ResultType` declared here:
96+ -- https://github.com/NixOS/nix/blob/bd7a0746361d42a121b2cef1571bda4f7c370c16/src/libutil/logging.hh#L29-L39
97+ typ :: Int <- v .: "type"
98+ id :: InternalId <- v .: "id"
99+ Result id <$> case typ of
100+ 101 -> LogLine <$> (fromSingleton =<< v .: "fields")
101+ 104 -> SetPhase <$> (fromSingleton =<< v .: "fields")
102+ 105 -> do
103+ (done, expected, running, failed) <- v .: "fields"
104+ pure $ Progress $ ProgressReport done expected running failed
105+ 106 -> do
106+ (actType, expected) <- v .: "fields"
107+ pure $ SetExpected actType expected
108+ _ -> pure OtherResultType
109+ "msg" -> do
110+ msg :: Text <- v .: "msg"
111+ pure $ Msg msg
112+ a -> fail $ "Unknown action: " <> cs a
113+ where
114+ fromSingleton arr = do
115+ case arr of
116+ [value] -> pure value
117+ _ -> fail "fromSingleton: Expected a single element"
backend/src/Garnix/BuildLogs/Types.hsnew file+14
@@ -0,0 +1,14 @@
1+module Garnix.BuildLogs.Types (LogLine (..), mkLogLine) where
2+
3+import Garnix.Prelude
4+import Garnix.Types
5+
6+data LogLine = LogLine
7+ { package :: Maybe PackageName,
8+ phase :: Maybe Text,
9+ log :: Text
10+ }
11+ deriving stock (Eq, Show, Generic)
12+
13+mkLogLine :: Text -> LogLine
14+mkLogLine = LogLine Nothing Nothing
backend/src/Garnix/DB.hsnew file+1995
@@ -0,0 +1,1995 @@
1+module Garnix.DB where
2+
3+import Control.Exception.Safe qualified
4+import Control.Exception.Safe qualified as Safe
5+import Control.Lens
6+import Control.Monad.Trans.Control (liftBaseOp_)
7+import Data.ByteString qualified
8+import Data.Map qualified as Map
9+import Data.Map.Strict (Map, fromList)
10+import Data.Pool (withResource)
11+import Data.Set qualified as Set
12+import Data.Text.IO (hPutStrLn)
13+import Database.PostgreSQL.Typed (PGDatabase (pgDBPass), pgConnect, pgSQL)
14+import Database.PostgreSQL.Typed qualified as PSQL
15+import Database.PostgreSQL.Typed.Array ()
16+import Database.PostgreSQL.Typed.Protocol qualified as PSQLP
17+import Database.PostgreSQL.Typed.Query (PGQuery, getQueryString)
18+import Database.PostgreSQL.Typed.TH (getTPGDatabase)
19+import Database.PostgreSQL.Typed.Types (unknownPGTypeEnv)
20+import Garnix.AccessToken.Types
21+import Garnix.Duration
22+import Garnix.Hosting.ServerPool.Types
23+import Garnix.Monad
24+import Garnix.Monad.Metrics (incrementEvent, timingAs)
25+import Garnix.Nix.Types as Nix
26+import Garnix.Password
27+import Garnix.Prelude
28+import Garnix.Types
29+
30+getUser :: GhLogin -> M User
31+getUser ghLogin = do
32+ res <-
33+ pgQuery
34+ [pgSQL|
35+ SELECT
36+ id,
37+ email,
38+ subscription_type,
39+ created_at
40+ FROM users
41+ WHERE github_login = ${ghLogin}
42+ |]
43+ case res of
44+ [] -> throw $ NoSuchUser ghLogin
45+ [(id', email', sub', cre')] -> pure $ User id' ghLogin email' sub' cre'
46+ _ -> throw $ OtherError "Got more than 1 user from getUser"
47+
48+getUserId :: GhLogin -> M UserId
49+getUserId ghLogin = do
50+ res <- pgQuery [pgSQL| SELECT id FROM users WHERE github_login = ${ghLogin} |]
51+ case res of
52+ [] -> throw $ NoSuchUser ghLogin
53+ [id] -> pure $ UserId id
54+ _ -> throw $ OtherError "Got more than 1 user from getUserId"
55+
56+newUser :: GhLogin -> Email -> SubscriptionType -> Bool -> M User
57+newUser ghLogin email' sub agreeToEmails' = do
58+ r <-
59+ pgQuery
60+ [pgSQL|
61+ INSERT INTO users
62+ ( github_login,
63+ email,
64+ subscription_type,
65+ agree_to_emails
66+ )
67+ VALUES
68+ ( ${ghLogin},
69+ ${email'},
70+ ${sub},
71+ ${agreeToEmails'}
72+ )
73+ ON CONFLICT DO NOTHING
74+ RETURNING id, created_at
75+ |]
76+ case r of
77+ [(id', cre')] ->
78+ pure
79+ $ User
80+ { _userId = id',
81+ _userGithubLogin = ghLogin,
82+ _userEmail = email',
83+ _userSubscriptionType = sub,
84+ _userCreatedAt = cre'
85+ }
86+ [] -> throw $ UserAlreadyExists ghLogin
87+ _ -> throw $ OtherError "impossible: more than two users created"
88+
89+getRepoConfig :: GhRepoOwner -> GhRepoName -> M RepoConfig
90+getRepoConfig repoOwner repoName = do
91+ repoConfig <-
92+ map (\(skipInputChecks, evalMemory) -> RepoConfig skipInputChecks (fromMaybe (defaultRepoConfig ^. maxEvalMemory) evalMemory))
93+ <$> pgQuery
94+ [pgSQL|
95+ SELECT
96+ skip_private_inputs_check_for_collaborators,
97+ max_eval_memory
98+ FROM repo_config
99+ WHERE repo_user = ${repoOwner}
100+ AND repo_name = ${repoName}
101+ |]
102+ case repoConfig of
103+ [] -> pure defaultRepoConfig
104+ [res] -> pure res
105+ _ -> throw $ OtherError "impossible: multiple entries for repo config"
106+
107+getBuild :: BuildId -> M Build
108+getBuild buildId = do
109+ res <-
110+ pgQueryPrism
111+ _Build
112+ [pgSQL|
113+ SELECT
114+ id,
115+ repo_user,
116+ repo_name,
117+ pr_from_fork,
118+ branch,
119+ repo_is_public,
120+ git_commit,
121+ package,
122+ package_type,
123+ system,
124+ req_user,
125+ status,
126+ start_time,
127+ end_time,
128+ drv_path,
129+ output_paths,
130+ github_run_id,
131+ persistence_name,
132+ wants_incrementalism,
133+ eval_host,
134+ uploaded_to_cache,
135+ already_built
136+ FROM builds
137+ WHERE id = ${buildId}
138+ |]
139+ case res of
140+ [r] -> pure r
141+ [] -> throw $ NoSuchBuild buildId
142+ _ -> throw $ OtherError "Impossible: more than one result"
143+
144+getOriginalBuildForDrvPath :: Maybe User -> FilePath -> M (Maybe OriginalBuild)
145+getOriginalBuildForDrvPath user drvPath = do
146+ let mghLogin = user ^? _Just . githubLogin
147+ isAdmin = user ^? _Just . subscriptionType == Just Admin
148+ res <-
149+ map (\(id, commit, status) -> OriginalBuild id commit status)
150+ <$> pgQuery
151+ [pgSQL|
152+ SELECT
153+ id,
154+ git_commit,
155+ status
156+ FROM builds
157+ WHERE (
158+ repo_is_public
159+ OR ${isAdmin}
160+ OR (${mghLogin}::text IS NOT NULL AND req_user = ${mghLogin})
161+ )
162+ AND already_built = false
163+ AND drv_path = ${drvPath}
164+ ORDER BY end_time DESC
165+ LIMIT 1
166+ |]
167+ case res of
168+ [r] -> pure $ Just r
169+ [] -> pure Nothing
170+ _ -> throw $ OtherError "Impossible: more than one result"
171+
172+makeNewBuildForGithubRunId :: GhLogin -> GhRunId -> Text -> M Build
173+makeNewBuildForGithubRunId reqUser ghRunId evalHost = do
174+ now <- liftIO getCurrentTime
175+ res <-
176+ pgQueryPrism
177+ _Build
178+ [pgSQL|
179+ INSERT INTO builds
180+ ( repo_user,
181+ repo_name,
182+ pr_from_fork,
183+ branch,
184+ repo_is_public,
185+ git_commit,
186+ package,
187+ package_type,
188+ system,
189+ req_user,
190+ status,
191+ start_time,
192+ end_time,
193+ drv_path,
194+ output_paths,
195+ github_run_id,
196+ persistence_name,
197+ wants_incrementalism,
198+ eval_host,
199+ uploaded_to_cache)
200+ SELECT
201+ repo_user,
202+ repo_name,
203+ pr_from_fork,
204+ branch,
205+ repo_is_public,
206+ git_commit,
207+ package,
208+ package_type,
209+ system,
210+ ${reqUser},
211+ NULL,
212+ ${now},
213+ NULL,
214+ NULL,
215+ NULL,
216+ NULL,
217+ NULL,
218+ wants_incrementalism,
219+ ${evalHost},
220+ FALSE
221+ FROM builds
222+ WHERE github_run_id = ${ghRunId}
223+ RETURNING
224+ id,
225+ repo_user,
226+ repo_name,
227+ pr_from_fork,
228+ branch,
229+ repo_is_public,
230+ git_commit,
231+ package,
232+ package_type,
233+ system,
234+ req_user,
235+ status,
236+ start_time,
237+ end_time,
238+ drv_path,
239+ output_paths,
240+ github_run_id,
241+ persistence_name,
242+ wants_incrementalism,
243+ eval_host,
244+ uploaded_to_cache,
245+ already_built
246+ |]
247+ case res of
248+ [r] -> pure r
249+ [] -> throw $ NoSuchBuildRunId ghRunId
250+ _ -> throw $ OtherError "Impossible: more than one result"
251+
252+getLatestBuildsMatching :: RepoInfo -> CommitHash -> M [Build]
253+getLatestBuildsMatching repoInfo commit = do
254+ pgQueryPrism
255+ _Build
256+ [pgSQL|
257+ SELECT DISTINCT ON (repo_user, repo_name, git_commit, package, package_type, system)
258+ id,
259+ repo_user,
260+ repo_name,
261+ pr_from_fork,
262+ branch,
263+ repo_is_public,
264+ git_commit,
265+ package,
266+ package_type,
267+ system,
268+ req_user,
269+ status,
270+ start_time,
271+ end_time,
272+ drv_path,
273+ output_paths,
274+ github_run_id,
275+ persistence_name,
276+ wants_incrementalism,
277+ eval_host,
278+ uploaded_to_cache,
279+ already_built
280+ FROM builds
281+ WHERE repo_user = ${repoInfo ^. ghRepoOwner}
282+ AND repo_name = ${repoInfo ^. ghRepoName}
283+ AND git_commit = ${commit}
284+ ORDER BY
285+ repo_user, repo_name, git_commit, package, package_type, system,
286+ start_time DESC
287+ |]
288+
289+getBuilds :: User -> M [Build]
290+getBuilds usr = do
291+ pgQueryPrism
292+ _Build
293+ [pgSQL|
294+ SELECT
295+ id,
296+ repo_user,
297+ repo_name,
298+ pr_from_fork,
299+ branch,
300+ repo_is_public,
301+ git_commit,
302+ package,
303+ package_type,
304+ system,
305+ req_user,
306+ status,
307+ start_time,
308+ end_time,
309+ drv_path,
310+ output_paths,
311+ github_run_id,
312+ persistence_name,
313+ wants_incrementalism,
314+ eval_host,
315+ uploaded_to_cache,
316+ already_built
317+ FROM builds
318+ WHERE req_user = ${usr ^. githubLogin}
319+ ORDER BY start_time DESC
320+ LIMIT 500
321+ |]
322+
323+setBuildUploaded :: BuildId -> M ()
324+setBuildUploaded buildId = do
325+ void
326+ $ pgExec
327+ [pgSQL|
328+ UPDATE builds
329+ SET uploaded_to_cache = TRUE
330+ WHERE id = ${buildId}
331+ |]
332+
333+getLatestBuildsForBranch :: GhRepoOwner -> GhRepoName -> Branch -> M [Build]
334+getLatestBuildsForBranch owner name branch = do
335+ pgQueryPrism
336+ _Build
337+ [pgSQL|
338+ SELECT
339+ id,
340+ repo_user,
341+ repo_name,
342+ pr_from_fork,
343+ branch,
344+ repo_is_public,
345+ git_commit,
346+ package,
347+ package_type,
348+ system,
349+ req_user,
350+ status,
351+ start_time,
352+ end_time,
353+ drv_path,
354+ output_paths,
355+ github_run_id,
356+ persistence_name,
357+ wants_incrementalism,
358+ eval_host,
359+ uploaded_to_cache,
360+ already_built
361+ FROM builds
362+ WHERE repo_user = ${owner}
363+ AND branch = ${branch}
364+ AND repo_name = ${name}
365+ AND git_commit = (
366+ SELECT git_commit
367+ FROM builds
368+ WHERE repo_user = ${owner}
369+ AND repo_name = ${name}
370+ AND branch = ${branch}
371+ AND package = 'Build starting'
372+ ORDER BY start_time DESC
373+ LIMIT 1
374+ )
375+ AND repo_is_public = TRUE
376+ |]
377+
378+data RegisterPushResult = NewPush | AlreadyPushed
379+ deriving stock (Eq, Show, Generic)
380+
381+registerPush :: GhRepoOwner -> GhRepoName -> CommitHash -> Branch -> M RegisterPushResult
382+registerPush repoOwner repoName commit branch = do
383+ pgQuery
384+ [pgSQL|
385+ INSERT INTO pushes
386+ (repo_user,
387+ repo_name,
388+ git_commit,
389+ branch
390+ )
391+ VALUES
392+ (${repoOwner},
393+ ${repoName},
394+ ${commit},
395+ ${branch}
396+ )
397+ ON CONFLICT DO NOTHING
398+ RETURNING True
399+ |]
400+ >>= \case
401+ [] -> pure AlreadyPushed
402+ [_ :: Maybe Bool] -> pure NewPush
403+ _ -> throw $ OtherError "Impossible: more than one result"
404+
405+getCommitsByOwnerAndRepo :: GhRepoOwner -> GhRepoName -> M [CommitSummary]
406+getCommitsByOwnerAndRepo repoOwner repoName = do
407+ map
408+ ( \( repoOwner :: GhRepoOwner,
409+ repoName :: GhRepoName,
410+ gitCommit :: CommitHash,
411+ branch :: Maybe Branch,
412+ reqUser :: GhLogin,
413+ isPublic :: Bool,
414+ startTime :: UTCTime,
415+ succeeded :: Int64,
416+ failed :: Int64,
417+ pending :: Int64,
418+ cancelled :: Int64
419+ ) ->
420+ CommitSummary repoOwner repoName (RepoIsPublic isPublic) gitCommit branch reqUser startTime succeeded failed pending cancelled
421+ )
422+ <$> pgQuery
423+ [pgSQL|!
424+ SELECT
425+ (array_agg(repo_user))[1],
426+ (array_agg(repo_name))[1],
427+ git_commit,
428+ (array_agg(branch))[1],
429+ (array_agg(req_user))[1],
430+ (array_agg(repo_is_public))[1],
431+ min(start_time) as commit_start_time,
432+ COUNT(*) FILTER (WHERE status = 'success') as succeeded,
433+ COUNT(*) FILTER (WHERE status = 'failure' OR status = 'timeout') as failed,
434+ COUNT(*) FILTER (WHERE status IS NULL) as pending,
435+ COUNT(*) FILTER (WHERE status = 'cancelled') as pending
436+ FROM (
437+ SELECT DISTINCT ON (git_commit, package_type, system, package) * FROM builds
438+ WHERE repo_user = ${repoOwner}
439+ AND repo_name = ${repoName}
440+ ORDER BY git_commit, package_type, system, package, start_time DESC
441+ ) AS sub
442+ GROUP BY git_commit
443+ ORDER BY commit_start_time DESC
444+ LIMIT 100
445+ |]
446+
447+getCommit :: GhRepoOwner -> GhRepoName -> CommitHash -> M (Maybe Commit)
448+getCommit owner name commit =
449+ pgQueryPrism
450+ _Commit
451+ [pgSQL|
452+ SELECT
453+ repo_user,
454+ repo_name,
455+ git_commit,
456+ status,
457+ meta_check
458+ FROM commits
459+ WHERE repo_user = ${owner}
460+ AND repo_name = ${name}
461+ AND git_commit = ${commit}
462+ |]
463+ >>= \case
464+ [r] -> pure $ Just r
465+ [] -> pure Nothing
466+ _ -> throw $ OtherError "Impossible: more than one result"
467+
468+newCommit :: GhRepoOwner -> GhRepoName -> CommitHash -> M ()
469+newCommit owner name commit =
470+ void
471+ $ pgExec
472+ [pgSQL|
473+ INSERT INTO commits
474+ (repo_user, repo_name, git_commit, status, meta_check)
475+ VALUES
476+ (${owner}, ${name}, ${commit}, 'evaluating', 'pending')
477+ ON CONFLICT (repo_user, repo_name, git_commit) DO UPDATE
478+ SET meta_check = 'pending'
479+ |]
480+
481+setCommitStatus :: GhRepoOwner -> GhRepoName -> CommitHash -> CommitStatus -> M ()
482+setCommitStatus owner name commit st =
483+ void
484+ $ pgExec
485+ [pgSQL|
486+ UPDATE commits
487+ SET status = ${st}
488+ WHERE repo_user = ${owner}
489+ AND repo_name = ${name}
490+ AND git_commit = ${commit}
491+ |]
492+
493+data CheckStatusUpdate = CheckStatusUpdate
494+ { _checkStatusUpdateFrom :: CheckStatus,
495+ _checkStatusUpdateTo :: CheckStatus
496+ }
497+
498+-- Note: be careful when changing this function.
499+--
500+-- One change you might consider doing is, removing the `from` argument and changing the WHERE
501+-- clause that references it to 'AND meta_check <> ${to}', and at first that may seem equivalent.
502+--
503+-- However, as the code is now, that results in a bug. Here's a scenario for two failed builds,
504+-- both being reran at roughly the same time:
505+--
506+-- buildA: completes successfully, gets all builds by commit (A: Success, B: Failure)
507+-- buildB: completes successfully, gets all builds by commit (A: Success, B: Success),
508+-- runs this function and sets the flag to 'CheckSuccess'
509+-- buildA: resumes and tries to set the check to fail
510+--
511+-- With the changed mentioned above, the check would go through these states: Pending -> Success -> Fail.
512+--
513+-- As it is, buildA's thread would still attempt to set the check to CheckFail, but only if its current
514+-- state is CheckPending, which is not. So no change will happen, which is what we want.
515+setMetaCheck :: GhRepoOwner -> GhRepoName -> CommitHash -> CheckStatusUpdate -> M Bool
516+setMetaCheck owner name commit (CheckStatusUpdate {_checkStatusUpdateFrom = from, _checkStatusUpdateTo = to}) = do
517+ if from == to
518+ then pure False
519+ else
520+ (== 1)
521+ <$> pgExec
522+ [pgSQL|
523+ UPDATE commits
524+ SET meta_check = ${to}
525+ WHERE repo_user = ${owner}
526+ AND repo_name = ${name}
527+ AND git_commit = ${commit}
528+ AND meta_check = ${from}
529+ |]
530+
531+getBuildsAndRunsByCommit :: GhRepoOwner -> GhRepoName -> CommitHash -> M FullCommitState
532+getBuildsAndRunsByCommit repoOwner repoName commitHash = do
533+ mCommit <- getCommit repoOwner repoName commitHash
534+ case mCommit of
535+ Nothing -> pure CommitEvaluating
536+ Just commit -> case commit ^. status of
537+ Evaluating -> pure CommitEvaluating
538+ Evaluated -> do
539+ builds <- getBuildsByCommit repoOwner repoName commitHash
540+ runs <- getRuns repoOwner repoName commitHash
541+ pure $ CommitEvaluated commit builds runs
542+
543+getBuildsByCommit :: GhRepoOwner -> GhRepoName -> CommitHash -> M [Build]
544+getBuildsByCommit repoOwner repoName commitHash = do
545+ pgQuery
546+ [pgSQL|
547+ SELECT DISTINCT ON (git_commit, package_type, system, package)
548+ id,
549+ repo_user,
550+ repo_name,
551+ pr_from_fork,
552+ branch,
553+ repo_is_public,
554+ git_commit,
555+ package,
556+ package_type,
557+ system,
558+ req_user,
559+ status,
560+ start_time,
561+ end_time,
562+ drv_path,
563+ output_paths,
564+ github_run_id,
565+ persistence_name,
566+ wants_incrementalism,
567+ eval_host,
568+ uploaded_to_cache,
569+ already_built
570+ FROM builds
571+ WHERE git_commit = ${commitHash}
572+ AND repo_user = ${repoOwner}
573+ AND repo_name = ${repoName}
574+ ORDER BY git_commit, package_type, system, package, start_time DESC
575+ LIMIT 1000
576+ |]
577+ <&> map
578+ ( \( id,
579+ repoUser,
580+ repoName,
581+ prFromFork,
582+ branch,
583+ repoIsPublic,
584+ gitCommit,
585+ package,
586+ packageType,
587+ system,
588+ reqUser,
589+ status,
590+ startTime,
591+ endTime,
592+ drvPath,
593+ outputPaths,
594+ githubRunId,
595+ persistenceName,
596+ wantsIncrementalism,
597+ evalHost,
598+ uploadedToCache,
599+ alreadyBuilt
600+ ) ->
601+ Build
602+ { _buildId = id,
603+ _buildRepoUser = repoUser,
604+ _buildRepoName = repoName,
605+ _buildPrFromFork = prFromFork,
606+ _buildBranch = branch,
607+ _buildRepoIsPublic = repoIsPublic,
608+ _buildGitCommit = gitCommit,
609+ _buildPackage = package,
610+ _buildPackageType = packageType,
611+ _buildSystem = system,
612+ _buildReqUser = reqUser,
613+ _buildStatus = status,
614+ _buildStartTime = startTime,
615+ _buildEndTime = endTime,
616+ _buildDrvPath = drvPath,
617+ _buildOutputPaths = outputPaths,
618+ _buildGithubRunId = githubRunId,
619+ _buildPersistenceName = persistenceName,
620+ _buildWantsIncrementalism = wantsIncrementalism,
621+ _buildEvalHost = evalHost,
622+ _buildUploadedToCache = uploadedToCache,
623+ _buildAlreadyBuilt = alreadyBuilt
624+ }
625+ )
626+
627+getRuns :: GhRepoOwner -> GhRepoName -> CommitHash -> M [Run]
628+getRuns repoOwner repoName commitHash = do
629+ pgQuery
630+ [pgSQL|
631+ SELECT id, name, repo_user, repo_name, git_commit, branch, status, req_user, start_time, end_time
632+ FROM runs
633+ WHERE git_commit = ${commitHash}
634+ AND repo_user = ${repoOwner}
635+ AND repo_name = ${repoName}
636+ |]
637+ <&> map
638+ ( \(id, name, repoOwner, repoName, gitCommit, branch, status, reqUser, startTime, endTime) ->
639+ Run
640+ { _runId = id,
641+ _runName = name,
642+ _runRepoUser = repoOwner,
643+ _runRepoName = repoName,
644+ _runGitCommit = gitCommit,
645+ _runBranch = branch,
646+ _runStatus = status,
647+ _runReqUser = reqUser,
648+ _runStartTime = startTime,
649+ _runEndTime = endTime
650+ }
651+ )
652+
653+getRun :: RunId -> M (Maybe Run)
654+getRun runId = do
655+ result <-
656+ pgQuery
657+ [pgSQL|
658+ SELECT id, name, repo_user, repo_name, git_commit, branch, status, req_user, start_time, end_time
659+ FROM runs
660+ WHERE id = ${runId}
661+ |]
662+ <&> map
663+ ( \(id, name, repoOwner, repoName, gitCommit, branch, status, reqUser, startTime, endTime) ->
664+ Run
665+ { _runId = id,
666+ _runName = name,
667+ _runRepoUser = repoOwner,
668+ _runRepoName = repoName,
669+ _runGitCommit = gitCommit,
670+ _runBranch = branch,
671+ _runStatus = status,
672+ _runReqUser = reqUser,
673+ _runStartTime = startTime,
674+ _runEndTime = endTime
675+ }
676+ )
677+ case result of
678+ [run] -> pure $ Just run
679+ [] -> pure Nothing
680+ _ -> throw $ OtherError "Impossible: more than one result"
681+
682+setRunStatus :: RunId -> Maybe Status -> M ()
683+setRunStatus runId status =
684+ void
685+ $ pgExec
686+ [pgSQL|
687+ UPDATE runs
688+ SET status = ${status},
689+ end_time = NOW()
690+ WHERE id = ${runId}
691+ |]
692+
693+newRun :: Text -> CommitInfo -> M Run
694+newRun name commitInfo = do
695+ let repoOwner = commitInfo ^. repoInfo . ghRepoOwner
696+ let repoName = commitInfo ^. repoInfo . ghRepoName
697+ let commitHash = commitInfo ^. commit
698+ let branch = commitInfo ^. Garnix.Types.branch
699+ let reqUser = commitInfo ^. Garnix.Types.reqUser
700+ result <-
701+ pgQuery
702+ [pgSQL|
703+ INSERT INTO runs
704+ (name, repo_user, repo_name, git_commit, branch, status, req_user)
705+ VALUES
706+ (${name}, ${repoOwner}, ${repoName}, ${commitHash}, ${branch}, NULL, ${reqUser})
707+ RETURNING
708+ id, name, repo_user, repo_name, git_commit, branch, status, req_user, start_time
709+ |]
710+ <&> map
711+ ( \(id, name, repoOwner, repoName, commitHash, branch, status, reqUser, startTime) ->
712+ Run
713+ { _runId = id,
714+ _runName = name,
715+ _runRepoUser = repoOwner,
716+ _runRepoName = repoName,
717+ _runGitCommit = commitHash,
718+ _runBranch = branch,
719+ _runStatus = status,
720+ _runReqUser = reqUser,
721+ _runStartTime = startTime,
722+ _runEndTime = Nothing
723+ }
724+ )
725+ case result of
726+ [x] -> pure x
727+ _ -> throw $ OtherError "newRun: Unexpected number of updates"
728+
729+-- todo remove?
730+tagCacheUpload :: GhRepoOwner -> GhRepoName -> [StorePath] -> M ()
731+tagCacheUpload repoOwner repoName =
732+ \case
733+ [] -> pure ()
734+ storePaths -> do
735+ let hashes = getHash <$> storePaths
736+ void
737+ $ pgExec
738+ [pgSQL|
739+ INSERT INTO cache_store_hashes
740+ (hash) VALUES (UNNEST(${hashes}::text[]))
741+ ON CONFLICT (hash) DO UPDATE SET accessed_at = NOW()
742+ |]
743+ void
744+ $ pgExec
745+ [pgSQL|
746+ INSERT INTO cache_store_hash_tags
747+ (hash, repo_owner, repo_name)
748+ VALUES (UNNEST(${hashes}::text[]), ${repoOwner}, ${repoName})
749+ ON CONFLICT DO NOTHING
750+ |]
751+
752+getReposForHash :: StoreHash -> M [(GhRepoOwner, GhRepoName)]
753+getReposForHash hash = do
754+ pgQuery
755+ [pgSQL|
756+ SELECT repo_owner, repo_name
757+ FROM cache_store_hash_tags
758+ WHERE hash = ${hash}
759+ |]
760+
761+data S3CacheStoreHash = S3CacheStoreHash
762+ { hash :: StoreHash,
763+ packageName :: Text,
764+ narHash :: Text,
765+ narSize :: Int64,
766+ public :: Bool,
767+ sig :: Text,
768+ references :: Text,
769+ fileSize :: Int64,
770+ fileHash :: Text
771+ }
772+ deriving (Generic, Show)
773+
774+finalizeS3CacheUpload :: S3CacheStoreHash -> M ()
775+finalizeS3CacheUpload s3CacheStoreHash = do
776+ let S3CacheStoreHash
777+ { hash,
778+ packageName,
779+ narHash,
780+ narSize,
781+ public,
782+ sig,
783+ references,
784+ fileSize,
785+ fileHash
786+ } = s3CacheStoreHash
787+ void
788+ $ pgExec
789+ [pgSQL|
790+ UPDATE cache_store_hashes
791+ SET
792+ accessed_at = NOW(),
793+ package_name = ${packageName},
794+ nar_hash = ${narHash},
795+ nar_size = ${narSize},
796+ public = ${public},
797+ sig = ${sig},
798+ "references" = ${references},
799+ file_size = ${fileSize},
800+ file_hash = ${fileHash},
801+ uploaded_at = NOW()
802+ WHERE hash = ${hash};
803+ |]
804+
805+tagCacheUploadForS3Cache :: GhRepoOwner -> GhRepoName -> StoreHash -> M ()
806+tagCacheUploadForS3Cache repoOwner repoName hash = do
807+ void
808+ $ pgExec
809+ [pgSQL|
810+ INSERT INTO cache_store_hash_tags
811+ (hash, repo_owner, repo_name)
812+ VALUES (${hash}, ${repoOwner}, ${repoName})
813+ ON CONFLICT DO NOTHING
814+ |]
815+
816+getS3CacheStoreHash :: StoreHash -> M (Maybe S3CacheStoreHash)
817+getS3CacheStoreHash hash = do
818+ result <-
819+ pgQuery
820+ [pgSQL|
821+ SELECT
822+ package_name,
823+ nar_hash,
824+ nar_size,
825+ public,
826+ sig,
827+ "references",
828+ file_size,
829+ file_hash
830+ FROM cache_store_hashes
831+ WHERE hash = ${hash}
832+ AND uploaded_at IS NOT NULL
833+ |]
834+ <&> catMaybes
835+ . fmap
836+ ( \case
837+ ( Just packageName,
838+ Just narHash,
839+ Just narSize,
840+ Just public,
841+ Just sig,
842+ Just references,
843+ Just fileSize,
844+ Just fileHash
845+ ) ->
846+ Just
847+ $ S3CacheStoreHash
848+ { hash,
849+ packageName,
850+ narHash,
851+ narSize,
852+ public,
853+ sig,
854+ references,
855+ fileSize,
856+ fileHash
857+ }
858+ _ -> Nothing
859+ )
860+ case result of
861+ [cacheStoreHash] -> pure $ Just cacheStoreHash
862+ [] -> pure Nothing
863+ _ -> throw $ OtherError "Impossible: more than one result"
864+
865+-- | Figure out what store paths you'll upload.
866+--
867+-- The argument is what store paths you want in the cache. The returned value
868+-- are the ones that are now the caller's responsibility. This is tracked in
869+-- the DB, so no one else will try uploading.
870+claimS3CachedStorePaths :: [StorePath] -> M [StorePath]
871+claimS3CachedStorePaths (sort -> storePaths) = do
872+ let hashes = fmap getHash storePaths
873+ let packageNames = fmap getName storePaths
874+ filtered :: [(Maybe StoreHash, Maybe Text)] <-
875+ pgQuery
876+ -- There are five states possible in the DB:
877+ -- - No cache entry exists
878+ -- - Old-style (non-S3) cache entry exists
879+ -- - The cache entry has been claimed, but not yet uploaded, and was
880+ -- claimed more than 10 hours ago
881+ -- - The cache entry has been claimed, but not yet uploaded
882+ -- - The cache entry has been uploaded to S3 (uploaded_at is set)
883+ --
884+ -- If it's either of the first three cases, we want to claim it. We do
885+ -- this by making sure the INSERT RETURNING returns it, which happens
886+ -- when there's an update or insert.
887+ [pgSQL|
888+ INSERT INTO cache_store_hashes (hash, package_name)
889+ SELECT hash, package_name
890+ FROM UNNEST(${hashes}::text[], ${packageNames}::text[]) AS t(hash, package_name)
891+ ON CONFLICT (hash) DO UPDATE SET
892+ accessed_at = NOW(),
893+ package_name = EXCLUDED.package_name
894+ WHERE cache_store_hashes.package_name IS NULL
895+ OR (cache_store_hashes.uploaded_at IS NULL AND
896+ cache_store_hashes.created_at < now() - interval '10 hours')
897+ RETURNING
898+ hash, package_name;
899+ |]
900+ forM filtered $ \case
901+ (Just hash, Just packageName) -> pure $ StorePath hash packageName
902+ _ -> throw $ OtherError "impossible: hashes and packageNames have the same length"
903+
904+-- * /api/account/tokens
905+
906+getAccessTokensForUser :: UserId -> M [AccessTokenMetadata]
907+getAccessTokensForUser userId = do
908+ map
909+ ( \(id, name, created_at, last_used, scope_cache, scope_api) ->
910+ let scopes =
911+ AccessTokenScopes
912+ { cache = scope_cache,
913+ api = scope_api
914+ }
915+ in AccessTokenMetadata id name created_at last_used scopes
916+ )
917+ <$> pgQuery
918+ [pgSQL|
919+ SELECT id, name, created_at, last_used, scope_cache, scope_api
920+ FROM access_tokens
921+ WHERE user_id = ${userId}
922+ |]
923+
924+getAccessTokenHashesForUser :: UserId -> M [(Int64, HashedPassword, AccessTokenScopes)]
925+getAccessTokenHashesForUser userId = do
926+ map
927+ ( \(id, token, scope_cache, scope_api) ->
928+ ( id,
929+ token,
930+ AccessTokenScopes
931+ { cache = scope_cache,
932+ api = scope_api
933+ }
934+ )
935+ )
936+ <$> pgQuery
937+ [pgSQL|
938+ SELECT id, token, scope_cache, scope_api
939+ FROM access_tokens
940+ WHERE user_id = ${userId}
941+ |]
942+
943+markAccessTokenUsed :: UserId -> Int64 -> M ()
944+markAccessTokenUsed userId tokenId = do
945+ void
946+ $ pgExec
947+ [pgSQL|
948+ UPDATE access_tokens
949+ SET last_used = NOW()
950+ WHERE id = ${tokenId}
951+ AND user_id = ${userId}
952+ |]
953+
954+insertAccessTokenForUser :: UserId -> Text -> AccessTokenScopes -> HashedPassword -> M ()
955+insertAccessTokenForUser userId name scopes tokenHash = do
956+ let cache = scopes ^. #cache
957+ let api = scopes ^. #api
958+ void
959+ $ pgExec
960+ [pgSQL|
961+ INSERT INTO access_tokens
962+ (name, token, user_id, scope_cache, scope_api)
963+ VALUES (${name}, ${tokenHash}, ${userId}, ${cache}, ${api})
964+ |]
965+
966+deleteAccessTokenForUser :: UserId -> Int64 -> M ()
967+deleteAccessTokenForUser userId tokenId = do
968+ void
969+ $ pgExec
970+ [pgSQL|
971+ DELETE FROM access_tokens
972+ WHERE id = ${tokenId}
973+ AND user_id = ${userId}
974+ |]
975+
976+-- * /api/build/commits
977+
978+getCommitsForReqUser :: User -> M [CommitSummary]
979+getCommitsForReqUser user = do
980+ map
981+ ( \( repoOwner :: GhRepoOwner,
982+ repoName :: GhRepoName,
983+ gitCommit :: CommitHash,
984+ branch :: Maybe Branch,
985+ reqUser :: GhLogin,
986+ isPublic :: Bool,
987+ startTime :: UTCTime,
988+ succeeded :: Int64,
989+ failed :: Int64,
990+ pending :: Int64,
991+ cancelled :: Int64
992+ ) ->
993+ CommitSummary repoOwner repoName (RepoIsPublic isPublic) gitCommit branch reqUser startTime succeeded failed pending cancelled
994+ )
995+ <$> pgQuery
996+ [pgSQL|!
997+ WITH
998+ -- First we collect all of the commits for the user. We do this since
999+ -- this query has an index specifically to make this fast, and all
1000+ -- future queries just operate on this or use the git_commit index.
1001+ commits_for_req_user AS (
1002+ SELECT git_commit, max(start_time) as commit_start_time FROM builds
1003+ WHERE req_user = ${user ^. githubLogin}
1004+ GROUP BY git_commit
1005+ ORDER BY commit_start_time DESC
1006+ LIMIT 100
1007+ ),
1008+
1009+ -- Now we can find all the builds we care about efficiently by joining:
1010+ all_related_builds AS (
1011+ SELECT
1012+ repo_user,
1013+ repo_name,
1014+ package_type,
1015+ system,
1016+ package,
1017+ builds.git_commit,
1018+ branch,
1019+ req_user,
1020+ repo_is_public,
1021+ builds.start_time,
1022+ status
1023+ FROM commits_for_req_user
1024+ LEFT JOIN builds
1025+ ON commits_for_req_user.git_commit = builds.git_commit
1026+ WHERE req_user = ${user ^. githubLogin}
1027+ ),
1028+
1029+ -- Now filter out re-runs using `SELECT DISTINCT`
1030+ without_reruns AS (
1031+ SELECT DISTINCT ON (git_commit, package_type, system, package) *
1032+ FROM all_related_builds
1033+ ORDER BY git_commit, package_type, system, package, start_time DESC
1034+ )
1035+
1036+ -- Finally, aggregate the status totals by git_commit
1037+ SELECT
1038+ (array_agg(repo_user))[1],
1039+ (array_agg(repo_name))[1],
1040+ git_commit,
1041+ (array_agg(branch))[1],
1042+ (array_agg(req_user))[1],
1043+ (array_agg(repo_is_public))[1],
1044+ max(start_time) as commit_start_time,
1045+ COUNT(*) FILTER (WHERE status = 'success') as succeeded,
1046+ COUNT(*) FILTER (WHERE status = 'failure' OR status = 'timeout') as failed,
1047+ COUNT(*) FILTER (WHERE status IS NULL) as pending,
1048+ COUNT(*) FILTER (WHERE status = 'cancelled') as cancelled
1049+ FROM without_reruns
1050+ GROUP BY git_commit
1051+ ORDER BY commit_start_time DESC
1052+ |]
1053+
1054+-- * /api/build/commit/{commit}
1055+
1056+getCommitSummary :: CommitHash -> M CommitSummary
1057+getCommitSummary commit = do
1058+ res <-
1059+ map
1060+ ( \( repoOwner :: GhRepoOwner,
1061+ repoName :: GhRepoName,
1062+ gitCommit :: CommitHash,
1063+ branch :: Maybe Branch,
1064+ reqUser :: GhLogin,
1065+ isPublic :: Bool,
1066+ startTime :: UTCTime,
1067+ succeeded :: Int64,
1068+ failed :: Int64,
1069+ pending :: Int64,
1070+ cancelled :: Int64
1071+ ) ->
1072+ CommitSummary repoOwner repoName (RepoIsPublic isPublic) gitCommit branch reqUser startTime succeeded failed pending cancelled
1073+ )
1074+ <$> pgQuery
1075+ [pgSQL|!
1076+ SELECT
1077+ (array_agg(repo_user))[1],
1078+ (array_agg(repo_name))[1],
1079+ git_commit,
1080+ (array_agg(branch))[1],
1081+ (array_agg(req_user))[1],
1082+ (array_agg(repo_is_public))[1],
1083+ min(start_time),
1084+ COUNT(*) FILTER (WHERE status = 'success') as succeeded,
1085+ COUNT(*) FILTER (WHERE status = 'failure' OR status = 'timeout') as failed,
1086+ COUNT(*) FILTER (WHERE status IS NULL) as pending,
1087+ COUNT(*) FILTER (WHERE status = 'cancelled') as cancelled
1088+ FROM (
1089+ SELECT DISTINCT ON (git_commit, package_type, system, package) * FROM builds
1090+ WHERE git_commit = ${commit}
1091+ ORDER BY git_commit, package_type, system, package, start_time DESC
1092+ ) AS sub
1093+ GROUP BY git_commit
1094+ |]
1095+ case res of
1096+ [r] -> pure r
1097+ [] -> throw $ NoSuchCommit commit
1098+ _ -> throw $ OtherError "Impossible: more than one result"
1099+
1100+-- * Internal stuff
1101+
1102+newBuildDB :: CommitInfo -> PackageInfo -> Text -> Bool -> M Build
1103+newBuildDB commitInfo packageInfo evalHost wantsIncrementalism = do
1104+ now <- liftIO getCurrentTime
1105+ changes <-
1106+ pgQueryPrism
1107+ _Build
1108+ [pgSQL|
1109+ INSERT INTO builds
1110+ (repo_user,
1111+ repo_name,
1112+ pr_from_fork,
1113+ branch,
1114+ repo_is_public,
1115+ git_commit,
1116+ package,
1117+ package_type,
1118+ system,
1119+ req_user,
1120+ start_time,
1121+ wants_incrementalism,
1122+ eval_host,
1123+ uploaded_to_cache
1124+ )
1125+ VALUES
1126+ (${commitInfo ^. (repoInfo . ghRepoOwner)},
1127+ ${commitInfo ^. (repoInfo . ghRepoName)},
1128+ ${commitInfo ^. prFromFork},
1129+ ${commitInfo ^. branch},
1130+ ${commitInfo ^. repoPublicity},
1131+ ${commitInfo ^. commit},
1132+ ${packageInfo ^. Garnix.Types.packageName},
1133+ ${packageInfo ^. packageType},
1134+ ${packageInfo ^. maybeSystem},
1135+ ${commitInfo ^. reqUser},
1136+ ${now},
1137+ ${wantsIncrementalism},
1138+ ${evalHost},
1139+ FALSE
1140+ )
1141+ ON CONFLICT DO NOTHING
1142+ RETURNING
1143+ id,
1144+ repo_user,
1145+ repo_name,
1146+ pr_from_fork,
1147+ branch,
1148+ repo_is_public,
1149+ git_commit,
1150+ package,
1151+ package_type,
1152+ system,
1153+ req_user,
1154+ status,
1155+ start_time,
1156+ end_time,
1157+ drv_path,
1158+ output_paths,
1159+ github_run_id,
1160+ persistence_name,
1161+ wants_incrementalism,
1162+ eval_host,
1163+ uploaded_to_cache,
1164+ already_built
1165+ |]
1166+ case changes of
1167+ [build] -> pure build
1168+ _ -> throw $ OtherError "Expected 1 column to be updated"
1169+
1170+reportBuildResultDB :: Build -> M ()
1171+reportBuildResultDB build = do
1172+ colsChanged <-
1173+ pgExec
1174+ [pgSQL|
1175+ UPDATE builds
1176+ SET status = ${build ^. status},
1177+ end_time = ${build ^. endTime},
1178+ drv_path = ${build ^. drvPath},
1179+ output_paths = ${build ^. outputPaths},
1180+ github_run_id = ${build ^. githubRunId},
1181+ persistence_name = ${build ^. persistenceName},
1182+ eval_host = ${build ^. evalHost},
1183+ already_built = ${build ^. alreadyBuilt}
1184+ WHERE id = ${build ^. id}
1185+ |]
1186+ case colsChanged of
1187+ 0 -> throw $ NoSuchBuild (build ^. id)
1188+ 1 -> pure ()
1189+ _ -> throw $ OtherError "Somehow updated more than 0 or 1 columns"
1190+
1191+-- * Servers
1192+
1193+appendToServerDeployLog :: ServerId -> Text -> M ()
1194+appendToServerDeployLog serverId logs = do
1195+ void
1196+ $ pgExec
1197+ [pgSQL|
1198+ UPDATE servers
1199+ SET deploy_logs = deploy_logs || ${logs} || E'\n'
1200+ WHERE id = ${serverId}
1201+ |]
1202+
1203+-- * Server pool
1204+
1205+newServerInPool :: ServerTier -> M PreprovisionedServerId
1206+newServerInPool tier = do
1207+ changes <-
1208+ pgQuery
1209+ [pgSQL|
1210+ INSERT INTO server_pool
1211+ ( id
1212+ , created_at
1213+ , server_tier
1214+ )
1215+ VALUES
1216+ ( DEFAULT
1217+ , NOW()
1218+ , ${tier}
1219+ )
1220+ RETURNING id
1221+ |]
1222+ case changes of
1223+ [provisionServerId] -> pure provisionServerId
1224+ _ -> throw $ OtherError "newServerInPool: Unexpected number of updates"
1225+
1226+deleteServerFromPool :: PreprovisionedServerId -> M ()
1227+deleteServerFromPool sId = do
1228+ changes <-
1229+ pgExec
1230+ [pgSQL|
1231+ DELETE FROM server_pool
1232+ WHERE id = ${sId}
1233+ |]
1234+ case changes of
1235+ 1 -> pure ()
1236+ _ -> throw $ OtherError "deleteServerFromPool: Unexpected number of updates"
1237+
1238+updatePreprovisionedServer :: PreprovisionedServer -> M ()
1239+updatePreprovisionedServer s = do
1240+ changes <-
1241+ pgExec
1242+ [pgSQL|
1243+ UPDATE server_pool
1244+ SET hetzner_id = ${s ^. hetznerServerId},
1245+ ipv4 = ${s ^. ipv4Addr},
1246+ ipv6 = ${s ^. ipv6Addr},
1247+ ready_at = ${s ^. readyAt}
1248+ WHERE id = ${s ^. id}
1249+ |]
1250+ case changes of
1251+ 1 -> pure ()
1252+ _ -> throw $ OtherError "updatePreprovisionedServer: Unexpected number of updates"
1253+
1254+setPreprovisionedReady :: PreprovisionedServerId -> M ()
1255+setPreprovisionedReady pId = do
1256+ changes <-
1257+ pgExec
1258+ [pgSQL|
1259+ UPDATE server_pool
1260+ SET ready_at = NOW()
1261+ WHERE id = ${pId}
1262+ |]
1263+ case changes of
1264+ 1 -> pure ()
1265+ _ -> throw $ OtherError "setPreprovisionedReady: Unexpected number of updates"
1266+
1267+getPreprovisionedServerCount :: ServerTier -> M Int64
1268+getPreprovisionedServerCount tier = do
1269+ mcount <- pgQuery [pgSQL| SELECT COUNT(*) FROM server_pool WHERE server_tier = ${tier} |]
1270+ case mcount of
1271+ [Just c] -> pure c
1272+ _ -> throw $ OtherError "getPreprovisionedServerCount: Unexpected return type"
1273+
1274+-- Claim a preprovisioned server, if there is one.
1275+claimServerDB :: ServerToSpinUp -> Maybe GhPullRequestId -> M (Maybe ServerInfo)
1276+claimServerDB serverToSpinUp pullRequest =
1277+ pgTransaction $ do
1278+ let serverTier = serverToSpinUp ^. #serverTier
1279+ claimed <-
1280+ pgQuery
1281+ [pgSQL|
1282+ DELETE FROM server_pool
1283+ WHERE hetzner_id IN (
1284+ SELECT hetzner_id
1285+ FROM server_pool
1286+ WHERE
1287+ ready_at IS NOT NULL AND
1288+ server_tier = ${serverTier}
1289+ ORDER BY ready_at ASC
1290+ LIMIT 1
1291+ )
1292+ RETURNING hetzner_id, ipv4, ipv6, server_tier
1293+ |]
1294+ case claimed of
1295+ [] -> pure Nothing
1296+ [(Just hetznerId, Just ipv4, Just ipv6, tier)] -> Just <$> newServer hetznerId ipv4 ipv6 tier
1297+ [_] -> throw $ OtherError "Impossible: hetzner_id, ipv4 or ipv6 is NULL"
1298+ _ : _ -> throw $ OtherError "Impossible: LIMIT 1 returned more than 1"
1299+ where
1300+ newServer :: HetznerServerId -> Text -> Text -> ServerTier -> M ServerInfo
1301+ newServer hetznerId ipv4 ipv6 tier = do
1302+ let buildId = serverToSpinUp ^. #build . id
1303+ let domainIsPrimary = serverToSpinUp ^. #domainIsPrimary
1304+ changes <-
1305+ pgQueryPrism
1306+ _ServerInfo
1307+ [pgSQL|
1308+ INSERT INTO servers
1309+ ( id
1310+ , configuration_build_id
1311+ , created_at
1312+ , deploy_logs
1313+ , pull_request
1314+ , hetzner_id
1315+ , ipv4
1316+ , ipv6
1317+ , server_tier
1318+ , is_primary
1319+ )
1320+ VALUES
1321+ ( DEFAULT
1322+ , ${buildId}
1323+ , NOW()
1324+ , ''
1325+ , ${pullRequest}
1326+ , ${hetznerId}
1327+ , ${ipv4}
1328+ , ${ipv6}
1329+ , ${tier}
1330+ , ${domainIsPrimary}
1331+ )
1332+ RETURNING
1333+ id,
1334+ hetzner_id,
1335+ ipv4,
1336+ ipv6,
1337+ created_at,
1338+ ended_at,
1339+ configuration_build_id,
1340+ pull_request,
1341+ ready_at,
1342+ (SELECT persistence_name
1343+ FROM builds
1344+ WHERE id = ${buildId}
1345+ LIMIT 1),
1346+ server_tier,
1347+ is_primary
1348+ |]
1349+ case changes of
1350+ [serverInfo] -> pure serverInfo
1351+ _ -> throw $ OtherError "claimServerDB: Unexpected number of updated"
1352+
1353+updateServerPostDeploy :: ServerInfo -> M ()
1354+updateServerPostDeploy s = do
1355+ changes <-
1356+ pgExec
1357+ [pgSQL|
1358+ UPDATE servers
1359+ SET hetzner_id = ${s ^. hetznerServerId},
1360+ configuration_build_id = ${s ^. configurationBuildId},
1361+ ipv4 = ${s ^. ipv4Addr},
1362+ ipv6 = ${s ^. ipv6Addr},
1363+ ended_at = ${s ^. endedAt},
1364+ ready_at = ${s ^. readyAt}
1365+ WHERE id = ${s ^. id}
1366+ |]
1367+ case changes of
1368+ 0 -> throw $ OtherError "updateServer: No such server"
1369+ 1 -> pure ()
1370+ _ -> throw $ OtherError "updateServer: Unexpected number of updated"
1371+
1372+deleteServerDB :: ServerId -> M ()
1373+deleteServerDB s = do
1374+ void
1375+ $ pgExec
1376+ [pgSQL|
1377+ UPDATE servers
1378+ SET ended_at = NOW()
1379+ WHERE id = ${s}
1380+ |]
1381+
1382+upsertHeartbeat :: [Text] -> M ()
1383+upsertHeartbeat hosts =
1384+ forM_ hosts $ \host -> do
1385+ pgQuery
1386+ [pgSQL|
1387+ INSERT INTO heartbeat
1388+ (hostname, last_heartbeat)
1389+ VALUES (${host}, NOW())
1390+ ON CONFLICT (hostname) DO UPDATE set last_heartbeat = NOW()
1391+ |]
1392+
1393+getRecentHeartbeats :: M [Text]
1394+getRecentHeartbeats =
1395+ pgQuery
1396+ [pgSQL|
1397+ SELECT hostname
1398+ FROM heartbeat
1399+ WHERE NOW() - last_heartbeat < interval '12 hours'
1400+ |]
1401+
1402+getShutdownCandidates :: M PrHostList
1403+getShutdownCandidates = do
1404+ PrHostList
1405+ <$> pgQueryPrism
1406+ _Host
1407+ [pgSQL|!
1408+ SELECT
1409+ builds.repo_user,
1410+ builds.repo_name,
1411+ builds.branch,
1412+ builds.package,
1413+ servers.pull_request,
1414+ servers.ipv4,
1415+ servers.ipv6,
1416+ builds.drv_path,
1417+ builds.persistence_name,
1418+ servers.id,
1419+ servers.hetzner_id,
1420+ servers.is_primary
1421+ FROM servers
1422+ INNER JOIN builds
1423+ ON servers.configuration_build_id = builds.id
1424+ WHERE servers.ipv4 IS NOT NULL
1425+ AND servers.pull_request is NOT NULL
1426+ AND (servers.ready_at IS NOT NULL
1427+ AND NOW() - servers.ready_at > interval '12 hours')
1428+ AND servers.ended_at IS NULL
1429+ |]
1430+
1431+getAllRunningHosts :: M [Host]
1432+getAllRunningHosts = do
1433+ pgQueryPrism
1434+ _Host
1435+ [pgSQL|!
1436+ SELECT
1437+ builds.repo_user,
1438+ builds.repo_name,
1439+ builds.branch,
1440+ builds.package,
1441+ servers.pull_request,
1442+ servers.ipv4,
1443+ servers.ipv6,
1444+ builds.drv_path,
1445+ builds.persistence_name,
1446+ servers.id,
1447+ servers.hetzner_id,
1448+ servers.is_primary
1449+ FROM servers
1450+ INNER JOIN builds
1451+ ON servers.configuration_build_id = builds.id
1452+ WHERE servers.ipv4 IS NOT NULL
1453+ AND servers.ready_at IS NOT NULL
1454+ AND servers.ended_at IS NULL
1455+ |]
1456+
1457+getRunningServersOf :: RepoInfo -> DeploymentType -> M [ServerInfo]
1458+getRunningServersOf repoInfo deploymentType = do
1459+ pgQueryPrism _ServerInfo $ case deploymentType of
1460+ BranchDeployment branch ->
1461+ [pgSQL|
1462+ SELECT
1463+ servers.id,
1464+ servers.hetzner_id,
1465+ servers.ipv4,
1466+ servers.ipv6,
1467+ servers.created_at,
1468+ servers.ended_at,
1469+ servers.configuration_build_id,
1470+ servers.pull_request,
1471+ servers.ready_at,
1472+ builds.persistence_name,
1473+ servers.server_tier,
1474+ servers.is_primary
1475+ FROM servers
1476+ INNER JOIN builds
1477+ ON servers.configuration_build_id = builds.id
1478+ WHERE builds.repo_user = ${repoInfo ^. ghRepoOwner}
1479+ AND builds.repo_name = ${repoInfo ^. ghRepoName}
1480+ AND builds.branch = ${branch}
1481+ AND servers.pull_request is NULL
1482+ AND servers.ended_at IS NULL
1483+ |]
1484+ GhPrDeployment prId ->
1485+ [pgSQL|
1486+ SELECT
1487+ servers.id,
1488+ servers.hetzner_id,
1489+ servers.ipv4,
1490+ servers.ipv6,
1491+ servers.created_at,
1492+ servers.ended_at,
1493+ servers.configuration_build_id,
1494+ servers.pull_request,
1495+ servers.ready_at,
1496+ builds.persistence_name,
1497+ servers.server_tier,
1498+ servers.is_primary
1499+ FROM servers
1500+ INNER JOIN builds
1501+ ON servers.configuration_build_id = builds.id
1502+ WHERE builds.repo_user = ${repoInfo ^. ghRepoOwner}
1503+ AND builds.repo_name = ${repoInfo ^. ghRepoName}
1504+ AND servers.ended_at IS NULL
1505+ AND servers.pull_request = ${prId}
1506+ |]
1507+
1508+getHetznerServerById :: [GhRepoOwner] -> ServerId -> M (Maybe HetznerServerId)
1509+getHetznerServerById owner serverId = do
1510+ res <-
1511+ pgQuery
1512+ [pgSQL|
1513+ SELECT hetzner_id
1514+ FROM servers
1515+ INNER JOIN builds
1516+ ON servers.configuration_build_id = builds.id
1517+ WHERE builds.repo_user = ANY(${owner})
1518+ AND servers.id = ${serverId}
1519+ AND ended_at is null;
1520+ |]
1521+ case res of
1522+ [Just hetznerId] -> pure $ Just hetznerId
1523+ [] -> pure Nothing
1524+ _ -> throw $ OtherError "Impossible: more than one result"
1525+
1526+getPrDeployDurationForOwner :: GhRepoOwner -> M Duration
1527+getPrDeployDurationForOwner owner = do
1528+ res <-
1529+ pgQuery
1530+ [pgSQL|
1531+ SELECT
1532+ SUM(date_part('EPOCH',
1533+ ( COALESCE(servers.ended_at, NOW()) -
1534+ GREATEST(servers.ready_at, date_trunc('month', NOW(), 'UTC'))
1535+ )
1536+ )) as _server_seconds
1537+ FROM servers
1538+ INNER JOIN builds
1539+ ON servers.configuration_build_id = builds.id
1540+ WHERE builds.repo_user = ${owner}
1541+ AND servers.pull_request IS NOT NULL
1542+ AND servers.ready_at IS NOT NULL
1543+ AND (servers.ended_at IS NULL OR
1544+ servers.ended_at >= date_trunc('month', NOW(), 'UTC'));
1545+ |]
1546+ case res of
1547+ [Just serverSeconds] -> pure $ fromSeconds @Double serverSeconds
1548+ [Nothing] -> pure emptyDuration
1549+ _ -> throw $ OtherError "Impossible: more than one result"
1550+
1551+getRunningBranchServersForOwner :: GhRepoOwner -> M (Map ServerTier Int64)
1552+getRunningBranchServersForOwner owner = do
1553+ res <-
1554+ pgQuery
1555+ [pgSQL|
1556+ SELECT
1557+ server_tier,
1558+ COUNT(*) AS count
1559+ FROM servers
1560+ INNER JOIN builds
1561+ ON servers.configuration_build_id = builds.id
1562+ WHERE builds.repo_user = ${owner}
1563+ AND servers.ready_at IS NOT NULL
1564+ AND servers.ended_at IS NULL
1565+ AND servers.pull_request is NULL
1566+ GROUP BY servers.server_tier
1567+ |]
1568+ pairs <- forM res $ \case
1569+ (serverTier, Just numberOfServers) -> pure (serverTier, numberOfServers)
1570+ (_, Nothing) -> throw $ OtherError "Impossible: non-numeric COUNT"
1571+ pure $ Map.fromList pairs
1572+
1573+getCurrentMonthUsage :: GhRepoOwner -> M Duration
1574+getCurrentMonthUsage owner = do
1575+ fromMaybe emptyDuration . Map.lookup owner <$> getCurrentMonthUsages [owner]
1576+
1577+getCurrentMonthUsages ::
1578+ [GhRepoOwner] ->
1579+ M (Map GhRepoOwner Duration)
1580+getCurrentMonthUsages owners = do
1581+ fromList
1582+ . map (\(repoOwner :: GhRepoOwner, seconds :: Maybe Double) -> (repoOwner, fromSeconds $ fromMaybe 0 seconds))
1583+ <$> pgQuery
1584+ [pgSQL|
1585+ SELECT
1586+ repo_user,
1587+ SUM(LEAST(120 * 60, date_part('EPOCH', (end_time - start_time)))) AS total_build_time
1588+ FROM builds
1589+ LEFT JOIN installations ON installations.repo_owner = builds.repo_user
1590+ WHERE repo_user = ANY(${owners})
1591+ AND end_time >=
1592+ CASE
1593+ -- if we're within the period, use start date
1594+ WHEN current_period_end > NOW() THEN current_period_start
1595+ -- if we don't have a period, use monthly cycles
1596+ WHEN current_period_end IS NULL then date_trunc('month', NOW())
1597+ -- otherwise, start from the end of the last period
1598+ ELSE current_period_end
1599+ END
1600+ AND comped = false
1601+ GROUP BY repo_user
1602+ |]
1603+
1604+updatePeriodForCustomer :: CustomerId -> UTCTime -> UTCTime -> M ()
1605+updatePeriodForCustomer (CustomerId customerId) startDate endDate =
1606+ void
1607+ $ pgExec
1608+ [pgSQL|
1609+ UPDATE installations
1610+ SET current_period_start = ${startDate},
1611+ current_period_end = ${endDate}
1612+ WHERE
1613+ stripe_customer = ${customerId}
1614+ |]
1615+
1616+getRepoKeyDB :: GhRepoOwner -> GhRepoName -> M (Maybe (PublicKey, PrivateKey))
1617+getRepoKeyDB owner name = do
1618+ results <-
1619+ pgQuery
1620+ [pgSQL|
1621+ SELECT public_key, private_key
1622+ FROM repo_secrets
1623+ WHERE repo_user = ${owner}
1624+ AND repo_name = ${name}
1625+ |]
1626+ case results of
1627+ [result] -> pure $ Just result
1628+ [] -> pure Nothing
1629+ _ -> throw $ OtherError "Impossible: Got more than 1 result from getRepoKeyDB"
1630+
1631+-- | In case of conflict, we return the key already in the DB, to prevent
1632+-- overwriting
1633+setRepoKeyDB ::
1634+ GhRepoOwner ->
1635+ GhRepoName ->
1636+ Candidate PublicKey ->
1637+ Candidate PrivateKey ->
1638+ M (PublicKey, PrivateKey)
1639+setRepoKeyDB owner name (Candidate pub) (Candidate priv) = do
1640+ void
1641+ $ pgQuery
1642+ [pgSQL|
1643+ INSERT INTO repo_secrets
1644+ ( repo_user,
1645+ repo_name,
1646+ public_key,
1647+ private_key
1648+ )
1649+ VALUES
1650+ ( ${owner},
1651+ ${name},
1652+ ${pub},
1653+ ${priv}
1654+ )
1655+ ON CONFLICT DO NOTHING
1656+ |]
1657+ getRepoKeyDB owner name >>= \case
1658+ Nothing -> throw $ OtherError "Impossible setRepoKeyDB: expected a set key"
1659+ Just v -> pure v
1660+
1661+getActionKeyDB :: GhRepoOwner -> GhRepoName -> PackageName -> M (Maybe (PublicKey, PrivateKey))
1662+getActionKeyDB owner name action = do
1663+ results <-
1664+ pgQuery
1665+ [pgSQL|
1666+ SELECT public_key, private_key
1667+ FROM action_secrets
1668+ WHERE repo_user = ${owner}
1669+ AND repo_name = ${name}
1670+ AND action_name = ${action}
1671+ |]
1672+ case results of
1673+ [result] -> pure $ Just result
1674+ [] -> pure Nothing
1675+ _ -> throw $ OtherError "Impossible: Got more than 1 result from getRepoKeyDB"
1676+
1677+-- | In case of conflict, we return the key already in the DB, to prevent
1678+-- overwriting
1679+setActionKeyDB ::
1680+ GhRepoOwner ->
1681+ GhRepoName ->
1682+ PackageName ->
1683+ Candidate PublicKey ->
1684+ Candidate PrivateKey ->
1685+ M (PublicKey, PrivateKey)
1686+setActionKeyDB owner name action (Candidate pub) (Candidate priv) = do
1687+ void
1688+ $ pgQuery
1689+ [pgSQL|
1690+ INSERT INTO action_secrets
1691+ ( repo_user,
1692+ repo_name,
1693+ action_name,
1694+ public_key,
1695+ private_key
1696+ )
1697+ VALUES
1698+ ( ${owner},
1699+ ${name},
1700+ ${action},
1701+ ${pub},
1702+ ${priv}
1703+ )
1704+ ON CONFLICT DO NOTHING
1705+ |]
1706+ getActionKeyDB owner name action >>= \case
1707+ Nothing -> throw $ OtherError "Impossible setRepoKeyDB: expected a set key"
1708+ Just v -> pure v
1709+
1710+isDenylisted :: GhRepoOwner -> GhRepoName -> M Bool
1711+isDenylisted owner name = do
1712+ result :: [Text] <-
1713+ pgQuery
1714+ [pgSQL|
1715+ SELECT repo_user
1716+ FROM denylist
1717+ WHERE repo_user = ${owner}
1718+ AND (repo_name = ${name} OR repo_name IS NULL)
1719+ |]
1720+ pure $ not $ null result
1721+
1722+addToWaitlist :: Email -> M ()
1723+addToWaitlist email = do
1724+ void
1725+ $ pgExec
1726+ [pgSQL|
1727+ INSERT INTO waitlist
1728+ ( email )
1729+ VALUES
1730+ ( ${email} )
1731+ ON CONFLICT DO NOTHING
1732+ |]
1733+
1734+-- * Installations
1735+
1736+getRepoOwnerForStripeCustomer :: CustomerId -> M (Maybe GhRepoOwner)
1737+getRepoOwnerForStripeCustomer customer = do
1738+ res :: [Maybe GhRepoOwner] <-
1739+ pgQuery
1740+ [pgSQL|
1741+ SELECT repo_owner
1742+ FROM installations
1743+ WHERE stripe_customer = ${getCustomerId customer}
1744+ |]
1745+ case res of
1746+ [Just owner] -> pure $ Just owner
1747+ [Nothing] -> pure Nothing
1748+ [] -> pure Nothing
1749+ _ : _ : _ -> throw $ OtherError "impossible: stripe_customer is unique"
1750+
1751+getInstallationStripeCustomer :: GhRepoOwner -> M (Maybe CustomerId)
1752+getInstallationStripeCustomer repoOwner = do
1753+ res :: [Maybe Text] <-
1754+ pgQuery
1755+ [pgSQL|
1756+ SELECT stripe_customer
1757+ FROM installations
1758+ WHERE repo_owner = ${repoOwner}
1759+ |]
1760+ case res of
1761+ [Just id] -> pure $ Just $ CustomerId id
1762+ [Nothing] -> pure Nothing
1763+ [] -> pure Nothing
1764+ _ : _ : _ -> throw $ OtherError "impossible: repo_owner is unique"
1765+
1766+setStripeCustomerId :: GhRepoOwner -> CustomerId -> M ()
1767+setStripeCustomerId repoOwner stripeCustomerId = do
1768+ void
1769+ $ pgExec
1770+ [pgSQL|
1771+ INSERT INTO installations
1772+ (repo_owner, stripe_customer) VALUES
1773+ (${repoOwner}, ${getCustomerId stripeCustomerId})
1774+ |]
1775+
1776+setRequestedCancellation :: GhRepoOwner -> Bool -> M ()
1777+setRequestedCancellation repoOwner requestedCancelation = do
1778+ void
1779+ $ pgExec
1780+ [pgSQL|
1781+ UPDATE installations
1782+ SET requested_cancellation = ${requestedCancelation}
1783+ WHERE repo_owner = ${repoOwner}
1784+ |]
1785+
1786+getInstallationStatus :: GhRepoOwner -> M InstallationStatus
1787+getInstallationStatus repoOwner = do
1788+ res :: [(Maybe UTCTime, Bool)] <-
1789+ pgQuery
1790+ [pgSQL|
1791+ SELECT current_period_end, requested_cancellation
1792+ FROM installations
1793+ WHERE repo_owner = ${repoOwner}
1794+ |]
1795+ case res of
1796+ [] -> pure NoActiveInstallation
1797+ [(Nothing, _)] -> pure NoActiveInstallation
1798+ [(Just endDate, False)] -> pure $ InstallationRenewing endDate
1799+ [(Just endDate, True)] -> pure $ InstallationCancelling endDate
1800+ _ : _ : _ -> throw $ OtherError "impossible: repo_owner is unique"
1801+
1802+getIncrementalTarget :: Build -> [CommitHash] -> M [Build]
1803+getIncrementalTarget build commits =
1804+ pgQueryPrism
1805+ _Build
1806+ [pgSQL|
1807+ SELECT
1808+ DISTINCT ON
1809+ (repo_user,
1810+ repo_name,
1811+ package,
1812+ package_type,
1813+ system,
1814+ git_commit
1815+ )
1816+ id,
1817+ repo_user,
1818+ repo_name,
1819+ pr_from_fork,
1820+ branch,
1821+ repo_is_public,
1822+ git_commit,
1823+ package,
1824+ package_type,
1825+ system,
1826+ req_user,
1827+ status,
1828+ start_time,
1829+ end_time,
1830+ drv_path,
1831+ output_paths,
1832+ github_run_id,
1833+ persistence_name,
1834+ wants_incrementalism,
1835+ eval_host,
1836+ uploaded_to_cache,
1837+ already_built
1838+ FROM builds
1839+ WHERE git_commit =
1840+ (SELECT
1841+ git_commit
1842+ FROM builds
1843+ WHERE git_commit = ANY(${commits}::text[])
1844+ GROUP BY git_commit
1845+ HAVING
1846+ bool_and(CASE WHEN end_time IS NULL THEN FALSE else TRUE END)
1847+ ORDER BY ARRAY_POSITION(${commits}, git_commit)
1848+ LIMIT 1)
1849+ AND repo_user = ${build ^. repoUser}
1850+ AND repo_name = ${build ^. repoName}
1851+ ORDER BY
1852+ repo_user,
1853+ repo_name,
1854+ package,
1855+ package_type,
1856+ system,
1857+ git_commit,
1858+ end_time ASC
1859+ |]
1860+
1861+-- * Health
1862+
1863+checkHealth :: M ()
1864+checkHealth = do
1865+ res <- (pgQuery [pgSQL|SELECT 1|] :: M [Maybe Int32])
1866+ case res of
1867+ [Just 1] -> pure ()
1868+ _ -> throw $ OtherError "DB Health check failed"
1869+
1870+-- * Tokens
1871+
1872+getUserInternalToken :: GhLogin -> M InternalCacheToken
1873+getUserInternalToken reqUser =
1874+ maybeGetDbToken >>= \case
1875+ Just token -> pure token
1876+ Nothing -> generateInternalCacheToken >>= insertUserToken
1877+ where
1878+ maybeGetDbToken :: M (Maybe InternalCacheToken)
1879+ maybeGetDbToken =
1880+ pgQuery
1881+ [pgSQL|
1882+ SELECT internal_token
1883+ FROM internal_access_tokens
1884+ WHERE github_login = ${reqUser}
1885+ |]
1886+ >>= \case
1887+ [] -> pure Nothing
1888+ [token] -> pure $ Just $ InternalCacheToken token
1889+ _ -> throw $ OtherError "getUserInternalToken/get: internal token should be unique"
1890+
1891+ insertUserToken :: InternalCacheToken -> M InternalCacheToken
1892+ insertUserToken (InternalCacheToken rawToken) =
1893+ pgQuery
1894+ [pgSQL|
1895+ INSERT INTO internal_access_tokens
1896+ (github_login, internal_token)
1897+ VALUES (${reqUser}, ${rawToken})
1898+ ON CONFLICT DO NOTHING
1899+ RETURNING internal_token
1900+ |]
1901+ >>= \case
1902+ [] -> throw $ OtherError "getUserInternalToken/insert: no token returned"
1903+ [token] -> pure $ InternalCacheToken token
1904+ _ -> throw $ OtherError "getUserInternalToken/insert: internal token should be unique"
1905+
1906+addVerifiedFod :: DrvPath -> StorePath -> M ()
1907+addVerifiedFod drvPath storePath = do
1908+ void
1909+ $ pgExec
1910+ [pgSQL|
1911+ INSERT INTO verified_fods
1912+ (drv_hash, store_path_hash)
1913+ VALUES
1914+ (${Nix.getHash $ Nix.getDrvPath drvPath}, ${Nix.getHash storePath})
1915+ ON CONFLICT DO NOTHING
1916+ |]
1917+
1918+keepUnverifiedFods :: Set (DrvPath, a) -> M (Set (DrvPath, a))
1919+keepUnverifiedFods drvPaths = do
1920+ let drvHashes :: [Text] = fmap (cs . getHash . getDrvPath . fst) $ Set.toList drvPaths
1921+ -- Use a Set for better asymptotics
1922+ verifiedResults :: Set StoreHash <-
1923+ Set.fromList
1924+ <$> pgQuery
1925+ [pgSQL|
1926+ SELECT drv_hash FROM verified_fods
1927+ WHERE drv_hash = ANY(${drvHashes}::text[])
1928+ |]
1929+ pure $ drvPaths
1930+ & Set.filter (\(drv, _a) -> getHash (getDrvPath drv) `Set.notMember` verifiedResults)
1931+
1932+-- * Helpers
1933+
1934+getDBConnection :: [Data.ByteString.ByteString] -> IO PGConnection
1935+getDBConnection dbPasswords = do
1936+ case dbPasswords of
1937+ [] -> error "No database passwords provided"
1938+ [dbPass] -> do
1939+ db <- getTPGDatabase
1940+ pgConnect $ db {pgDBPass = dbPass}
1941+ dbPass : rest -> do
1942+ db <- getTPGDatabase
1943+ result <- Control.Exception.Safe.try $ pgConnect $ db {pgDBPass = dbPass}
1944+ case result of
1945+ Left (e :: PSQLP.PGError) -> do
1946+ hPutStrLn stderr $ "error connecting to the DB with one (of multiple) passwords: " <> show e
1947+ hPutStrLn stderr "trying other passwords now"
1948+ getDBConnection rest
1949+ Right conn -> pure conn
1950+
1951+pgQueryPrism :: (PGQuery q a) => Prism' x a -> q -> M [x]
1952+pgQueryPrism p q = withConnection $ \conn -> timingAs #dbQueryTime $ do
1953+ incrementEvent #dbQueries
1954+ let queryStr = cs $ getQueryString unknownPGTypeEnv q
1955+ res <-
1956+ liftIO (PSQL.pgQuery conn q)
1957+ `catchIOError` (throw . DbError queryStr . cs . show)
1958+ pure $ res ^.. traverse . re p
1959+
1960+pgQuery :: (PGQuery q a) => q -> M [a]
1961+pgQuery q = withConnection $ \conn -> timingAs #dbQueryTime $ do
1962+ incrementEvent #dbQueries
1963+ let queryStr = cs $ getQueryString unknownPGTypeEnv q
1964+ liftIO (PSQL.pgQuery conn q)
1965+ `catchIOError` (throw . DbError queryStr . cs . show)
1966+
1967+pgTransaction :: M a -> M a
1968+pgTransaction inner = do
1969+ view #dbConn >>= \case
1970+ Transaction _ -> throw TransactionAlreadyStarted
1971+ ConnectionPool pool -> do
1972+ liftBaseOp (withResource pool)
1973+ $ \conn ->
1974+ liftBaseOp_ (Safe.bracketOnError_ (PSQLP.pgBegin conn) (PSQLP.pgRollback conn)) $ do
1975+ result <- try $ local (#dbConn .~ Transaction conn) inner
1976+ case result of
1977+ Left e -> do
1978+ liftIO $ PSQLP.pgRollback conn
1979+ throw $ err e
1980+ Right r -> do
1981+ liftIO $ PSQLP.pgCommit conn
1982+ pure r
1983+
1984+pgExec :: (PGQuery q ()) => q -> M Int
1985+pgExec q = withConnection $ \conn -> timingAs #dbQueryTime $ do
1986+ incrementEvent #dbQueries
1987+ let queryStr = cs $ getQueryString unknownPGTypeEnv q
1988+ liftIO (PSQL.pgExecute conn q)
1989+ `catchIOError` (throw . DbError queryStr . cs . show)
1990+
1991+withConnection :: (PGConnection -> M a) -> M a
1992+withConnection act = do
1993+ view #dbConn >>= \case
1994+ Transaction conn -> act conn
1995+ ConnectionPool pool -> liftBaseOp (withResource pool) act
backend/src/Garnix/DB/FeatureFlags.hsnew file+51
@@ -0,0 +1,51 @@
1+module Garnix.DB.FeatureFlags where
2+
3+import Control.Concurrent.Lifted (modifyMVar)
4+import Data.Aeson qualified as Aeson
5+import Data.Aeson.Types qualified as Aeson
6+import Database.PostgreSQL.Typed (pgSQL)
7+import Garnix.DB qualified as DB
8+import Garnix.DB.FeatureFlags.Types
9+import Garnix.Monad
10+import Garnix.Prelude
11+import Garnix.Types
12+import System.Random (randomRIO)
13+
14+withRecachedFeatureFlags :: M a -> M a
15+withRecachedFeatureFlags action = do
16+ newConfig <- liftIO getFeatureFlagConfig
17+ local (#featureFlagConfig .~ newConfig) $ do
18+ action
19+
20+isFeatureOn :: FeatureId -> M Bool
21+isFeatureOn id = do
22+ FeatureFlagConfigDbo config <- do
23+ FeatureFlagConfig mvar <- view #featureFlagConfig
24+ modifyMVar mvar $ \case
25+ Just config -> pure (Just config, config)
26+ Nothing -> do
27+ value :: [Aeson.Value] <-
28+ DB.pgQuery
29+ [pgSQL|
30+ SELECT config FROM feature_flags ORDER BY created_at DESC LIMIT 1;
31+ |]
32+ case value of
33+ [] -> pure def
34+ [json] -> case Aeson.parseEither parseJSON json of
35+ Right (config :: FeatureFlagConfigDbo) -> pure (Just config, config)
36+ Left e -> do
37+ log Critical $ "invalid json in feature_flags table: " <> cs e
38+ pure (Nothing, def)
39+ _ -> throw $ OtherError "impossible: more than two configs returned"
40+ case lookup id config of
41+ Nothing -> pure False
42+ Just config -> case config of
43+ Percentage percentage -> do
44+ random <- randomRIO (0, 99)
45+ pure $ random < percentage
46+ DisableFeature -> pure False
47+
48+whenFeature :: FeatureId -> M () -> M ()
49+whenFeature id action = do
50+ enabled <- isFeatureOn id
51+ when enabled action
backend/src/Garnix/DB/FeatureFlags/Types.hsnew file+62
@@ -0,0 +1,62 @@
1+module Garnix.DB.FeatureFlags.Types
2+ ( FeatureFlagConfig (..),
3+ FeatureFlagConfigDbo (..),
4+ FeatureId (..),
5+ FeatureConfig (..),
6+ getFeatureFlagConfig,
7+ )
8+where
9+
10+import Control.Concurrent (MVar, newMVar)
11+import Data.Aeson qualified as Aeson
12+import Data.OpenApi (ToSchema)
13+import Garnix.Prelude
14+
15+newtype FeatureFlagConfig = FeatureFlagConfig (MVar (Maybe FeatureFlagConfigDbo))
16+
17+newtype FeatureFlagConfigDbo = FeatureFlagConfigDbo [(FeatureId, FeatureConfig)]
18+ deriving stock (Generic, Show, Eq)
19+ deriving newtype (ToJSON, FromJSON)
20+
21+instance Default FeatureFlagConfigDbo where
22+ def = FeatureFlagConfigDbo mempty
23+
24+instance PGColumn "json" FeatureFlagConfigDbo where
25+ pgDecode _ =
26+ either (\message -> error $ "Cannot decode FeatureFlagConfig json value: " <> cs message) identity
27+ . Aeson.eitherDecode @FeatureFlagConfigDbo
28+ . cs
29+
30+instance ToSchema FeatureFlagConfigDbo
31+
32+data FeatureId
33+ = FodChecks
34+ | BwrapBuildSandbox
35+ deriving (Generic, Ord, Eq, Show)
36+
37+instance ToJSON FeatureId
38+
39+instance ToJSONKey FeatureId where
40+ toJSONKey = Aeson.genericToJSONKey Aeson.defaultJSONKeyOptions
41+
42+instance FromJSON FeatureId
43+
44+instance FromJSONKey FeatureId where
45+ fromJSONKey = Aeson.genericFromJSONKey Aeson.defaultJSONKeyOptions
46+
47+instance ToSchema FeatureId
48+
49+data FeatureConfig
50+ = Percentage Int
51+ | DisableFeature
52+ deriving (Generic, Show, Eq)
53+
54+instance ToJSON FeatureConfig
55+
56+instance FromJSON FeatureConfig
57+
58+instance ToSchema FeatureConfig
59+
60+getFeatureFlagConfig :: IO FeatureFlagConfig
61+getFeatureFlagConfig = do
62+ FeatureFlagConfig <$> newMVar Nothing
backend/src/Garnix/DB/ModuleValues.hsnew file+330
@@ -0,0 +1,330 @@
1+module Garnix.DB.ModuleValues where
2+
3+import Data.Aeson ((.:), (.=))
4+import Data.Aeson qualified as JSON
5+import Data.Aeson.Key qualified as JSON
6+import Data.Aeson.KeyMap qualified as JSON
7+import Data.Functor ((<&>))
8+import Data.Map.Strict (Map)
9+import Data.Row (Rec, (.+), (.==), type (.+), type (.==))
10+import Database.PostgreSQL.Typed (pgSQL)
11+import Garnix.DB qualified as DB
12+import Garnix.Monad
13+import Garnix.Prelude hiding (get)
14+import Garnix.Types
15+
16+type GetRepoAndModuleValues =
17+ Rec
18+ ( "repo_user" .== Maybe GhRepoOwner
19+ .+ "repo_name" .== Maybe GhRepoName
20+ .+ "user_config" .== [ModuleValue]
21+ .+ "modules" .== [Module]
22+ )
23+
24+type UpdateRepoModuleValues =
25+ Rec
26+ ( "repo_user" .== Maybe GhRepoOwner
27+ .+ "repo_name" .== Maybe GhRepoName
28+ .+ "user_config" .== [ModuleValue]
29+ )
30+
31+type ModuleValue =
32+ Rec
33+ ( "module_name" .== Text
34+ .+ "git_commit" .== Maybe CommitHash
35+ .+ "values" .== ModuleConfig
36+ )
37+
38+type Module =
39+ Rec
40+ ( "name" .== Text
41+ .+ "repo_user" .== GhRepoOwner
42+ .+ "repo_name" .== GhRepoName
43+ .+ "git_commit" .== CommitHash
44+ .+ "schema" .== JSON.Value
45+ .+ "description" .== Maybe Text
46+ )
47+
48+data ModuleConfig = ModuleConfig NixIdentifier NixValue
49+ deriving stock (Generic)
50+
51+instance FromJSON ModuleConfig where
52+ parseJSON = JSON.withObject "ModuleConfig" $ \o -> do
53+ tag <- o .: "tag"
54+ when (tag /= "set") $ fail $ "expected 'set' tag at the top level, got: " <> tag
55+ obj <- o .: "value"
56+ case JSON.toList obj of
57+ [(identifier, value)] -> do
58+ nixValue <- parseJSON value
59+ pure $ ModuleConfig (NixIdentifier $ JSON.toText identifier) nixValue
60+ _ -> fail $ "expected a single value at the top level, got: " <> cs (show obj)
61+
62+instance ToJSON ModuleConfig where
63+ toJSON (ModuleConfig identifier value) =
64+ JSON.object
65+ [ "tag" .= ("set" :: Text),
66+ "value"
67+ .= JSON.object
68+ [ JSON.fromText (getNixIdentifier identifier) .= value
69+ ]
70+ ]
71+
72+instance PGColumn "json" ModuleConfig where
73+ pgDecode _ =
74+ either (\message -> error $ "Cannot decode ModuleConfig json value: " <> cs message) identity
75+ . JSON.eitherDecode @ModuleConfig
76+ . cs
77+
78+instance PGParameter "json" ModuleConfig where
79+ pgEncode _ = cs . JSON.encode
80+
81+newtype NixIdentifier = NixIdentifier {getNixIdentifier :: Text}
82+ deriving stock (Eq, Ord, Generic, Show)
83+ deriving newtype (FromJSONKey, ToJSONKey)
84+
85+data GithubRepository = GithubRepository
86+ { repoUser :: GhRepoOwner,
87+ repoName :: GhRepoName
88+ }
89+ deriving stock (Eq, Ord, Generic, Show)
90+ deriving (FromJSON, ToJSON)
91+
92+data EncryptedSecret = EncryptedSecret
93+ { encryptedFor :: GithubRepository,
94+ encryptedValue :: Text
95+ }
96+ deriving stock (Eq, Ord, Generic, Show)
97+ deriving (FromJSON, ToJSON)
98+
99+data NixValue
100+ = Secret EncryptedSecret
101+ | NixString Text
102+ | NixPath Text
103+ | NixRaw Text
104+ | NixBool Bool
105+ | NixInt Int
106+ | NixNull
107+ | NixList [NixValue]
108+ | NixSet (Map NixIdentifier NixValue)
109+ deriving stock (Eq, Ord, Generic, Show)
110+
111+instance FromJSON NixValue where
112+ parseJSON = JSON.withObject "NixValue" $ \o -> do
113+ tag :: Text <- o .: "tag"
114+ case tag of
115+ "encryptedSecret" -> Secret <$> o .: "value"
116+ "string" -> NixString <$> o .: "value"
117+ "path" -> NixPath <$> o .: "value"
118+ "raw" -> NixRaw <$> o .: "value"
119+ "bool" -> NixBool <$> o .: "value"
120+ "int" -> NixInt <$> o .: "value"
121+ "null" -> pure NixNull
122+ "list" -> NixList <$> o .: "value"
123+ "set" -> NixSet <$> o .: "value"
124+ other -> fail $ "unexpected tag found: " <> cs (show other)
125+
126+instance ToJSON NixValue where
127+ toJSON = \case
128+ Secret s -> withTag "encryptedSecret" s
129+ NixString s -> withTag "string" s
130+ NixPath p -> withTag "path" p
131+ NixRaw r -> withTag "raw" r
132+ NixBool b -> withTag "bool" b
133+ NixInt n -> withTag "int" n
134+ NixNull -> JSON.object ["tag" .= ("null" :: Text)]
135+ NixList l -> withTag "list" l
136+ NixSet s -> withTag "set" s
137+ where
138+ withTag :: (ToJSON a) => Text -> a -> JSON.Value
139+ withTag tag value =
140+ JSON.object
141+ [ "tag" .= tag,
142+ "value" .= value
143+ ]
144+
145+get :: GhLogin -> M (Maybe GetRepoAndModuleValues)
146+get ghLogin =
147+ DB.pgQuery
148+ [pgSQL|
149+ SELECT id, repo_user, repo_name
150+ FROM module_user_repo
151+ WHERE github_login = ${ghLogin}
152+ |]
153+ >>= \case
154+ [] -> pure Nothing
155+ [(id, repo_user, repo_name)] -> do
156+ userConfig <- getConfig id
157+ modules <- getModules (fst <$> userConfig)
158+ pure
159+ $ Just
160+ ( #repo_user .== repo_user
161+ .+ #repo_name .== repo_name
162+ .+ #user_config .== (snd <$> userConfig)
163+ .+ #modules .== modules
164+ )
165+ _ -> throw $ OtherError "Got more than 1 value for ModuleValues.get"
166+ where
167+ getConfig :: Int64 -> M [(Int64, ModuleValue)]
168+ getConfig id =
169+ DB.pgQuery
170+ [pgSQL|
171+ SELECT modules.id, modules.name, modules.git_commit, module_values.values
172+ FROM module_user_repo
173+ JOIN module_values on module_user_repo.id = module_user_repo_id
174+ JOIN modules on modules.id = module_id
175+ WHERE module_user_repo_id = ${id}
176+ |]
177+ <&> fmap
178+ ( \(id, name, git_commit, values) ->
179+ ( id,
180+ #module_name .== name
181+ .+ #values .== values
182+ .+ #git_commit .== Just git_commit
183+ )
184+ )
185+
186+ getModules :: [Int64] -> M [Module]
187+ getModules ids =
188+ DB.pgQuery
189+ [pgSQL|
190+ SELECT repo_user, repo_name, git_commit, schema, name, description
191+ FROM modules
192+ WHERE
193+ id = ANY(${ids})
194+ OR
195+ ( enabled = true
196+ AND name NOT IN (SELECT name from modules where id = ANY(${ids}))
197+ )
198+
199+ |]
200+ <&> fmap
201+ ( \(repo_user, repo_name, git_commit, schema, name, description) ->
202+ #name .== name
203+ .+ #repo_user .== repo_user
204+ .+ #repo_name .== repo_name
205+ .+ #git_commit .== CommitHash git_commit
206+ .+ #schema .== schema
207+ .+ #description .== description
208+ )
209+
210+update :: GhLogin -> UpdateRepoModuleValues -> M ()
211+update ghLogin row = do
212+ let repo_user = row ^. #repo_user
213+ repo_name = row ^. #repo_name
214+ user_config = row ^. #user_config
215+ void
216+ $ DB.pgExec
217+ [pgSQL|
218+ INSERT INTO module_user_repo
219+ (github_login, repo_user, repo_name)
220+ VALUES (${ghLogin}, ${repo_user}, ${repo_name})
221+ ON CONFLICT (github_login) DO
222+ UPDATE
223+ SET repo_user = ${repo_user},
224+ repo_name = ${repo_name}
225+ |]
226+ DB.pgTransaction $ do
227+ removeAllValues
228+ traverse_ addValue user_config
229+ where
230+ removeAllValues :: M ()
231+ removeAllValues = do
232+ void
233+ $ DB.pgExec
234+ [pgSQL|
235+ DELETE FROM module_values
236+ WHERE module_user_repo_id = (SELECT id FROM module_user_repo WHERE github_login = ${ghLogin})
237+ |]
238+
239+ addValue :: ModuleValue -> M ()
240+ addValue value =
241+ let values = value ^. #values
242+ module_name = value ^. #module_name
243+ git_commit = value ^. #git_commit
244+ in case git_commit of
245+ Just commit ->
246+ void
247+ $ DB.pgExec
248+ [pgSQL|
249+ INSERT INTO module_values
250+ (module_user_repo_id, module_id, values)
251+ VALUES (
252+ (SELECT id FROM module_user_repo WHERE github_login = ${ghLogin}),
253+ (SELECT id FROM modules WHERE name = ${module_name} and git_commit = ${commit}),
254+ ${values}
255+ )
256+ |]
257+ Nothing ->
258+ void
259+ $ DB.pgExec
260+ [pgSQL|
261+ INSERT INTO module_values
262+ (module_user_repo_id, module_id, values)
263+ VALUES (
264+ (SELECT id FROM module_user_repo WHERE github_login = ${ghLogin}),
265+ (SELECT id FROM modules WHERE name = ${module_name} and enabled = true),
266+ ${values}
267+ )
268+ |]
269+
270+delete :: GhLogin -> M ()
271+delete ghLogin = do
272+ void
273+ $ DB.pgExec
274+ [pgSQL|
275+ DELETE FROM module_values
276+ WHERE module_user_repo_id =
277+ (SELECT id
278+ FROM module_user_repo
279+ WHERE github_login = ${ghLogin})
280+ |]
281+ void
282+ $ DB.pgExec
283+ [pgSQL|
284+ DELETE FROM module_user_repo
285+ WHERE github_login = ${ghLogin}
286+ |]
287+
288+getAvailableModules :: M [Module]
289+getAvailableModules =
290+ DB.pgQuery
291+ [pgSQL|
292+ SELECT repo_user, repo_name, git_commit, schema, name, description
293+ FROM modules
294+ WHERE enabled = true
295+ |]
296+ <&> fmap
297+ ( \(repo_user, repo_name, git_commit, schema, name, description) ->
298+ #name .== name
299+ .+ #repo_user .== repo_user
300+ .+ #repo_name .== repo_name
301+ .+ #git_commit .== CommitHash git_commit
302+ .+ #schema .== schema
303+ .+ #description .== description
304+ )
305+
306+insertLatestVersion :: Module -> M ()
307+insertLatestVersion mod =
308+ DB.pgTransaction $ do
309+ let name = mod ^. #name
310+ repo_user = mod ^. #repo_user
311+ repo_name = mod ^. #repo_name
312+ git_commit = mod ^. #git_commit
313+ description = mod ^. #description
314+ schema = mod ^. #schema
315+ void
316+ $ DB.pgExec
317+ [pgSQL|
318+ UPDATE modules
319+ SET enabled = false
320+ WHERE name = ${name}
321+ |]
322+ result <-
323+ DB.pgExec
324+ [pgSQL|
325+ INSERT INTO modules
326+ (repo_user, repo_name, git_commit, enabled, name, description, schema)
327+ VALUES
328+ (${repo_user}, ${repo_name}, ${git_commit}, true, ${name}, ${description}, ${schema})
329+ |]
330+ when (result /= 1) $ throw $ OtherError "Could not insert module to DB."
backend/src/Garnix/Duration.hsnew file+106
@@ -0,0 +1,106 @@
1+module Garnix.Duration
2+ ( Duration,
3+ addDuration,
4+ addTime,
5+ diffTime,
6+ multiplyDuration,
7+ divideDuration,
8+ emptyDuration,
9+ fromDays,
10+ fromHours,
11+ fromMilliSeconds,
12+ fromMinutes,
13+ fromSeconds,
14+ maxDuration,
15+ subTime,
16+ subtractDuration,
17+ threadDelay,
18+ toMicroseconds,
19+ toMilliseconds,
20+ toMinutes,
21+ toSeconds,
22+ )
23+where
24+
25+import Control.Concurrent qualified as M (threadDelay)
26+import Garnix.Prelude
27+import Prelude qualified
28+
29+newtype Duration = Seconds {inSeconds :: Double}
30+ deriving stock (Eq, Ord)
31+ deriving newtype (ToJSON, FromJSON)
32+
33+instance Show Duration where
34+ show =
35+ cs
36+ . fmt
37+ [ ("d", fromDays @Int 1),
38+ ("h", fromHours @Int 1),
39+ ("m", fromMinutes @Int 1),
40+ ("s", fromSeconds @Int 1),
41+ ("ms", fromMilliSeconds @Int 1)
42+ ]
43+ where
44+ fmt :: [(Text, Duration)] -> Duration -> Text
45+ fmt [] (Seconds {inSeconds}) = "Duration " <> show inSeconds
46+ fmt ((unitName, unitDuration) : rest) duration =
47+ let durationInUnit :: Int = floor $ duration `divideDuration` unitDuration
48+ in (if durationInUnit >= 1 then show durationInUnit <> unitName <> " " else "")
49+ <> fmt rest (duration `subtractDuration` Seconds (inSeconds unitDuration * realToFrac durationInUnit))
50+
51+emptyDuration :: Duration
52+emptyDuration = Seconds 0
53+
54+addDuration :: Duration -> Duration -> Duration
55+addDuration (Seconds a) (Seconds b) = Seconds $ a + b
56+
57+subtractDuration :: Duration -> Duration -> Duration
58+subtractDuration (Seconds a) (Seconds b) = Seconds $ a - b
59+
60+multiplyDuration :: (Real a) => Duration -> a -> Duration
61+multiplyDuration (Seconds s) r = Seconds (s * realToFrac r)
62+
63+divideDuration :: Duration -> Duration -> Double
64+divideDuration (Seconds a) (Seconds b) = a / b
65+
66+maxDuration :: Duration -> Duration -> Duration
67+maxDuration (Seconds a) (Seconds b) = Seconds $ max a b
68+
69+fromMilliSeconds :: (Real a) => a -> Duration
70+fromMilliSeconds ms = Seconds (realToFrac ms / 1000)
71+
72+fromSeconds :: (Real a) => a -> Duration
73+fromSeconds = Seconds . realToFrac
74+
75+fromMinutes :: (Real a) => a -> Duration
76+fromMinutes m = Seconds (60 * realToFrac m)
77+
78+fromHours :: (Real a) => a -> Duration
79+fromHours h = Seconds (60 * 60 * realToFrac h)
80+
81+fromDays :: (Real a) => a -> Duration
82+fromDays d = Seconds (60 * 60 * 24 * realToFrac d)
83+
84+toMilliseconds :: Duration -> Int
85+toMilliseconds secs = round $ 1000 * inSeconds secs
86+
87+toMicroseconds :: Duration -> Int
88+toMicroseconds secs = 1000 * toMilliseconds secs
89+
90+toSeconds :: Duration -> Double
91+toSeconds = inSeconds
92+
93+toMinutes :: Duration -> Double
94+toMinutes d = inSeconds d / 60
95+
96+diffTime :: UTCTime -> UTCTime -> Duration
97+diffTime a b = Seconds $ realToFrac $ nominalDiffTimeToSeconds $ diffUTCTime a b
98+
99+addTime :: Duration -> UTCTime -> UTCTime
100+addTime duration = addUTCTime (secondsToNominalDiffTime $ realToFrac $ inSeconds duration)
101+
102+subTime :: Duration -> UTCTime -> UTCTime
103+subTime duration = addTime $ Seconds (inSeconds duration * (-1))
104+
105+threadDelay :: (MonadIO m) => Duration -> m ()
106+threadDelay = liftIO . M.threadDelay . toMicroseconds
backend/src/Garnix/Entitlements.hsnew file+450
@@ -0,0 +1,450 @@
1+module Garnix.Entitlements
2+ ( Hosting (..),
3+ addDefaultEntitlements,
4+ addProduct,
5+ addProductByPriceId,
6+ getHosting,
7+ queryCiTimeEntitlements,
8+ hasRemainingCiTime,
9+
10+ -- * products
11+ ProductToken (..),
12+ displayName,
13+ title,
14+ baseCiTime,
15+ defaultPlanName,
16+ getPlan,
17+ getPlans,
18+ getPlanByProductToken,
19+ getPlanByName,
20+
21+ -- * usage limits
22+ emptyUsageLimits,
23+ setExtraUsageLimits,
24+ )
25+where
26+
27+import Data.Map.Strict (Map, alter)
28+import Data.Map.Strict qualified as Map
29+import Data.Text qualified as T
30+import Database.PostgreSQL.Typed (pgSQL)
31+import Garnix.DB qualified as DB
32+import Garnix.Duration
33+import Garnix.Monad (M, throw)
34+import Garnix.MonetaryCost
35+import Garnix.Prelude
36+import Garnix.StripeLib (PriceId (..))
37+import Garnix.StripeLib qualified as StripeLib
38+import Garnix.Types
39+
40+defaultPlanName :: Text
41+defaultPlanName = "free-v1"
42+
43+fallbackPackagesPerFlake :: Int32
44+fallbackPackagesPerFlake = 100
45+
46+fallbackEvaluationTimeout :: Int16
47+fallbackEvaluationTimeout = 30
48+
49+fallbackBuildTimeout :: Int16
50+fallbackBuildTimeout = 120
51+
52+addDefaultEntitlements :: GhRepoOwner -> M ()
53+addDefaultEntitlements owner = addProduct owner defaultPlanName
54+
55+addProduct :: GhRepoOwner -> Text -> M ()
56+addProduct owner product = do
57+ void
58+ $ DB.pgExec
59+ [pgSQL|
60+ INSERT INTO repo_owner_has_product
61+ ( repo_owner, product )
62+ VALUES
63+ ( ${owner}, ${product} )
64+ ON CONFLICT DO NOTHING
65+ |]
66+
67+addProductByPriceId :: GhRepoOwner -> StripeLib.PriceId -> M ()
68+addProductByPriceId owner priceId = do
69+ res :: [Text] <-
70+ DB.pgQuery
71+ [pgSQL|
72+ SELECT name
73+ FROM products
74+ WHERE price_id = ${StripeLib.getPriceId priceId}
75+ |]
76+ case res of
77+ [name] -> addProduct owner name
78+ [] -> throw $ OtherError $ "Cannot find product with price_id: " <> StripeLib.getPriceId priceId
79+ _ -> throw $ OtherError "impossible: price_id should be unique"
80+
81+data Hosting = Hosting
82+ { planIncludedBranchDeploymentHosts :: Int64,
83+ extraBranchHostingSpend :: MonetaryCost,
84+ maxPrDeploymentTime :: Duration,
85+ largerServers :: Bool
86+ }
87+ deriving (Eq, Show, Generic)
88+
89+getHosting :: GhRepoOwner -> M Hosting
90+getHosting repoOwner = do
91+ hosts :: [(Maybe Int64, Maybe Int64, Maybe Bool)] <-
92+ DB.pgQuery
93+ [pgSQL|
94+ SELECT max(hosting), max(pr_hosting), bool_or(larger_servers)
95+ FROM repo_owner_has_product
96+ INNER JOIN products
97+ ON repo_owner_has_product.product = products.name
98+ WHERE repo_owner_has_product.repo_owner = ${repoOwner}
99+ |]
100+ extraUsageLimits <- getExtraUsageLimits repoOwner
101+ case hosts of
102+ [(hosts, prMinutes, largerServers)] -> do
103+ pure
104+ $ Hosting
105+ { planIncludedBranchDeploymentHosts = fromMaybe 0 hosts,
106+ extraBranchHostingSpend = extraUsageLimits ^. #hostingSpend,
107+ maxPrDeploymentTime = maybe emptyDuration fromMinutes prMinutes `addDuration` (extraUsageLimits ^. #prDeployTime),
108+ largerServers = fromMaybe False largerServers
109+ }
110+ [] ->
111+ pure
112+ $ Hosting
113+ { planIncludedBranchDeploymentHosts = 0,
114+ extraBranchHostingSpend = extraUsageLimits ^. #hostingSpend,
115+ maxPrDeploymentTime = emptyDuration,
116+ largerServers = False
117+ }
118+ _ : _ : _ -> throw $ OtherError "impossible: more than one result from aggregate query"
119+
120+queryCiTimeEntitlements :: GhRepoOwner -> M Duration
121+queryCiTimeEntitlements repoOwner = do
122+ extraCiMinutesResult :: [Int32] <-
123+ DB.pgQuery
124+ [pgSQL|
125+ SELECT extra_ci_time_in_minutes
126+ FROM repo_owner_usage_limits
127+ WHERE repo_owner = ${repoOwner}
128+ |]
129+ extraCiTime <- case extraCiMinutesResult of
130+ [minutes] -> pure $ fromMinutes minutes
131+ [] -> pure emptyDuration
132+ _ -> throw $ OtherError "impossible: more than one extra ci minutes for repo owner"
133+ hosts :: [Maybe Int64] <-
134+ DB.pgQuery
135+ [pgSQL|
136+ SELECT max(ci_minutes)
137+ FROM repo_owner_has_product
138+ INNER JOIN products
139+ ON repo_owner_has_product.product = products.name
140+ WHERE repo_owner_has_product.repo_owner = ${repoOwner}
141+ |]
142+ planMinutes <- case hosts of
143+ [Just result] -> pure $ fromMinutes result
144+ [Nothing] -> pure emptyDuration
145+ [] -> pure emptyDuration
146+ _ : _ : _ -> throw $ OtherError "impossible: more than one result from aggregate query"
147+ pure $ planMinutes `addDuration` extraCiTime
148+
149+hasRemainingCiTime :: GhRepoOwner -> M Bool
150+hasRemainingCiTime owner = do
151+ total <- queryCiTimeEntitlements owner
152+ used <- DB.getCurrentMonthUsage owner
153+ pure $ total >= used
154+
155+-- * products
156+
157+newtype ProductToken = ProductToken {getProductToken :: Text}
158+ deriving newtype (Eq, Show, FromJSON, ToJSON, FromHttpApiData)
159+
160+mergePlans :: [(ProductPlan, Maybe PriceId)] -> Maybe ProductPlan
161+mergePlans allPlans = case filter (isJust . snd) allPlans of
162+ [] -> merge $ fmap fst allPlans
163+ withPriceIds -> do
164+ allMerged <- merge $ fmap fst allPlans
165+ withPriceIdsMerged <- merge $ fmap fst withPriceIds
166+ Just
167+ $ ProductPlan
168+ { _productPlanDisplayName = withPriceIdsMerged ^. displayName,
169+ _productPlanDescription = withPriceIdsMerged ^. description,
170+ _productPlanBaseCiTime = allMerged ^. baseCiTime,
171+ _productPlanMaximumPrDeploymentTime = allMerged ^. maximumPrDeploymentTime,
172+ _productPlanIncludedBranchDeploymentHosts = allMerged ^. includedBranchDeploymentHosts,
173+ _productPlanMaximumPackagesPerFlake = allMerged ^. maximumPackagesPerFlake,
174+ _productPlanPackageEvaluationTimeout = allMerged ^. packageEvaluationTimeout,
175+ _productPlanPackageBuildTimeout = allMerged ^. packageBuildTimeout,
176+ _productPlanExtraUsage = allMerged ^. extraUsage,
177+ _productPlanIsPaid = allMerged ^. isPaid
178+ }
179+ where
180+ merge :: [ProductPlan] -> Maybe ProductPlan
181+ merge plans = case sortBy (compare `on` _productPlanDisplayName) plans of
182+ [] -> Nothing
183+ a : as -> Just $ foldl' merge a as
184+ where
185+ merge a b =
186+ ProductPlan
187+ { _productPlanDisplayName = a ^. displayName <> ", " <> b ^. displayName,
188+ _productPlanDescription = case catMaybes [a ^. description, b ^. description] of
189+ [] -> Nothing
190+ snippets -> Just $ T.intercalate ", " snippets,
191+ _productPlanBaseCiTime = maxDuration (a ^. baseCiTime) (b ^. baseCiTime),
192+ _productPlanMaximumPrDeploymentTime = maxDuration (a ^. maximumPrDeploymentTime) (b ^. maximumPrDeploymentTime),
193+ _productPlanIncludedBranchDeploymentHosts = max (a ^. includedBranchDeploymentHosts) (b ^. includedBranchDeploymentHosts),
194+ _productPlanMaximumPackagesPerFlake = max (a ^. maximumPackagesPerFlake) (b ^. maximumPackagesPerFlake),
195+ _productPlanPackageEvaluationTimeout = max (a ^. packageEvaluationTimeout) (b ^. packageEvaluationTimeout),
196+ _productPlanPackageBuildTimeout = max (a ^. packageBuildTimeout) (b ^. packageBuildTimeout),
197+ _productPlanExtraUsage =
198+ ExtraUsageLimits
199+ { ciTime = maxDuration (a ^. extraUsage . #ciTime) (b ^. extraUsage . #ciTime),
200+ prDeployTime = maxDuration (a ^. extraUsage . #prDeployTime) (b ^. extraUsage . #prDeployTime),
201+ hostingSpend = max (a ^. extraUsage . #hostingSpend) (b ^. extraUsage . #hostingSpend)
202+ },
203+ _productPlanIsPaid = (a ^. isPaid) || (b ^. isPaid)
204+ }
205+
206+getPlan :: GhRepoOwner -> M ProductPlan
207+getPlan repoOwner =
208+ Map.lookup repoOwner <$> getPlans [repoOwner] >>= \case
209+ Nothing -> throw $ OtherError $ "Impossible: no plan found for " <> getGhLogin (getGhRepoOwner repoOwner)
210+ Just plan -> pure plan
211+
212+getPlans :: [GhRepoOwner] -> M (Map GhRepoOwner ProductPlan)
213+getPlans orgs = do
214+ forM_ orgs addDefaultEntitlements
215+ res ::
216+ [ ( GhRepoOwner,
217+ Text,
218+ Maybe Text,
219+ Maybe Text,
220+ Maybe Int64,
221+ Maybe Int64,
222+ Maybe Int64,
223+ Maybe Text,
224+ Maybe Int32,
225+ Maybe Int16,
226+ Maybe Int16,
227+ Maybe Int32,
228+ Maybe Int32,
229+ Maybe Int32
230+ )
231+ ] <-
232+ DB.pgQuery
233+ [pgSQL|
234+ SELECT
235+ repo_owner_has_product.repo_owner,
236+ name,
237+ title,
238+ description,
239+ hosting,
240+ pr_hosting,
241+ ci_minutes,
242+ price_id,
243+ packages_per_flake,
244+ package_eval_timeout_in_minutes,
245+ package_build_timeout_in_minutes,
246+ extra_ci_time_in_minutes,
247+ extra_pr_hosting_in_minutes,
248+ extra_hosting_spending_limit_in_usd
249+ FROM repo_owner_has_product
250+ INNER JOIN products
251+ ON repo_owner_has_product.product = products.name
252+ LEFT JOIN repo_owner_usage_limits
253+ on repo_owner_usage_limits.repo_owner = repo_owner_has_product.repo_owner
254+ WHERE repo_owner_has_product.repo_owner = ANY(${orgs})
255+ |]
256+ pure
257+ $ Map.mapMaybe mergePlans
258+ $ foldl'
259+ ( \acc
260+ ( repoOwner,
261+ name,
262+ title,
263+ description,
264+ branchHosting,
265+ prDeployMinutes,
266+ ciMinutes,
267+ priceId,
268+ packagesPerFlake,
269+ packageEvalTimeout,
270+ packageBuildTimeout,
271+ extraCiMinutes,
272+ extraPrMinutes,
273+ extraHostingSpendInUsd
274+ ) ->
275+ Data.Map.Strict.alter
276+ ( Just
277+ . ( ( ProductPlan
278+ { _productPlanDisplayName = fromMaybe name title,
279+ _productPlanDescription = description,
280+ _productPlanBaseCiTime = maybe emptyDuration fromMinutes ciMinutes,
281+ _productPlanMaximumPrDeploymentTime = maybe emptyDuration fromMinutes prDeployMinutes,
282+ _productPlanIncludedBranchDeploymentHosts = fromMaybe 0 branchHosting,
283+ _productPlanMaximumPackagesPerFlake = fromMaybe fallbackPackagesPerFlake packagesPerFlake,
284+ _productPlanPackageEvaluationTimeout = fromMaybe fallbackEvaluationTimeout packageEvalTimeout,
285+ _productPlanPackageBuildTimeout = fromMaybe fallbackBuildTimeout packageBuildTimeout,
286+ _productPlanExtraUsage =
287+ ExtraUsageLimits
288+ { ciTime = maybe emptyDuration fromMinutes extraCiMinutes,
289+ prDeployTime = maybe emptyDuration fromMinutes extraPrMinutes,
290+ hostingSpend = usd $ fromMaybe 0 extraHostingSpendInUsd
291+ },
292+ _productPlanIsPaid = isJust priceId
293+ },
294+ fmap PriceId priceId
295+ )
296+ :
297+ )
298+ . fromMaybe []
299+ )
300+ repoOwner
301+ acc
302+ )
303+ mempty
304+ res
305+
306+getPlanByProductToken :: ProductToken -> M (Maybe (Text, Maybe PriceId, ProductPlan))
307+getPlanByProductToken productToken = do
308+ res <-
309+ fmap
310+ ( \( name :: Text,
311+ title :: Maybe Text,
312+ description :: Maybe Text,
313+ priceId :: Maybe Text,
314+ branchHosting :: Maybe Int64,
315+ prDeployMinutes :: Maybe Int64,
316+ ciMinutes :: Maybe Int64,
317+ packagesPerFlake :: Maybe Int32,
318+ packageEvalTimeout :: Maybe Int16,
319+ packageBuildTimeout :: Maybe Int16
320+ ) ->
321+ Just
322+ ( name,
323+ fmap PriceId priceId,
324+ ProductPlan
325+ { _productPlanDisplayName = fromMaybe name title,
326+ _productPlanDescription = description,
327+ _productPlanBaseCiTime = maybe emptyDuration fromMinutes ciMinutes,
328+ _productPlanMaximumPrDeploymentTime = maybe emptyDuration fromMinutes prDeployMinutes,
329+ _productPlanIncludedBranchDeploymentHosts = fromMaybe 0 branchHosting,
330+ _productPlanMaximumPackagesPerFlake = fromMaybe fallbackPackagesPerFlake packagesPerFlake,
331+ _productPlanPackageEvaluationTimeout = fromMaybe fallbackEvaluationTimeout packageEvalTimeout,
332+ _productPlanPackageBuildTimeout = fromMaybe fallbackBuildTimeout packageBuildTimeout,
333+ _productPlanExtraUsage = emptyUsageLimits,
334+ _productPlanIsPaid = isJust priceId
335+ }
336+ )
337+ )
338+ <$> DB.pgQuery
339+ [pgSQL|
340+ SELECT
341+ name,
342+ title,
343+ description,
344+ price_id,
345+ hosting,
346+ pr_hosting,
347+ ci_minutes,
348+ packages_per_flake,
349+ package_eval_timeout_in_minutes,
350+ package_build_timeout_in_minutes
351+ FROM products
352+ WHERE token = ${getProductToken productToken}
353+ |]
354+ case res of
355+ [p] -> pure p
356+ [] -> pure Nothing
357+ _ -> throw $ OtherError "Impossible: price_id is unique"
358+
359+getPlanByName :: Text -> M (Maybe ProductPlan)
360+getPlanByName planName = do
361+ res <-
362+ DB.pgQuery
363+ [pgSQL|
364+ SELECT
365+ name,
366+ description,
367+ ci_minutes,
368+ pr_hosting,
369+ hosting,
370+ packages_per_flake,
371+ package_eval_timeout_in_minutes,
372+ package_build_timeout_in_minutes,
373+ price_id
374+ FROM products
375+ WHERE name = ${planName}
376+ |]
377+ let plans =
378+ map
379+ ( \( name :: Text,
380+ description :: Maybe Text,
381+ ciMinutes :: Maybe Int64,
382+ prDeployMinutes :: Maybe Int64,
383+ hosting :: Maybe Int64,
384+ packagesPerFlake :: Maybe Int32,
385+ packageEvalTimeoutInMinutes :: Maybe Int16,
386+ packageBuildTimeoutInMinutes :: Maybe Int16,
387+ priceId :: Maybe Text
388+ ) ->
389+ ProductPlan
390+ { _productPlanDisplayName = name,
391+ _productPlanDescription = description,
392+ _productPlanBaseCiTime = maybe emptyDuration fromMinutes ciMinutes,
393+ _productPlanMaximumPrDeploymentTime = maybe emptyDuration fromMinutes prDeployMinutes,
394+ _productPlanIncludedBranchDeploymentHosts = fromMaybe 0 hosting,
395+ _productPlanMaximumPackagesPerFlake = fromMaybe fallbackPackagesPerFlake packagesPerFlake,
396+ _productPlanPackageEvaluationTimeout = fromMaybe fallbackEvaluationTimeout packageEvalTimeoutInMinutes,
397+ _productPlanPackageBuildTimeout = fromMaybe fallbackBuildTimeout packageBuildTimeoutInMinutes,
398+ _productPlanExtraUsage = emptyUsageLimits,
399+ _productPlanIsPaid = isJust priceId
400+ }
401+ )
402+ res
403+ case plans of
404+ [plan] -> pure (Just plan)
405+ [] -> pure Nothing
406+ _ -> throw $ OtherError $ "Impossible: more than one plan with name " <> planName
407+
408+setExtraUsageLimits :: GhRepoOwner -> ExtraUsageLimits -> M ()
409+setExtraUsageLimits owner newLimits = do
410+ let extraCiMinutes :: Int32 = floor $ toMinutes $ newLimits ^. #ciTime
411+ let extraPrMinutes :: Int32 = floor $ toMinutes $ newLimits ^. #prDeployTime
412+ let extraHostingSpendInUsd :: Int32 = getCostInCents (newLimits ^. #hostingSpend) `quot` 100
413+ void
414+ $ DB.pgExec
415+ [pgSQL|
416+ INSERT INTO repo_owner_usage_limits
417+ ( repo_owner, extra_ci_time_in_minutes, extra_pr_hosting_in_minutes, extra_hosting_spending_limit_in_usd )
418+ VALUES
419+ ( ${owner}, ${extraCiMinutes}, ${extraPrMinutes}, ${extraHostingSpendInUsd} )
420+ ON CONFLICT (repo_owner) DO UPDATE
421+ SET
422+ extra_ci_time_in_minutes = ${extraCiMinutes},
423+ extra_pr_hosting_in_minutes = ${extraPrMinutes},
424+ extra_hosting_spending_limit_in_usd = ${extraHostingSpendInUsd}
425+ |]
426+
427+getExtraUsageLimits :: GhRepoOwner -> M ExtraUsageLimits
428+getExtraUsageLimits owner = do
429+ result <-
430+ map
431+ ( \(extraCiMinutes :: Int32, extraPrMinutes :: Int32, extraHostingSpendInUsd :: Int32) ->
432+ ExtraUsageLimits
433+ { ciTime = fromMinutes extraCiMinutes,
434+ prDeployTime = fromMinutes extraPrMinutes,
435+ hostingSpend = usd extraHostingSpendInUsd
436+ }
437+ )
438+ <$> DB.pgQuery
439+ [pgSQL|
440+ SELECT
441+ extra_ci_time_in_minutes,
442+ extra_pr_hosting_in_minutes,
443+ extra_hosting_spending_limit_in_usd
444+ FROM repo_owner_usage_limits
445+ WHERE repo_owner = ${owner}
446+ |]
447+ case result of
448+ [limits] -> pure limits
449+ [] -> pure emptyUsageLimits
450+ _ -> throw $ OtherError "impossible: more than one usage limits row for repo owner"
backend/src/Garnix/ExpiringCache.hsnew file+76
@@ -0,0 +1,76 @@
1+module Garnix.ExpiringCache
2+ ( ExpiringCache,
3+ mkCache,
4+ lookupCache,
5+ clearCache,
6+ )
7+where
8+
9+import Control.Concurrent.Lifted
10+import Data.Bitraversable (bitraverse)
11+import Data.Map.Strict (Map, insert, lookup)
12+import Garnix.Duration
13+import Garnix.Monad
14+import Garnix.Prelude hiding (insert, lookup)
15+import Garnix.Types
16+
17+data CacheValue a = CacheValue
18+ { value :: Either ErrorWithContext a,
19+ expiresAt :: UTCTime
20+ }
21+
22+data ExpiringCache k v = ExpiringCache
23+ { name :: Maybe Text,
24+ cacheMVar :: MVar (Map k (MVar (Maybe (CacheValue v)))),
25+ ttl :: Duration,
26+ errorTtl :: Duration
27+ }
28+
29+mkCache :: (Ord k) => Maybe Text -> Duration -> Duration -> IO (ExpiringCache k v)
30+mkCache name ttl errorTtl = do
31+ cacheMVar <- newMVar mempty
32+ pure $ ExpiringCache {name, cacheMVar, ttl, errorTtl}
33+
34+clearCache :: (MonadIO m, Ord key) => ExpiringCache key val -> m ()
35+clearCache ExpiringCache {cacheMVar} = void $ liftIO $ swapMVar cacheMVar mempty
36+
37+lookupCache :: (Show key, Ord key) => ExpiringCache key val -> key -> M val -> M val
38+lookupCache ExpiringCache {name, cacheMVar, ttl, errorTtl} key notInCacheAction = do
39+ inner <- modifyMVar cacheMVar $ \map -> case lookup key map of
40+ Just innerMVar -> pure (map, innerMVar)
41+ Nothing -> do
42+ innerMVar <- newMVar Nothing
43+ pure (insert key innerMVar map, innerMVar)
44+ result <- modifyMVar inner $ \mVal -> do
45+ now <- liftIO getCurrentTime
46+ let getNewCacheValue = do
47+ newVal <- try notInCacheAction
48+ let expiresAt = addTime (either (const errorTtl) (const ttl) newVal) now
49+ pure (Just $ CacheValue newVal expiresAt, newVal)
50+ logCacheEvent event = case name of
51+ Just name -> log Informational $ name <> ": " <> event <> " for " <> show key
52+ Nothing -> pure ()
53+ case mVal of
54+ Nothing -> do
55+ logCacheEvent "cache miss"
56+ getNewCacheValue
57+ Just (CacheValue {value, expiresAt}) ->
58+ if expiresAt <= now
59+ then do
60+ logCacheEvent "cache expired"
61+ getNewCacheValue
62+ else do
63+ logCacheEvent "cache hit"
64+ (mVal,) <$> bitraverse wrapAsCachedError pure value
65+ either rethrow pure result
66+ where
67+ wrapAsCachedError :: ErrorWithContext -> M ErrorWithContext
68+ wrapAsCachedError error = do
69+ spans <- asks spanCtx
70+ pure
71+ $ ErrorWithContext
72+ { callstack = callStack,
73+ spans,
74+ severity = error ^. #severity,
75+ err = CachedError error
76+ }
backend/src/Garnix/FlakeInputAuthorization.hsnew file+225
@@ -0,0 +1,225 @@
1+{-# LANGUAGE EmptyDataDeriving #-}
2+{-# LANGUAGE NamedFieldPuns #-}
3+{-# LANGUAGE QuasiQuotes #-}
4+
5+module Garnix.FlakeInputAuthorization
6+ ( checkAuthorization,
7+ -- exported for tests
8+ FlakeInput (..),
9+ GithubFlakeInput (..),
10+ _parseFlakeMetaData,
11+ _extractPrivateReposFromErrors,
12+ )
13+where
14+
15+import Control.Lens.Regex.Text qualified as RE
16+import Cradle
17+import Data.Aeson.KeyMap (KeyMap)
18+import Data.Aeson.KeyMap qualified as KeyMap
19+import Data.Aeson.Types (Parser, Value, withObject, (.:))
20+import Data.Map.Strict qualified as Map
21+import Data.Text qualified as T
22+import Garnix.Monad
23+import Garnix.NixConfig
24+import Garnix.Prelude
25+import Garnix.Sandbox
26+import Garnix.Types hiding (owner, repo)
27+import Network.URI
28+import System.Directory.Extra (canonicalizePath, doesPathExist)
29+import System.FilePath (isAbsolute)
30+
31+-- | Throws if access should be denied.
32+-- Otherwise, returns the needed authorization to fetch all flake inputs.
33+checkAuthorization :: (HasCallStack) => FlakeDir -> RepoConfig -> CommitInfo -> M NixConfig
34+checkAuthorization flakeDir repoConfig commitInfo = do
35+ cacheDir <- getNixXdgCacheDir
36+ nixConfig <- view #userNixConfig
37+ curDir <- view #workingDir
38+ flakeDir' <- safeGetAbsoluteFlakeDir flakeDir
39+ (exitCode, StdoutTrimmed stdout, StderrRaw stderr) <-
40+ (>>= run)
41+ $ cmd "nix"
42+ & addArgs ["flake", "metadata", "--json", flakeDir']
43+ & addNixConfigEnvironment nixConfig
44+ & setWorkingDir curDir
45+ & pure
46+ & inNixSandbox [] (Just cacheDir)
47+ output <- case exitCode of
48+ ExitSuccess -> pure stdout
49+ ExitFailure e -> do
50+ log Informational $ "'nix flake metadata --json' failed with stderr: " <> cs stderr
51+ case _extractPrivateReposFromErrors (cs stderr) of
52+ Nothing ->
53+ throw
54+ $ RunProcessError
55+ { command = "nix",
56+ arguments = ["flake", "metadata", "--json", cs flakeDir'],
57+ stdErr = cs stderr,
58+ stdOut = stdout,
59+ exitCode = e
60+ }
61+ Just privateInputs ->
62+ throw
63+ $ OtherError
64+ $ T.intercalate "\n"
65+ $ flip map privateInputs
66+ $ \input ->
67+ showPretty input <> " is private or doesn't exist (or your flake.lock file is outdated).\nIf it is private and you would like to use it, see https://garnix.io/docs/private_inputs."
68+ inputs <- aesonDecode "output of 'nix flake metadata --json'" _parseFlakeMetaData output
69+ githubInputs <- checkInputsAllowed curDir inputs
70+ let repoInfo' = commitInfo ^. repoInfo
71+ privateInputs <- filterM (\(GithubFlakeInput owner repo) -> not . isRepoPublic <$> getRepoPublicity (repoInfo' ^. installationAuth) owner repo) githubInputs
72+ selfRepoPublicity <- getRepoPublicity (repoInfo' ^. installationAuth) (repoInfo' ^. ghRepoOwner) (repoInfo' ^. ghRepoName)
73+ case privateInputs of
74+ [] -> pure $ NixConfig mempty
75+ _
76+ | isRepoPublic selfRepoPublicity -> do
77+ let skipPrivateInputChecks = repoConfig ^. skipPrivateInputsCheckForCollaborators
78+ unless skipPrivateInputChecks $ do
79+ throw
80+ $ OtherError
81+ $ "Public repository has private dependencies, which is not allowed. Private dependencies: "
82+ <> T.unwords (fmap showPretty privateInputs)
83+ pure $ githubAccessTokenNixConfig $ repoInfo' ^. ghToken
84+ | isJust $ commitInfo ^. prFromFork ->
85+ throw
86+ $ OtherError
87+ "Repository has private dependencies, but PR is from fork."
88+ _ -> do
89+ baseRepoCollaborators' <- getRepoCollaborators (repoInfo' ^. installationAuth) (repoInfo' ^. ghRepoOwner) (repoInfo' ^. ghRepoName) <?> "Getting repo collaborators"
90+ baseRepoCollaborators <- case baseRepoCollaborators' of
91+ RepoNotFound -> throw $ OtherError "checkAuthorization: base repo not found"
92+ GhCollaborators collaborators -> pure collaborators
93+ forM_ privateInputs $ \privateInput -> do
94+ when ((repoInfo' ^. ghRepoOwner) /= owner privateInput) $ do
95+ throw $ OtherError $ showPretty privateInput <> " is private or doesn't exist.\nIf it is private and you would like to use it, see https://garnix.io/docs/private_inputs."
96+ let skipPrivateInputChecks = repoConfig ^. skipPrivateInputsCheckForCollaborators
97+ unless skipPrivateInputChecks $ do
98+ thisInputCollaborators' <-
99+ getRepoCollaborators (repoInfo' ^. installationAuth) (repoInfo' ^. ghRepoOwner) (repo privateInput)
100+ thisInputCollaborators <- case thisInputCollaborators' of
101+ RepoNotFound -> throw $ OtherError $ "checkAuthorization: repo " <> (getGhLogin . getGhRepoOwner $ repoInfo' ^. ghRepoOwner) <> "/" <> getGhRepoName (repoInfo' ^. ghRepoName) <> " not found"
102+ GhCollaborators collaborators -> pure collaborators
103+ let missingUsers = filter (`notElem` thisInputCollaborators) baseRepoCollaborators
104+ unless (null missingUsers)
105+ $ throw
106+ $ OtherError
107+ $ "Aborting, as some collaborators of this repository "
108+ <> "don't have access to a required private dependency ("
109+ <> showPretty privateInput
110+ <> "). The users missing permissions are: "
111+ <> showPretty missingUsers
112+ pure $ githubAccessTokenNixConfig $ repoInfo' ^. ghToken
113+
114+_extractPrivateReposFromErrors :: Text -> Maybe [Text]
115+_extractPrivateReposFromErrors s =
116+ case s ^.. [RE.regex|while fetching the input '(github:.*)'\n|] . RE.groups of
117+ [] -> Nothing
118+ matches -> Just $ flip map matches $ \case
119+ [match] -> match
120+ _ -> error "impossible: regex only has one match group"
121+
122+githubAccessTokenNixConfig :: GhToken -> NixConfig
123+githubAccessTokenNixConfig token = NixConfig $ Map.insert "access-tokens" ("github.com=" <> cs (getGhToken token)) mempty
124+
125+checkInputsAllowed :: FilePath -> [FlakeInput] -> M [GithubFlakeInput]
126+checkInputsAllowed repoDir inputs = do
127+ maybes <- forM inputs $ \flakeInput -> case flakeInput of
128+ Github githubFlakeInput -> pure $ Just githubFlakeInput
129+ PathInput path -> do
130+ unlessM (pathInputIsOk repoDir path) $ do
131+ log Informational $ "disallowed flake input: " <> show (pretty flakeInput)
132+ throw $ OtherError $ "flake inputs of type 'path:' not allowed: " <> show (pretty flakeInput)
133+ pure Nothing
134+ FileInput url -> do
135+ case uriScheme <$> parseURI url of
136+ Just "http:" -> pure Nothing
137+ Just "https:" -> pure Nothing
138+ _ -> do
139+ log Informational $ "disallowed flake input: " <> show (pretty flakeInput)
140+ throw $ OtherError $ "flake input disallowed: " <> show (pretty flakeInput)
141+ RawRepoUrlInput url -> do
142+ case uriScheme <$> parseURI url of
143+ Just "http:" -> pure Nothing
144+ Just "https:" -> pure Nothing
145+ Just "ssh:" -> pure Nothing
146+ _ -> do
147+ log Informational $ "disallowed flake input: " <> show (pretty flakeInput)
148+ throw $ OtherError $ "flake input disallowed: " <> show (pretty flakeInput)
149+ pure $ catMaybes maybes
150+ where
151+ pathInputIsOk :: FilePath -> FilePath -> M Bool
152+ pathInputIsOk repoDir inputPath
153+ | isAbsolute inputPath = pure False
154+ | otherwise = do
155+ let combined = repoDir </> inputPath
156+ exists <- liftIO $ doesPathExist combined
157+ if not exists
158+ then pure False
159+ else do
160+ normalizedInputPath <- liftIO $ canonicalizePath combined
161+ pure $ (repoDir <> "/") `isPrefixOf` normalizedInputPath
162+
163+data FlakeInput
164+ = Github GithubFlakeInput
165+ | PathInput FilePath
166+ | FileInput FilePath
167+ | RawRepoUrlInput String
168+ deriving stock (Show, Eq, Ord)
169+
170+instance Pretty FlakeInput where
171+ pretty = \case
172+ Github input -> pretty input
173+ PathInput path -> "path:" <> pretty path
174+ FileInput url -> pretty url
175+ RawRepoUrlInput url -> pretty url
176+
177+data GithubFlakeInput = GithubFlakeInput {owner :: GhRepoOwner, repo :: GhRepoName}
178+ deriving stock (Show, Eq, Ord)
179+
180+instance Pretty GithubFlakeInput where
181+ pretty GithubFlakeInput {owner, repo} =
182+ "github:" <> pretty owner <> "/" <> pretty repo
183+
184+_parseFlakeMetaData :: Value -> Parser [FlakeInput]
185+_parseFlakeMetaData json = do
186+ locks <- withObject "flake metadata" pure json >>= (.: "locks")
187+ nodes <- locks .: "nodes"
188+ rootKey <- locks .: "root"
189+ nodes :: KeyMap Value <-
190+ KeyMap.delete rootKey
191+ <$> parseJSON nodes
192+ fmap catMaybes
193+ $ forM (KeyMap.elems nodes)
194+ $ withObject "flake input node"
195+ $ \o -> do
196+ parseFlakeInput o "original"
197+
198+parseFlakeInput :: KeyMap Value -> Text -> Parser (Maybe FlakeInput)
199+parseFlakeInput input key = do
200+ obj <- input .: fromString (cs key)
201+ typ :: Maybe Text <- obj .: "type"
202+ case typ of
203+ Just "indirect" -> do
204+ when
205+ (key == "locked")
206+ (fail "Type of locked inputs can't be \"indirect\".")
207+ parseFlakeInput input "locked"
208+ Just "github" ->
209+ Just . Github <$> (GithubFlakeInput <$> obj .: "owner" <*> obj .: "repo")
210+ Just "path" -> do
211+ Just . PathInput <$> obj .: "path"
212+ Just "file" -> do
213+ Just . FileInput <$> obj .: "url"
214+ Just "git" -> do
215+ Just . RawRepoUrlInput <$> obj .: "url"
216+ Just "hg" -> do
217+ Just . RawRepoUrlInput <$> obj .: "url"
218+ Just typ | typ `elem` otherBlessedTypes -> pure Nothing
219+ Just typ ->
220+ fail $ "unsupported flake input type: " <> cs typ
221+ Nothing ->
222+ fail "missing flake input type"
223+
224+otherBlessedTypes :: [Text]
225+otherBlessedTypes = ["tarball", "gitlab", "sourcehut"]
backend/src/Garnix/GetAttributes.hsnew file+114
@@ -0,0 +1,114 @@
1+module Garnix.GetAttributes where
2+
3+import Control.Lens.Regex.Text qualified as RE
4+import Cradle
5+import Data.Text qualified as T
6+import Garnix.Attribute
7+import Garnix.Monad
8+import Garnix.Monad.Metrics
9+import Garnix.Monad.Pool (withPoolM)
10+import Garnix.NixConfig (addNixConfigEnvironment)
11+import Garnix.Prelude
12+import Garnix.Sandbox
13+import Garnix.Types
14+import Garnix.YamlConfig
15+
16+subAttrs ::
17+ (HasCallStack) =>
18+ GhRepoOwner ->
19+ FlakeDir ->
20+ Attribute ->
21+ M [Attribute]
22+subAttrs repoOwner flakeDir attr = do
23+ cacheDir <- getNixXdgCacheDir
24+ nixConfig <- view #userNixConfig
25+ curDir <- view #workingDir
26+ attr' <- localAttr flakeDir attr
27+ let args = ["eval", attr', "--apply", "builtins.attrNames", "--json"]
28+ res <-
29+ withPoolM nixEvalPool repoOwner $ do
30+ (exitCode, StdoutTrimmed stdout, StderrRaw stderr) <-
31+ (>>= run)
32+ $ cmd "nix"
33+ & addArgs args
34+ & addNixConfigEnvironment nixConfig
35+ & setWorkingDir curDir
36+ & pure
37+ & inNixSandbox [] (Just cacheDir)
38+ case exitCode of
39+ ExitSuccess -> pure stdout
40+ ExitFailure _ ->
41+ if isDoesNotProvideAttributeError $ cs stderr
42+ then pure "[]"
43+ else throw $ ErrorGettingAttributesToBuild (cs stderr)
44+ parsed <- aesonDecode ("output of 'nix" <> T.unwords args <> "'") parseJSON res
45+ pure $ catMaybes [addSubAttr attr r | r <- parsed]
46+
47+ifIsAttr :: (HasCallStack) => GhRepoOwner -> FlakeDir -> Attribute -> M [Attribute]
48+ifIsAttr repoOwner flakeDir attr = do
49+ cacheDir <- getNixXdgCacheDir
50+ nixConfig <- view #userNixConfig
51+ curDir <- view #workingDir
52+ attr' <- localAttr flakeDir attr
53+ -- Probably there is a prettier way of checking that the attr exists
54+ let args = ["eval", attr', "--apply", "x : {}", "--json"]
55+ res <-
56+ withPoolM nixEvalPool repoOwner $ do
57+ (exitCode, StdoutTrimmed stdout, StderrRaw stderr) <-
58+ (>>= run)
59+ $ cmd "nix"
60+ & addArgs args
61+ & addNixConfigEnvironment nixConfig
62+ & setWorkingDir curDir
63+ & pure
64+ & inNixSandbox [] (Just cacheDir)
65+ case exitCode of
66+ ExitSuccess -> pure $ Just stdout
67+ ExitFailure _ ->
68+ if isDoesNotProvideAttributeError $ cs stderr
69+ then pure Nothing
70+ else throw $ ErrorGettingAttributesToBuild (cs stderr)
71+ case res of
72+ Nothing -> pure []
73+ Just _ -> pure [attr]
74+
75+isDoesNotProvideAttributeError :: Text -> Bool
76+isDoesNotProvideAttributeError s =
77+ let regex = [RE.regex|error: flake '.*' does not provide attribute .*|]
78+ in case s ^.. regex . RE.groups of
79+ [] -> False
80+ _ -> True
81+
82+-- | Get all the attributes that will be built.
83+getAttributesToBuild ::
84+ (HasCallStack) =>
85+ CommitInfo ->
86+ GarnixConfig ->
87+ M [Attribute]
88+getAttributesToBuild commitInfo cfg = timingAs #getAttrsToBuildTime $ do
89+ -- We first check whether a child could even match a config entry. Only
90+ -- then do we try to figure out it's children. And finally we filter those
91+ -- that in fact match.
92+ -- We do the first step because the second might fail. But we don't care
93+ -- whether it does if it couldn't have mattered what the result was.
94+ general <- forM allParentAttrs $ \attr ->
95+ if attr `mightMatchConfig` cfg
96+ then subAttrs (commitInfo ^. repoInfo . ghRepoOwner) (cfg ^. flakeDir) attr
97+ else pure []
98+ defaults <- forM allDirectAttrs $ \attr ->
99+ if matchesConfig attr cfg (commitInfo ^. branch)
100+ then ifIsAttr (commitInfo ^. repoInfo . ghRepoOwner) (cfg ^. flakeDir) attr
101+ else pure []
102+ pure [attr | attr <- join (general <> defaults), matchesConfig attr cfg (commitInfo ^. branch)]
103+
104+{-
105+Note [Async process exceptions]
106+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
107+
108+It's still unclear to me why, but typed-process' commands handle async
109+exceptions poorly. In particular, running them inside an `async` and cancelling
110+the process causes the subprocess to persist.
111+
112+That's bad in general, but is particularly noticeable where we use flock to
113+figure out how long to tail for.
114+-}
backend/src/Garnix/GithubInterface.hsnew file+502
@@ -0,0 +1,502 @@
1+module Garnix.GithubInterface
2+ ( realGithubInterface,
3+ fromRunReport,
4+ -- exported for testing
5+ _retryWhen,
6+ _retryGithubRequest,
7+ )
8+where
9+
10+import Control.Retry (RetryPolicyM, fullJitterBackoff, limitRetries, retryOnError, retrying)
11+import Data.Aeson qualified as Aeson
12+import Data.Aeson.Lens (key, _Array, _Integer, _Integral, _String)
13+import Data.Row (Rec, type (.==))
14+import Data.Tagged (Tagged (..))
15+import Data.Time.Clock.POSIX (utcTimeToPOSIXSeconds)
16+import Data.Vector qualified as Vector
17+import Garnix.BuildLogs
18+import Garnix.Duration
19+import Garnix.GithubInterface.Types
20+import Garnix.Monad hiding
21+ ( getInstallations,
22+ getInstalledOrgs,
23+ getReposInInstallationAccessibleTo,
24+ )
25+import Garnix.Prelude
26+import Garnix.Types hiding (statusCode)
27+import GitHub qualified as GH
28+import GitHub.App.Auth qualified as GHA
29+import GitHub.App.Request qualified as GHA
30+import GitHub.Data.Id
31+import GitHub.Data.Installations qualified as GHA
32+import GitHub.Data.Name qualified as GH
33+import Network.HTTP.Client qualified
34+import Network.HTTP.Client.Internal qualified
35+import Network.HTTP.Conduit (HttpException (..), HttpExceptionContent (..), responseStatus)
36+import Network.HTTP.Types (status500, statusCode)
37+import Network.Wreq qualified as Wreq
38+import Web.JWT qualified as JWT
39+
40+realGithubInterface :: (HasCallStack) => GithubInterface
41+realGithubInterface =
42+ GithubInterface
43+ { _githubInterfaceGetInstallation = \id' -> do
44+ appAuth <- view #githubAppAuth
45+ liftIO $ GHA.mkInstallationAuth appAuth id',
46+ _githubInterfaceGetInstallations = getInstallations,
47+ _githubInterfaceGetGarnixInstallationId = \(GhRepoOwner (GhLogin owner)) (GhRepoName repoName) -> do
48+ appAuth <- view #githubAppAuth
49+ currentTime <- utcTimeToPOSIXSeconds <$> liftIO getCurrentTime
50+ let expireTime = currentTime + (550 :: NominalDiffTime)
51+ let claims =
52+ JWT.JWTClaimsSet
53+ { JWT.iat = JWT.numericDate currentTime,
54+ JWT.exp = JWT.numericDate expireTime,
55+ JWT.iss = JWT.stringOrURI . show . GH.untagId . GHA.aaAppId $ appAuth,
56+ JWT.sub = Nothing,
57+ JWT.nbf = Nothing,
58+ JWT.aud = Nothing,
59+ JWT.jti = Nothing,
60+ JWT.unregisteredClaims = mempty
61+ }
62+ let jwt = JWT.encodeSigned (JWT.EncodeRSAPrivateKey $ GHA.aaPrivateKey appAuth) mempty claims
63+ let url =
64+ "https://api.github.com/repos"
65+ </> cs owner
66+ </> cs repoName
67+ </> "installation"
68+ response <-
69+ retryWreq $ withWreqOptions $ \options -> do
70+ Wreq.getWith
71+ ( options
72+ & Wreq.auth
73+ ?~ Wreq.oauth2Bearer (cs jwt)
74+ & Wreq.checkResponse
75+ ?~ \_ _ -> pure ()
76+ )
77+ url
78+ handleGithubWreqErrors "_githubInterfaceGetGarnixInstallationId" response >>= \mResponse ->
79+ pure $ mResponse >>= (^? key "id" . _Integer),
80+ _githubInterfaceGetAccessToken = \iAuth -> do
81+ mgr <- view #manager
82+ _retryGithubRequest (liftIO (GHA.obtainAccessToken mgr iAuth))
83+ >>= handleGithubRequestErrors "getAccessToken" "garnix" "garnix-github-app"
84+ >>= \case
85+ (GH.OAuth v) -> pure $ GhToken $ cs v
86+ _ -> throw $ OtherError "getAccessToken: unexpected auth token type",
87+ _githubInterfaceGetDefaultBranch = \miAuth owner repo ->
88+ case miAuth of
89+ Nothing -> do
90+ mgr <- view #manager
91+ _retryGithubRequest (liftIO (GH.executeRequestWithMgr' mgr (GH.repositoryR (coerce owner) (coerce repo))))
92+ >>= handleGithubRequestErrors "getDefaultBranch" owner repo
93+ >>= \branch -> pure $ Branch <$> GH.repoDefaultBranch branch
94+ Just iAuth ->
95+ executeAppRequest iAuth (GH.repositoryR (coerce owner) (coerce repo))
96+ >>= handleGithubRequestErrors "getDefaultBranch" owner repo
97+ >>= \branch -> pure $ Branch <$> GH.repoDefaultBranch branch,
98+ _githubInterfaceGetHeadCommit = getHeadCommitForBranch,
99+ _githubInterfaceNewBuildReport = createBuildReportGH,
100+ _githubInterfaceUpdateBuildReport = updateBuildReportGH,
101+ _githubInterfaceDoesRepoFileExist = \commitInfo path -> do
102+ let GhRepoOwner (GhLogin owner') = commitInfo ^. repoInfo . ghRepoOwner
103+ GhRepoName repoName' = commitInfo ^. repoInfo . ghRepoName
104+ CommitHash commit' = commitInfo ^. commit
105+ url =
106+ "https://api.github.com/repos"
107+ </> cs owner'
108+ </> cs repoName'
109+ </> "contents"
110+ </> path
111+ <> "?ref="
112+ <> cs commit'
113+
114+ response <-
115+ retryWreq $ withWreqOptions $ \options ->
116+ Wreq.headWith
117+ ( options
118+ & Wreq.auth
119+ ?~ Wreq.oauth2Token (cs $ getGhToken (commitInfo ^. repoInfo . ghToken))
120+ & Wreq.checkResponse
121+ ?~ \_ _ -> pure ()
122+ )
123+ url
124+ maybe FileDoesntExist (const FileExists)
125+ <$> handleGithubWreqErrors "_githubInterfaceDoesRepoFileExist" response,
126+ _githubInterfaceGetRemote = \commitInfo -> do
127+ pure $ case commitInfo ^. prFromFork of
128+ Just (PrFromFork fromFork) ->
129+ RemoteUrl
130+ $ "https://github.com/"
131+ <> fromFork
132+ <> ".git"
133+ Nothing ->
134+ let GhRepoOwner (GhLogin owner') = commitInfo ^. repoInfo . ghRepoOwner
135+ GhRepoName repo' = commitInfo ^. repoInfo . ghRepoName
136+ in RemoteUrl
137+ $ "https://x-access-token:"
138+ <> getGhToken (commitInfo ^. repoInfo . ghToken)
139+ <> "@github.com/"
140+ <> owner'
141+ <> "/"
142+ <> repo'
143+ <> ".git",
144+ _githubInterfaceGetRepoCollaborators = \iAuth owner@(GhRepoOwner (GhLogin repoOwner)) repo@(GhRepoName repoName) -> do
145+ let req = GH.collaboratorsOnR (GH.mkName Proxy repoOwner) (GH.mkName Proxy repoName) GH.FetchAll
146+ executeAppRequest iAuth req
147+ >>= handleGithubRequestErrors "getRepoCollaborators" owner repo
148+ >>= \case
149+ collaborators ->
150+ pure
151+ $ GhCollaborators
152+ $ GhLogin
153+ . GH.untagName
154+ . GH.simpleUserLogin
155+ <$> Vector.toList collaborators,
156+ _githubInterfaceGetRepoPublicity = \iAuth owner@(GhRepoOwner (GhLogin repoOwner)) repo@(GhRepoName repoName) -> do
157+ let req = GH.repositoryR (GH.mkName Proxy repoOwner) (GH.mkName Proxy repoName)
158+ executeAppRequest iAuth req
159+ >>= handleGithubRequestErrors "getRepoPublicity" owner repo
160+ >>= \r -> pure $ RepoIsPublic $ not $ GH.repoPrivate r,
161+ _githubInterfaceGetInstalledOrgs = getInstalledOrgs,
162+ _githubInterfaceGetReposInInstallationAccessibleTo = getReposInInstallationAccessibleTo,
163+ _githubInterfaceOpenGithubPullRequest = openGithubPullRequestInternal
164+ }
165+
166+getInstallations :: GhToken -> M [GH.Id GHA.Installation]
167+getInstallations (GhToken userToken) = do
168+ let auth = GH.OAuth (cs userToken)
169+ request =
170+ GH.query
171+ [ "user",
172+ "installations"
173+ ]
174+ [("per_page", Just "100")]
175+ result <- executeRequest auth request
176+ case result of
177+ Left githubError
178+ | isTimeout githubError -> throw GithubRequestTimeout
179+ | otherwise -> do
180+ log Error
181+ $ "getInstallations failed. Error was:"
182+ <> show githubError
183+ throw $ OtherError $ show githubError
184+ Right v ->
185+ pure
186+ $ (v :: Aeson.Value)
187+ ^.. key "installations"
188+ . _Array
189+ . each
190+ . key "id"
191+ . _Integral
192+ . to (GH.mkId Proxy)
193+
194+getHeadCommitForBranch :: GhToken -> GhRepoOwner -> GhRepoName -> Branch -> M CommitHash
195+getHeadCommitForBranch (GhToken token) owner repo branch = do
196+ let auth = GH.OAuth (cs token)
197+ request =
198+ GH.query
199+ [ "repos",
200+ cs . getGhLogin . getGhRepoOwner $ owner,
201+ cs . getGhRepoName $ repo,
202+ "branches",
203+ cs branch
204+ ]
205+ []
206+ executeRequest @Aeson.Value auth request
207+ >>= handleGithubRequestErrors "getHeadCommit" owner repo
208+ >>= maybe throwBadFormat pure
209+ . (^? key "commit" . key "sha" . _String . to CommitHash)
210+ where
211+ throwBadFormat :: M a
212+ throwBadFormat = do
213+ let context = show owner <> "/" <> show repo <> "/" <> show branch
214+ log Error $ "_githubInterfaceGetHeadCommit failed for '" <> context <> "': Could not find 'commit.sha'."
215+ throw . OtherError $ "Could not get the HEAD commit for " <> context
216+
217+openGithubPullRequestInternal :: GhRepoOwner -> GhRepoName -> PullRequest -> M PullRequestResult
218+openGithubPullRequestInternal ghOwner@(GhRepoOwner (GhLogin owner)) ghRepo@(GhRepoName repo) pr = do
219+ installationId <- getGarnixInstallationId ghOwner ghRepo
220+ iAuth <- case installationId of
221+ Nothing -> throw $ GarnixAppUnauthorized ghOwner ghRepo
222+ Just id -> getInstallation (Id $ fromInteger id)
223+ let createPullRequest =
224+ GH.CreatePullRequest
225+ { GH.createPullRequestTitle = pr ^. title,
226+ GH.createPullRequestBody = pr ^. body,
227+ GH.createPullRequestHead = getBranch $ pr ^. headBranch,
228+ GH.createPullRequestBase = getBranch $ pr ^. baseBranch
229+ }
230+ request = GH.createPullRequestR (GH.mkName Proxy owner) (GH.mkName Proxy repo) createPullRequest
231+ result <- executeAppRequest iAuth request
232+ case result of
233+ Left githubError
234+ | isTimeout githubError -> throw GithubRequestTimeout
235+ | otherwise -> do
236+ log Error
237+ $ "openGithubPullRequest failed for installation '"
238+ <> owner
239+ <> "/"
240+ <> repo
241+ <> "'. Error was: "
242+ <> show githubError
243+ throw $ OtherError $ show githubError
244+ Right (GH.PullRequest {GH.pullRequestHtmlUrl = GH.URL url}) ->
245+ pure $ PullRequestResult {_pullRequestResultUrl = url}
246+
247+getReposInInstallationAccessibleTo :: GH.Id GHA.Installation -> GhToken -> M [Text]
248+getReposInInstallationAccessibleTo installationId (GhToken userToken) = do
249+ let auth = GH.OAuth (cs userToken)
250+ request =
251+ GH.query
252+ [ "user",
253+ "installations",
254+ show $ GH.untagId installationId,
255+ "repositories"
256+ ]
257+ [("per_page", Just "100")]
258+ result <- executePaginatedRequest auth request
259+ case result of
260+ Left githubError
261+ | isTimeout githubError -> throw GithubRequestTimeout
262+ | otherwise -> do
263+ log Error
264+ $ "getReposInInstallationAccessibleTo failed for installation "
265+ <> show installationId
266+ <> ". Error was: "
267+ <> show githubError
268+ throw $ OtherError $ show githubError
269+ Right vs -> do
270+ repositories <-
271+ forM vs
272+ $ \( v ::
273+ Rec
274+ ( "repositories"
275+ .== [ Rec
276+ ("full_name" .== Text)
277+ ]
278+ )
279+ ) -> do
280+ pure $ map (^. #full_name) $ v ^. #repositories
281+ pure $ concat repositories
282+
283+getInstalledOrgs :: GhToken -> M [GhUserOrgMembership]
284+getInstalledOrgs (GhToken tok) = do
285+ -- This endpoint lists org memberships for which these conditions are met:
286+ --
287+ -- 1. The user is a member of the org,
288+ -- 2. the garnix app is installed in one of the repos of the org.
289+ --
290+ -- Additionally, our github app has to have the permission `"members": "read"`.
291+ -- Without that permission the endpoint returns a 200 response, but the
292+ -- result is always the empty list.
293+ --
294+ -- I think this is *not* documented on github, see
295+ -- https://docs.github.com/en/rest/orgs/members?apiVersion=2022-11-28#list-organization-memberships-for-the-authenticated-user
296+ let endpoint = "https://api.github.com/user/memberships/orgs?per_page=100&state=active"
297+ response <-
298+ retryWreq $ withWreqOptions $ \options ->
299+ Wreq.getWith
300+ ( options
301+ & Wreq.auth
302+ ?~ Wreq.oauth2Token (cs tok)
303+ & Wreq.checkResponse
304+ ?~ \_ _ -> pure ()
305+ )
306+ endpoint
307+
308+ log Informational
309+ $ "getInstalledOrgs status: "
310+ <> cs (show $ response ^. Wreq.responseStatus . Wreq.statusCode)
311+ <> "("
312+ <> cs (response ^. Wreq.responseStatus . Wreq.statusMessage)
313+ <> ")"
314+ <> " response: "
315+ <> cs (response ^. Wreq.responseBody)
316+
317+ when (response ^. Wreq.responseStatus . Wreq.statusCode >= 400)
318+ $ throw
319+ $ OtherError
320+ $ "getInstalledOrgs unexpected status code from github: "
321+ <> show (response ^. Wreq.responseStatus . Wreq.statusCode)
322+
323+ memberships :: [GhUserOrgMembership] <-
324+ aesonDecode
325+ ("response from " <> cs endpoint)
326+ parseJSON
327+ (response ^. Wreq.responseBody . to cs)
328+ log Informational $ "getInstalledOrgs all org members: " <> show memberships
329+ when (length memberships > 99) $ do
330+ throw . OtherError $ "too many installations - pagination not implemented yet"
331+ pure memberships
332+
333+-- * Making Github requests
334+
335+createBuildReportGH :: (HasCallStack) => RepoInfo -> GhRunReport -> M GhRunId
336+createBuildReportGH (RepoInfo iAuth _ owner@(GhRepoOwner (GhLogin repoUser)) repo@(GhRepoName repoName)) report = do
337+ run <- fromRunReport report
338+ res <-
339+ executeAppRequest @Aeson.Value iAuth
340+ $ GH.Command GH.Post ["repos", repoUser, repoName, "check-runs"] (Aeson.encode run)
341+ handleGithubRequestErrors "createBuildReportGH" owner repo res >>= \case
342+ v ->
343+ case v ^? key "id" . _Integer of
344+ Nothing -> throw $ FailedToParseCreateReportResult v
345+ Just v -> pure $ fromInteger v
346+
347+updateBuildReportGH :: (HasCallStack) => GhRunId -> GhRunReport -> RepoInfo -> M ()
348+updateBuildReportGH runId report (RepoInfo iAuth _ owner@(GhRepoOwner (GhLogin repoUser)) repo@(GhRepoName repoName)) = do
349+ run <- fromRunReport report
350+ res <-
351+ executeAppRequest @Aeson.Value iAuth
352+ $ GH.Command GH.Patch ["repos", repoUser, repoName, "check-runs", show runId] (Aeson.encode run)
353+ handleGithubRequestErrors "updateBuildReportGH" owner repo res $> ()
354+
355+fromRunReport :: GhRunReport -> M GhRun
356+fromRunReport (GhRunReport name commit url status' title summary logs') = do
357+ fromRelativeUrl <- relativeUrlConverter
358+ pure
359+ $ GhRun
360+ { _ghRunName = name,
361+ _ghRunHeadSha = commit,
362+ _ghRunDetailsUrl = fromRelativeUrl <$> url,
363+ _ghRunStatus = case status' of
364+ RunReportStatusInProgress -> "in_progress"
365+ _ -> "completed",
366+ _ghRunOutput =
367+ Just
368+ $ RunOutput
369+ { _runOutputTitle = title,
370+ _runOutputSummary = summary,
371+ _runOutputText = processLogsForGithub logs'
372+ },
373+ _ghRunConclusion = case status' of
374+ RunReportStatusInProgress -> Nothing
375+ RunReportStatusSuccess -> Just "success"
376+ RunReportStatusFailure -> Just "failure"
377+ RunReportStatusTimeout -> Just "timed_out"
378+ RunReportStatusCancelled -> Just "cancelled"
379+ }
380+
381+handleGithubRequestErrors :: (HasCallStack) => Text -> GhRepoOwner -> GhRepoName -> Either GH.Error a -> M a
382+handleGithubRequestErrors method owner name = \case
383+ Left githubError | hasStatus (== 404) githubError -> do
384+ log Informational $ "Request " <> method <> " failed with 404. Error: " <> show githubError
385+ throw $ NoSuchRepo {_owner = owner, _name = name}
386+ Left githubError | hasStatus (== 403) githubError -> do
387+ log Informational $ "Request " <> method <> " failed with 403. Error: " <> show githubError
388+ throw $ GarnixAppUnauthorized owner name
389+ Left githubError | isTimeout githubError -> do
390+ log Informational $ "Request " <> method <> " timed out"
391+ throw GithubRequestTimeout
392+ Left err -> do
393+ log Error $ method <> " failed for '" <> show owner <> "/" <> show name <> "'. Error: " <> show err
394+ throw $ OtherError $ show err
395+ Right res -> pure res
396+
397+isTimeout :: GH.Error -> Bool
398+isTimeout (GH.HTTPError (HttpExceptionRequest _ ResponseTimeout)) = True
399+isTimeout (GH.HTTPError (HttpExceptionRequest _ ConnectionTimeout)) = True
400+isTimeout (GH.HTTPError (HttpExceptionRequest _ NoResponseDataReceived)) = True
401+isTimeout (GH.HTTPError (HttpExceptionRequest _ (StatusCodeException r _))) =
402+ statusCode (responseStatus r) == 504
403+isTimeout _ = False
404+
405+handleGithubWreqErrors :: (Show a) => Text -> Wreq.Response a -> M (Maybe a)
406+handleGithubWreqErrors method response = do
407+ let status = response ^. Wreq.responseStatus . Wreq.statusCode
408+ case status of
409+ 403 -> log Notice (method <> ": Github responded with 403") $> Nothing
410+ 404 -> log Notice (method <> ": Github responded with 404") $> Nothing
411+ st | st > 399 -> do
412+ log Critical
413+ $ method
414+ <> ": Github responded with"
415+ <> cs (show $ response ^. Wreq.responseStatus . Wreq.statusCode)
416+ <> "("
417+ <> cs (response ^. Wreq.responseStatus . Wreq.statusMessage)
418+ <> ")"
419+ <> " response: "
420+ <> cs (show $ response ^. Wreq.responseBody)
421+ throw $ OtherError $ "Unexpected Github response for " <> method <> ": " <> show st
422+ _ -> pure . Just $ response ^. Wreq.responseBody
423+
424+executeRequest :: (FromJSON a, Show a) => GH.Auth -> GH.Request rw a -> M (Either GH.Error a)
425+executeRequest auth request = do
426+ mgr <- view #manager
427+ _retryGithubRequest $ liftIO $ GH.executeRequestWithMgr mgr auth request
428+
429+executeAppRequest :: (FromJSON a, Show a) => GHA.InstallationAuth -> GH.Request rw a -> M (Either GH.Error a)
430+executeAppRequest iAuth request = do
431+ mgr <- view #manager
432+ _retryGithubRequest $ liftIO $ GHA.executeAppRequestWithMgr mgr iAuth request
433+
434+executePaginatedRequest :: forall a rw. (FromJSON a) => GH.Auth -> GH.Request rw a -> M (Either GH.Error [a])
435+executePaginatedRequest auth request = runExceptT $ do
436+ firstRequest <- liftIO $ GH.makeHttpRequest (Just auth) request
437+ inner firstRequest
438+ where
439+ inner :: Network.HTTP.Client.Request -> ExceptT GH.Error M [a]
440+ inner rawRequest = do
441+ manager' <- view #manager
442+ response <-
443+ retryExceptT
444+ ( liftIO (Network.HTTP.Client.httpLbs rawRequest manager')
445+ `catch` \(e :: HttpException) -> throwError (GH.HTTPError e)
446+ )
447+ parsed <- unTagged @'GH.MtJSON $ GH.parseResponse rawRequest response
448+ rest <- case GH.getNextUrl response of
449+ Nothing -> pure []
450+ Just nextUrl -> do
451+ nextRequest <-
452+ either (throwError . GH.ParseError . cs) pure
453+ $ Network.HTTP.Client.Internal.setUriEither rawRequest nextUrl
454+ inner nextRequest
455+ pure $ parsed : rest
456+
457+retryPolicy :: (MonadIO m) => RetryPolicyM m
458+retryPolicy =
459+ limitRetries 5 <> fullJitterBackoff (toMicroseconds (fromSeconds @Int 1))
460+
461+_retryWhen :: (Show response) => (response -> Bool) -> M response -> M response
462+_retryWhen shouldRetry action =
463+ retrying retryPolicy shouldRetryInner $ \_status -> do
464+ action
465+ where
466+ shouldRetryInner _status response = do
467+ let retrying = shouldRetry response
468+ when retrying $ do
469+ withTextSpan ("github-api-retry", "true") $ do
470+ log Warning $ show response
471+ pure retrying
472+
473+retryWreq :: (Show body) => M (Wreq.Response body) -> M (Wreq.Response body)
474+retryWreq = _retryWhen $ \response -> (response ^. Wreq.responseStatus) >= status500
475+
476+_retryGithubRequest :: (Show a) => M (Either GH.Error a) -> M (Either GH.Error a)
477+_retryGithubRequest = _retryWhen shouldRetry
478+
479+shouldRetry :: Either GH.Error a -> Bool
480+shouldRetry = \case
481+ Right _ -> False
482+ Left error -> case error of
483+ GH.ParseError _ -> True
484+ GH.JsonError _ -> True
485+ GH.UserError _ -> False
486+ GH.HTTPError error -> case error of
487+ InvalidUrlException _ _ -> False
488+ HttpExceptionRequest _ error -> case error of
489+ StatusCodeException response _ -> statusCode (responseStatus response) >= 500
490+ InvalidRequestHeader _ -> False
491+ InvalidDestinationHost _ -> False
492+ InvalidProxyEnvironmentVariable _ _ -> False
493+ InvalidProxySettings _ -> False
494+ _ -> True
495+
496+retryExceptT :: (MonadIO m) => ExceptT GH.Error m a -> ExceptT GH.Error m a
497+retryExceptT action = retryOnError retryPolicy (\_status error -> pure $ hasStatus (>= 500) error) (const action)
498+
499+hasStatus :: (Int -> Bool) -> GH.Error -> Bool
500+hasStatus check (GH.HTTPError (HttpExceptionRequest _ (StatusCodeException r _))) =
501+ check (statusCode (responseStatus r))
502+hasStatus _ _ = False
backend/src/Garnix/GithubInterface/Types.hsnew file+26
@@ -0,0 +1,26 @@
1+module Garnix.GithubInterface.Types (GhRole (..), GhUserOrgMembership (..)) where
2+
3+import Data.Aeson (withObject, withText, (.:))
4+import Garnix.Prelude
5+import Garnix.Types hiding (Admin)
6+
7+data GhRole = Admin | Other Text
8+ deriving stock (Show, Eq)
9+
10+instance FromJSON GhRole where
11+ parseJSON = withText "GhRole" $ \role -> pure $ case role of
12+ "admin" -> Admin
13+ other -> Other other
14+
15+data GhUserOrgMembership = GhUserOrgMembership
16+ { organizationName :: GhRepoOwner,
17+ role :: GhRole
18+ }
19+ deriving stock (Show, Eq)
20+
21+instance FromJSON GhUserOrgMembership where
22+ parseJSON = withObject "GhUserOrgMembership" $ \v -> do
23+ org <- v .: "organization"
24+ name <- withObject "GhUserOrgMembership.organization" (.: "login") org
25+ role <- parseJSON =<< (v .: "role")
26+ pure $ GhUserOrgMembership name role
backend/src/Garnix/HetznerInterface.hsnew file+190
@@ -0,0 +1,190 @@
1+module Garnix.HetznerInterface
2+ ( realHetznerInterface,
3+ _parseCreateServerResponse,
4+ )
5+where
6+
7+import Control.Lens (_Show)
8+import Data.Aeson qualified as Aeson
9+import Data.Aeson.Lens (key, _Integer, _String)
10+import Data.Text.Lens (unpacked)
11+import Garnix.Hosting.ServerPool.Types
12+import Garnix.Monad
13+import Garnix.Prelude hiding (get, put)
14+import Garnix.Types
15+import Network.HTTP.Client (Response)
16+import Network.Wreq qualified as Wreq
17+
18+realHetznerInterface :: HetznerInterface
19+realHetznerInterface =
20+ HetznerInterface
21+ { _hetznerInterfaceProvisionServer = provisionServer',
22+ _hetznerInterfaceUpdateMetadata = updateMetadata',
23+ _hetznerInterfaceDeleteServer = deleteServer',
24+ _hetznerInterfaceGetServerStatus = getServerStatus'
25+ }
26+
27+-- | Provisions a new server, returning the server info. The server may still
28+-- be in "initializing" state.
29+provisionServer' :: PreprovisionedServerId -> HetznerLocation -> HetznerServerType -> M PreprovisionedServer
30+provisionServer' (PreprovisionedServerId serverId) serverLocation serverType = do
31+ let json =
32+ [aesonQQ|
33+ {
34+ "name": #{show serverId},
35+ "server_type": #{hetznerServerTypeToName serverType},
36+ "location": #{hetznerLocationToName serverLocation},
37+ "start_after_create": true,
38+ "image": "ubuntu-22.04",
39+ "labels": {},
40+ "ssh_keys": [
41+ "garnix_server_ssh_hosting_pub"
42+ ],
43+ "user_data": "#cloud-config\nruncmd:\n- curl https://raw.githubusercontent.com/jfroche/nixos-infect/403911527f0eccabacf00a7924aa6208ab98c3b4/nixos-infect | NIX_INSTALL_URL=https://releases.nixos.org/nix/nix-2.19.3/install PROVIDER=hetznercloud NIX_CHANNEL=nixos-24.11 bash 2>&1 | tee /tmp/infect.log\n",
44+ "automount": false
45+ }
46+ |]
47+ response <- post "https://api.hetzner.cloud/v1/servers" json
48+ let code = response ^. Wreq.responseStatus . Wreq.statusCode
49+ if 200 <= code && code < 300
50+ then do
51+ now <- liftIO getCurrentTime
52+ case _parseCreateServerResponse now (response ^. Wreq.responseBody) of
53+ Nothing ->
54+ throw
55+ $ ProvisioningError
56+ { message =
57+ "Couldn't parse Hetzner server provisioning response. "
58+ <> "(Note that the server may need to be deleted!) "
59+ <> "Response: "
60+ <> show response
61+ }
62+ Just serverInfo -> pure serverInfo
63+ else throw $ ProvisioningError {message = cs $ show response}
64+
65+updateMetadata' ::
66+ RepoInfo ->
67+ DeploymentType ->
68+ Build ->
69+ ServerId ->
70+ HetznerServerId ->
71+ M ()
72+updateMetadata' repoInfo deploymentType build (ServerId serverId) (HetznerServerId hetznerId) = do
73+ let (branchOrPr, branchOrPrValue) = case deploymentType of
74+ BranchDeployment (Branch branch) -> ("branch", branch)
75+ GhPrDeployment (GhPullRequestId prId) -> ("pr", show prId)
76+ let json =
77+ [aesonQQ|
78+ {
79+ "name": #{serverId ^. re hashIdText},
80+ "labels": {
81+ "repo_org": #{repoInfo ^. ghRepoOwner},
82+ "repo_name": #{repoInfo ^. ghRepoName},
83+ $branchOrPr: #{branchOrPrValue},
84+ "nixosConfiguration": #{getPackageName $ build ^. package},
85+ "commit": #{build ^. gitCommit}
86+ }
87+ }
88+ |]
89+ response <- put (cs $ "https://api.hetzner.cloud/v1/servers/" <> show hetznerId) json
90+ let code = response ^. Wreq.responseStatus . Wreq.statusCode
91+ if 200 <= code && code < 300
92+ then pure ()
93+ else do
94+ throw
95+ $ ProvisioningError
96+ { message = "Non-2XX response from Hetzner when updating server metadata. Response: " <> show response
97+ }
98+
99+deleteServer' :: HetznerServerId -> M ()
100+deleteServer' serverId = do
101+ let sId = show (coerce serverId :: Int32)
102+ resp <- delete . cs $ "https://api.hetzner.cloud/v1/servers/" <> sId
103+ case resp ^? Wreq.responseBody . key "action" . key "status" of
104+ Just "error" -> do
105+ log Error $ "Hetzner deletion failed with: " <> show (resp ^. Wreq.responseBody)
106+ throw $ ProvisioningError $ "Could not delete server. Response: " <> show resp
107+ _ -> pure ()
108+
109+getServerStatus' :: HetznerServerId -> M Text
110+getServerStatus' hetznerId = do
111+ response <-
112+ get
113+ . cs
114+ $ "https://api.hetzner.cloud/v1/servers/"
115+ <> show (coerce hetznerId :: Int32)
116+ let code = response ^. Wreq.responseStatus . Wreq.statusCode
117+ if 200 <= code && code < 300
118+ then
119+ let status' =
120+ response
121+ ^? Wreq.responseBody
122+ . key "server"
123+ . key "status"
124+ . _String
125+ in case status' of
126+ Just st -> pure st
127+ Nothing -> throw $ ProvisioningError $ "getServerStatus': Missing 'status' field. Response: " <> show response
128+ else throw $ ProvisioningError $ "getServerStatus': non-2XX status code. Response: " <> show response
129+
130+-- * Response parsing
131+
132+_parseCreateServerResponse :: UTCTime -> LazyByteString -> Maybe PreprovisionedServer
133+_parseCreateServerResponse created bs = bs ^? key "server" >>= parsePreprovisionedServer created
134+
135+parsePreprovisionedServer :: UTCTime -> Aeson.Value -> Maybe PreprovisionedServer
136+parsePreprovisionedServer created v = do
137+ serverId <- v ^? key "name" . _String . unpacked . _Show
138+ hetznerId <- HetznerServerId . fromIntegral <$> v ^? key "id" . _Integer
139+ ip4 <- v ^? key "public_net" . key "ipv4" . key "ip" . _String
140+ ip6 <- v ^? key "public_net" . key "ipv6" . key "ip" . _String
141+ pure
142+ $ PreprovisionedServer
143+ { _preprovisionedServerId = PreprovisionedServerId serverId,
144+ _preprovisionedServerHetznerServerId = hetznerId,
145+ _preprovisionedServerIpv4Addr = ip4,
146+ _preprovisionedServerIpv6Addr = ip6,
147+ _preprovisionedServerCreatedAt = created,
148+ _preprovisionedServerReadyAt = Nothing
149+ }
150+
151+-- We want to use *our* created time, not hetzner's.
152+
153+-- * Hetzner request helpers
154+
155+put :: String -> Aeson.Value -> M (Response LazyByteString)
156+put req body = do
157+ opts <- wreqOptions
158+ liftIO (Wreq.putWith opts req body) `catch` \(e :: SomeException) ->
159+ throw $ ProvisioningError $ "Error from PUT request: " <> show e
160+
161+post :: String -> Aeson.Value -> M (Response LazyByteString)
162+post req body = do
163+ opts <- wreqOptions
164+ liftIO (Wreq.postWith opts req body) `catch` \(e :: SomeException) ->
165+ throw $ ProvisioningError $ "Error from POST request: " <> show e
166+
167+get :: String -> M (Response LazyByteString)
168+get req = do
169+ opts <- wreqOptions
170+ liftIO (Wreq.getWith opts req) `catch` \(e :: SomeException) ->
171+ throw $ ProvisioningError $ "Error from GET request: " <> show e
172+
173+delete :: String -> M (Response LazyByteString)
174+delete req = do
175+ opts <- wreqOptions
176+ liftIO (Wreq.deleteWith opts req) `catch` \(e :: SomeException) ->
177+ throw $ ProvisioningError $ "Error from DELETE request: " <> show e
178+
179+wreqOptions :: M Wreq.Options
180+wreqOptions = do
181+ token <- view #hetznerToken
182+ mgr <- view #manager
183+ pure
184+ $ Wreq.defaults
185+ & Wreq.manager
186+ .~ Right mgr
187+ & Wreq.auth
188+ ?~ Wreq.oauth2Bearer token
189+ & Wreq.checkResponse
190+ ?~ \_ _ -> pure ()
backend/src/Garnix/Hosting/Deploy.hsnew file+482
@@ -0,0 +1,482 @@
1+module Garnix.Hosting.Deploy
2+ ( rolloutNewServerVersion,
3+ startServer,
4+ stopUnusedServers,
5+ stopServer,
6+ redeployServer,
7+ -- Exported for tests
8+ _costBreakdown,
9+ DeployCounts (..),
10+ checkDeployPlan,
11+ )
12+where
13+
14+import Control.Concurrent.Async.Lifted qualified as Async
15+import Control.Lens (traversed)
16+import Cradle
17+import Data.Map (Map)
18+import Data.Map qualified as Map
19+import Data.Maybe (mapMaybe)
20+import Data.Text qualified as T
21+import Garnix.API.Keys (getRepoKeys)
22+import Garnix.BuildLogs.Types (mkLogLine)
23+import Garnix.DB qualified as DB
24+import Garnix.Duration
25+import Garnix.Entitlements qualified as Entitlements
26+import Garnix.Hosting.ServerPool qualified as ServerPool
27+import Garnix.Hosting.ServerPool.Types
28+import Garnix.Monad
29+import Garnix.Monad.Polling (PollingConfig (PollingConfig), withPolling)
30+import Garnix.Monad.SubProcess (runSubProcess)
31+import Garnix.MonetaryCost
32+import Garnix.Nix.StorePath (withStorePath)
33+import Garnix.Nix.Types
34+import Garnix.Prelude
35+import Garnix.Reporters.Utils (withRunReporter)
36+import Garnix.Request
37+import Garnix.Types
38+import Garnix.YamlConfig
39+
40+-- | Deploys new server versions, deletes old ones.
41+rolloutNewServerVersion ::
42+ Reporter ->
43+ CommitInfo ->
44+ DeploymentType ->
45+ M [ServerInfo]
46+rolloutNewServerVersion reporter commitInfo deploymentType =
47+ withTextSpan ("deployment_type", fromDeploymentType (const "branch") (const "pr") deploymentType)
48+ $ (<?> "Rolling out new servers")
49+ $ do
50+ plan <- getDeployPlan reporter commitInfo deploymentType
51+ executeDeployPlan reporter commitInfo plan deploymentType
52+
53+stopUnusedServers :: M ()
54+stopUnusedServers = do
55+ (PrHostList runningServers) <- DB.getShutdownCandidates
56+ heartbeat <- DB.getRecentHeartbeats
57+ let toSpinDown = filter (haveNotSentHeartbeat heartbeat) runningServers
58+ traverse_ (\s -> stopServer (s ^. serverId) (s ^. hetznerId)) toSpinDown
59+ where
60+ haveNotSentHeartbeat :: [Text] -> Host -> Bool
61+ haveNotSentHeartbeat heartbeats host =
62+ let hostName = hostToDomainName host <> ".garnix.me"
63+ in hostName `notElem` heartbeats
64+
65+getDeployPlan ::
66+ Reporter ->
67+ CommitInfo ->
68+ DeploymentType ->
69+ M DeployPlan
70+getDeployPlan reporter commitInfo deploymentType = do
71+ withErrorReporter reporter commitInfo $ do
72+ cfg <- getConfig
73+ let wantedPackagesMapping :: Map PackageName (ServerTier, Bool) = Map.fromList $ case deploymentType of
74+ BranchDeployment thisBranch -> flip mapMaybe (cfg ^. serverSection)
75+ $ \s -> case s ^. deploySection of
76+ OnBranch branch serverTier isPrimary | branch == thisBranch -> Just (s ^. configuration, (serverTier, isPrimary))
77+ _ -> Nothing
78+ GhPrDeployment _prId ->
79+ (cfg ^. serverSection)
80+ & filter (\s -> s ^. deploySection == OnPullRequest)
81+ & map (\s -> (s ^. configuration, (def, False)))
82+ let wantedPackages = Map.keys wantedPackagesMapping
83+ existing <- DB.getRunningServersOf (commitInfo ^. repoInfo) deploymentType
84+ wantedBuilds <- withPolling (PollingConfig (fromSeconds @Int 2) (fromHours @Int 2)) $ do
85+ builds <- DB.getLatestBuildsMatching (commitInfo ^. repoInfo) (commitInfo ^. commit)
86+ let overallPackageFinished = case find (\build -> build ^. packageType == TypeOverall) builds of
87+ Nothing -> False
88+ Just b -> isJust (b ^. status)
89+ wantedAndStarted =
90+ filter
91+ ( \build ->
92+ build
93+ ^. package
94+ `elem` wantedPackages
95+ && build
96+ ^. packageType
97+ == TypeNixosConfiguration
98+ )
99+ builds
100+ wantedAndFinished = filter (\build -> isJust (build ^. status)) wantedAndStarted
101+ wantedAndNotStarted = filter (`notElem` wantedAndStarted ^.. each . package) wantedPackages
102+ uploadedAllBuilds = all (\b -> b ^. uploadedToCache == Just True) wantedAndFinished
103+ when (overallPackageFinished && not (null wantedAndNotStarted)) $ do
104+ throw $ DeploymentWantsNixosConfigurationsThatDontExist wantedAndNotStarted
105+ pure
106+ $ if sort wantedPackages == sort (wantedAndFinished ^.. each . package) && uploadedAllBuilds
107+ then Just wantedAndFinished
108+ else Nothing
109+
110+ let toRedeploy =
111+ [ (server, build)
112+ | server <- existing,
113+ build <- wantedBuilds,
114+ server ^. buildPersistenceName == build ^. persistenceName,
115+ isJust (build ^. persistenceName)
116+ ]
117+ toSpinDown = filter (`notElem` (fst <$> toRedeploy)) existing
118+ toSpinUp <-
119+ wantedBuilds
120+ & filter (`notElem` (snd <$> toRedeploy))
121+ & mapM
122+ ( \build -> case Map.lookup (build ^. package) wantedPackagesMapping of
123+ Just (serverTier, domainIsPrimary) -> pure $ ServerToSpinUp {serverTier, build, domainIsPrimary}
124+ Nothing -> throw $ OtherError "impossible: wantedPackagesMap should contain all deployable packages"
125+ )
126+ let plan = DeployPlan toSpinDown toSpinUp toRedeploy
127+ unless (null toSpinUp) $ do
128+ checkDeployPlan (commitInfo ^. repoInfo) deploymentType plan
129+ pure plan
130+
131+-- | Will catch all exceptions, create a new reporter to report the error to the user and rethrow.
132+withErrorReporter :: Reporter -> CommitInfo -> M a -> M a
133+withErrorReporter reporter commitInfo action = do
134+ checkResult <- try action
135+ case checkResult of
136+ Left error -> do
137+ run <- DB.newRun "deployment plan" commitInfo
138+ runReporter <- createNewRun reporter (ReportRun run)
139+ reportLogs runReporter (mkLogLine $ userMessage $ toErrorDetails error)
140+ reportComplete runReporter RunReportStatusFailure
141+ rethrow error
142+ Right a -> pure a
143+
144+checkDeployPlan ::
145+ RepoInfo ->
146+ DeploymentType ->
147+ DeployPlan ->
148+ M ()
149+checkDeployPlan repoInfo deploymentType plan = do
150+ checkEntitlement (ghPrDeployment deploymentType) plan (repoInfo ^. ghRepoOwner)
151+ checkServerTiers repoInfo plan
152+ checkSubdomainValidity repoInfo deploymentType $ fmap (^. #build) $ plan ^. #toSpinUp
153+ checkAllBuildsSucceeded plan
154+
155+data DeployCounts = DeployCounts
156+ { includedInPlan :: Int64,
157+ notIncludedInPlan :: Int64
158+ }
159+ deriving (Generic, Show)
160+
161+totalCount :: DeployCounts -> Int64
162+totalCount d = d ^. #includedInPlan + d ^. #notIncludedInPlan
163+
164+costForTierDeployment :: ServerTier -> DeployCounts -> MonetaryCost
165+costForTierDeployment tier deployCounts = serverTierToCost tier `multiplyCost` (deployCounts ^. #notIncludedInPlan)
166+
167+totalDeploymentCost :: Map ServerTier DeployCounts -> MonetaryCost
168+totalDeploymentCost =
169+ foldr' addCost (usd 0)
170+ . map (uncurry costForTierDeployment)
171+ . Map.toList
172+
173+_costBreakdown :: Map ServerTier DeployCounts -> [Text]
174+_costBreakdown m =
175+ m
176+ & Map.toList
177+ & filter (\(_, d) -> totalCount d > 0)
178+ & map
179+ ( \(tier, deployCounts) ->
180+ let tierCostStr = formatCost (serverTierToCost tier)
181+ in serverTierToText tier
182+ <> " (x"
183+ <> show (totalCount deployCounts)
184+ <> ") = "
185+ <> formatCost (costForTierDeployment tier deployCounts)
186+ <> case deployCounts of
187+ DeployCounts 0 1 -> ""
188+ DeployCounts 0 _ -> " (" <> tierCostStr <> " each)"
189+ DeployCounts n 0 -> " (" <> show n <> " included in plan)"
190+ DeployCounts n 1 -> " (" <> show n <> " included in plan)"
191+ DeployCounts inc notInc ->
192+ " (" <> show inc <> " included in plan, " <> show notInc <> " not included at " <> tierCostStr <> " each)"
193+ )
194+
195+checkEntitlement :: Maybe GhPullRequestId -> DeployPlan -> GhRepoOwner -> M ()
196+checkEntitlement mPrId plan repoOwner = do
197+ hostingLimits <- Entitlements.getHosting repoOwner
198+ case mPrId of
199+ Just _ -> do
200+ when (hostingLimits ^. #maxPrDeploymentTime == emptyDuration)
201+ $ throw
202+ $ EntitlementError
203+ $ "Sorry, hosting is not allowed for "
204+ <> getGhLogin (getGhRepoOwner repoOwner)
205+ <> "."
206+ usedMinutes <- DB.getPrDeployDurationForOwner repoOwner
207+ when (usedMinutes > hostingLimits ^. #maxPrDeploymentTime)
208+ $ throw
209+ $ EntitlementError "Sorry, you have exhausted all your PR deployment minutes."
210+ Nothing -> do
211+ currentHosts <- DB.getRunningBranchServersForOwner repoOwner
212+ let allHostsAfterDeploy :: Map ServerTier DeployCounts =
213+ currentHosts
214+ & adjustCount succ (plan ^.. #toSpinUp . each . #serverTier)
215+ & adjustCount pred (plan ^.. #toSpinDown . each . tier)
216+ & Map.mapWithKey
217+ ( \tier count ->
218+ let includedInPlan =
219+ if tier == serverTierIncludedWithPlans
220+ then min count (hostingLimits ^. #planIncludedBranchDeploymentHosts)
221+ else 0
222+ in DeployCounts
223+ { includedInPlan,
224+ notIncludedInPlan = count - includedInPlan
225+ }
226+ )
227+ let costAfterDeploy = totalDeploymentCost allHostsAfterDeploy
228+ when (costAfterDeploy > hostingLimits ^. #extraBranchHostingSpend) $ do
229+ throw
230+ $ EntitlementError
231+ $ "Deploying this would result in "
232+ <> formatCost costAfterDeploy
233+ <> " extra monthly server cost above your plan. "
234+ <> ( if hostingLimits ^. #extraBranchHostingSpend > usd 0
235+ then "However, you have configured a max spend of " <> formatCost (hostingLimits ^. #extraBranchHostingSpend) <> " per month on deployed hosts. "
236+ else "You have not configured any additional spending budget on deployed hosts. "
237+ )
238+ <> "You can configure this spending limit on your account page at garnix.io.\n\n"
239+ <> "Here is a breakdown of what would be deployed by this commit and the associated costs:\n"
240+ <> T.unlines (map (" " <>) $ _costBreakdown allHostsAfterDeploy)
241+ where
242+ adjustCount :: (Int64 -> Int64) -> [ServerTier] -> Map ServerTier Int64 -> Map ServerTier Int64
243+ adjustCount adjust servers map = foldr' (Map.alter (Just . adjust . fromMaybe 0)) map servers
244+
245+checkServerTiers :: RepoInfo -> DeployPlan -> M ()
246+checkServerTiers repoInfo plan = do
247+ hostingLimits <- Entitlements.getHosting (repoInfo ^. ghRepoOwner)
248+ unless (hostingLimits ^. #largerServers) $ do
249+ forM_ (plan ^.. #toSpinUp . each . #serverTier) $ \serverTier ->
250+ when (serverTier /= def) $ do
251+ throw $ EntitlementError $ "Only server tier `" <> serverTierToText def <> "` supported."
252+
253+checkSubdomainValidity ::
254+ RepoInfo ->
255+ DeploymentType ->
256+ [Build] ->
257+ M ()
258+checkSubdomainValidity
259+ (RepoInfo _ _ (GhRepoOwner (GhLogin repoOwner')) (GhRepoName repoName'))
260+ deploymentType
261+ wantedServers = do
262+ unless (isValidSubdomainString repoOwner')
263+ $ throw
264+ $ NameIsNotValidSubdomain RepoOwnerSubdomain repoOwner'
265+ unless (isValidSubdomainString repoName')
266+ $ throw
267+ $ NameIsNotValidSubdomain RepoNameSubdomain repoName'
268+ case deploymentType of
269+ BranchDeployment (Branch branch') -> do
270+ unless (isValidSubdomainString branch')
271+ $ throw
272+ $ NameIsNotValidSubdomain BranchSubdomain branch'
273+ GhPrDeployment _ -> pure ()
274+ forM_ wantedServers $ \build -> do
275+ let packageName = getPackageName $ build ^. package
276+ unless (isValidSubdomainString packageName)
277+ $ throw
278+ $ NameIsNotValidSubdomain PackageNameSubdomain packageName
279+ let persistence = build ^. persistenceName
280+ forM_ persistence $ \persistenceName -> do
281+ unless (isValidSubdomainString persistenceName)
282+ $ throw
283+ $ NameIsNotValidSubdomain PersistenceNameSubdomain persistenceName
284+
285+checkAllBuildsSucceeded :: DeployPlan -> M ()
286+checkAllBuildsSucceeded plan = do
287+ forM_ (plan ^.. #toSpinUp . each . #build) $ \build -> do
288+ let packageName = getPackageName $ build ^. package
289+ case build ^. status of
290+ Just Success -> pure ()
291+ Just Failure -> throw $ OtherError $ packageName <> " failed"
292+ Just Timeout -> throw $ OtherError $ packageName <> " timed out"
293+ Just Cancelled -> throw $ OtherError $ packageName <> " cancelled"
294+ -- this should be unreachable since checkAllBuildsSucceeded is only called after waiting for all builds to finish
295+ Nothing -> throw $ OtherError $ packageName <> " has no status"
296+
297+executeDeployPlan ::
298+ Reporter ->
299+ CommitInfo ->
300+ DeployPlan ->
301+ DeploymentType ->
302+ M [ServerInfo]
303+executeDeployPlan = curry4
304+ $ mockable #executeDeployPlanMock
305+ $ \(reporter, commitInfo, DeployPlan currentServers wantedServers redeployServers, deploymentType) -> do
306+ serverInfos <- Async.mapConcurrently (startServer reporter commitInfo deploymentType) wantedServers
307+ redeployedServers <- Async.mapConcurrently (uncurry (redeployServer reporter commitInfo deploymentType)) redeployServers
308+ deployedServerInfos <- toggleServerFlags serverInfos currentServers <?> "Toggling servers ready flags."
309+ Async.mapConcurrently_ (\s -> stopServer (s ^. id) (s ^. hetznerServerId)) currentServers
310+ return (deployedServerInfos <> redeployedServers)
311+
312+stopServer :: ServerId -> HetznerServerId -> M ()
313+stopServer serverId hetznerId = do
314+ deleteServer hetznerId
315+ DB.deleteServerDB serverId
316+
317+startServer ::
318+ Reporter ->
319+ CommitInfo ->
320+ DeploymentType ->
321+ ServerToSpinUp ->
322+ M ServerInfo
323+startServer = curry4
324+ $ mockable #startServerMock
325+ $ \(reporter, commitInfo, deploymentType, serverToSpinUp) -> do
326+ run <- DB.newRun ("deployment " <> getPackageName (serverToSpinUp ^. #build . package)) commitInfo
327+ withRunReporter reporter (ReportRun run) $ \runReporter -> do
328+ serverInfo <- ServerPool.createServer (commitInfo ^. repoInfo) deploymentType serverToSpinUp
329+ (serverInfo, stderr) <-
330+ setupServer (commitInfo ^. repoInfo) (serverToSpinUp ^. #build) serverInfo `whenError` \error -> do
331+ let logs = showPretty (err error)
332+ DB.appendToServerDeployLog (serverInfo ^. id) logs
333+ let logs =
334+ T.unlines
335+ [ "Server has been successfully deployed to: https://"
336+ <> getPackageName (serverToSpinUp ^. #build . package)
337+ <> "."
338+ <> fromDeploymentType getBranch (("pull-" <>) . show . getGhPullRequestId) deploymentType
339+ <> "."
340+ <> getGhRepoName (commitInfo ^. repoInfo . ghRepoName)
341+ <> "."
342+ <> getGhLogin (getGhRepoOwner (commitInfo ^. repoInfo . ghRepoOwner))
343+ <> ".garnix.me",
344+ "ipv4: " <> serverInfo ^. ipv4Addr,
345+ "ipv6: " <> serverInfo ^. ipv6Addr,
346+ "",
347+ "logs:"
348+ <> stderr
349+ ]
350+ DB.appendToServerDeployLog (serverInfo ^. id) logs
351+ reportLogs runReporter (mkLogLine logs)
352+ reportComplete runReporter RunReportStatusSuccess
353+ pure serverInfo
354+
355+redeployServer :: Reporter -> CommitInfo -> DeploymentType -> ServerInfo -> Build -> M ServerInfo
356+redeployServer reporter commitInfo deploymentType serverInfo build = do
357+ withStorePath build "out" $ \case
358+ Nothing -> throw $ OtherError "Store path is missing"
359+ Just storePath -> do
360+ run <- DB.newRun ("redeployment " <> getPackageName (build ^. package)) commitInfo
361+ withRunReporter reporter (ReportRun run) $ \runReporter -> do
362+ copyClosure (SshUser "garnix") serverInfo storePath <?> "Copying closure for redeployment"
363+ deploymentLogs <- switchToConfiguration (SshUser "garnix") serverInfo storePath <?> "Switching to redeployment configuration"
364+ now <- liftIO getCurrentTime
365+ let serverInfo' =
366+ serverInfo
367+ & configurationBuildId
368+ .~ build
369+ ^. id
370+ & readyAt
371+ ?~ now
372+ DB.updateServerPostDeploy serverInfo' <?> "Updating DB about the redeployed server"
373+ let logs =
374+ T.unlines
375+ [ "Server has been successfully redeployed to: https://"
376+ <> getPackageName (build ^. package)
377+ <> "."
378+ <> fromDeploymentType getBranch (("pull-" <>) . show . getGhPullRequestId) deploymentType
379+ <> "."
380+ <> getGhRepoName (commitInfo ^. repoInfo . ghRepoName)
381+ <> "."
382+ <> getGhLogin (getGhRepoOwner (commitInfo ^. repoInfo . ghRepoOwner))
383+ <> ".garnix.me",
384+ "ipv4: " <> serverInfo ^. ipv4Addr,
385+ "ipv6: " <> serverInfo ^. ipv6Addr,
386+ "",
387+ "logs:"
388+ <> deploymentLogs
389+ ]
390+ reportLogs runReporter (mkLogLine logs)
391+ reportComplete runReporter RunReportStatusSuccess
392+ pure serverInfo'
393+
394+-- | Toggles the wanted servers to be ready and the current servers to end.
395+toggleServerFlags :: [ServerInfo] -> [ServerInfo] -> M [ServerInfo]
396+toggleServerFlags wanted current = do
397+ currentTime <- liftIO getCurrentTime
398+ let wanted' = wanted & traversed . readyAt ?~ currentTime
399+ current' = current & traversed . endedAt ?~ currentTime
400+ DB.pgTransaction $ traverse_ DB.updateServerPostDeploy (wanted' <> current')
401+ pure wanted'
402+
403+-- | Provisions a *new* server, and deploys the given NixOS configuration there.
404+--
405+-- Does *not* delete the old server.
406+setupServer ::
407+ RepoInfo ->
408+ Build ->
409+ ServerInfo ->
410+ M (ServerInfo, Text)
411+setupServer = curry3
412+ $ mockable #setupServerMock
413+ $ \(repoInfo, build, serverInfo) -> do
414+ withStorePath build "out" $ \case
415+ Nothing -> throw $ OtherError "Store path is missing"
416+ Just storePath -> do
417+ copyKeys repoInfo serverInfo
418+ copyClosure (SshUser "root") serverInfo storePath <?> "Copying closure"
419+ stderr <- switchToConfiguration (SshUser "root") serverInfo storePath <?> "Switching to configuration"
420+ return (serverInfo, stderr)
421+
422+newtype SshUser = SshUser Text
423+
424+copyClosure :: SshUser -> ServerInfo -> StorePath -> M ()
425+copyClosure (SshUser user) server storePath = do
426+ (ip, sshArgs) <- ServerPool.sshArgsFor server
427+ retryingFor (fromMinutes @Int 1) $ do
428+ runSubProcess
429+ $ cmd "nix-copy-closure"
430+ & addArgs ["--to", user <> "@" <> ip, cs storePath]
431+ & modifyEnvVar "NIX_SSHOPTS" (const $ Just $ cs $ T.intercalate " " sshArgs)
432+
433+copyKeys :: RepoInfo -> ServerInfo -> M ()
434+copyKeys repoInfo server = do
435+ (ip, sshArgs) <- ServerPool.sshArgsFor server
436+ let keyLocation = "/var/garnix/keys/repo-key"
437+ let doRemotely cmd =
438+ runSubProcess
439+ $ Cradle.cmd "ssh"
440+ & addArgs (sshArgs <> ["root@" <> cs ip, cmd])
441+ doRemotely $ "mkdir -p " <> cs (takeDirectory keyLocation)
442+ (_, privKey) <-
443+ getRepoKeys (repoInfo ^. ghRepoOwner) (repoInfo ^. ghRepoName)
444+ <?> "Get private keys"
445+ repoSecretsKey <- view #repoSecretsEncryptionKeyPath
446+ exportResult <-
447+ liftIO
448+ ( exportKeys
449+ ExportKeysOpts
450+ { privateKey = privKey,
451+ ipAddr = ip,
452+ targetPath = keyLocation,
453+ sshArgs
454+ }
455+ repoSecretsKey
456+ )
457+ <?> "Export private keys to server"
458+ whenIs _Left exportResult $ throw . ProvisioningError
459+ doRemotely $ "chmod 400 " <> cs keyLocation
460+
461+switchToConfiguration :: SshUser -> ServerInfo -> StorePath -> M Text
462+switchToConfiguration (SshUser user) server storePath = do
463+ (ip, sshArgs) <- ServerPool.sshArgsFor server
464+ StderrRaw stderr <-
465+ withError (errLens %~ toActivationError server)
466+ $ runSubProcess
467+ $ cmd "ssh"
468+ & addArgs
469+ ( sshArgs
470+ <> [ user <> "@" <> ip,
471+ "sudo",
472+ cs $ cs storePath </> "bin/switch-to-configuration",
473+ "switch"
474+ ]
475+ )
476+ & silenceStdout
477+ pure $ cs stderr
478+ where
479+ toActivationError serverInfo = \case
480+ RunProcessError {stdErr} ->
481+ ActivationError serverInfo stdErr
482+ error -> error
backend/src/Garnix/Hosting/Helpers.hsnew file+227
@@ -0,0 +1,227 @@
1+module Garnix.Hosting.Helpers
2+ ( getRunningAndRecentServersForOwners,
3+ getBranchDeploymentBillingLineItems,
4+ calculateBranchDeploymentBillingLineItems,
5+ groupIdentifierToLineItemDescription,
6+ BranchServerGroupIdentifier (..),
7+ BranchServerBillingLineItem (..),
8+ RunningServer (..),
9+ )
10+where
11+
12+import Data.Map qualified as Map
13+import Data.Maybe (mapMaybe)
14+import Data.Ord
15+import Database.PostgreSQL.Typed (pgSQL)
16+import Garnix.DB qualified as DB
17+import Garnix.Duration
18+import Garnix.Hosting.ServerPool.Types
19+import Garnix.Monad
20+import Garnix.MonetaryCost
21+import Garnix.Prelude
22+import Garnix.Types
23+
24+data ServerStatus
25+ = Online
26+ | Failed
27+ | Booting
28+ | Ended
29+ deriving stock (Eq, Show, Generic)
30+
31+instance ToJSON ServerStatus where
32+ toJSON = ourToJSON
33+
34+data RunningServer = RunningServer
35+ { _runningServerId :: ServerId,
36+ _runningServerType :: DeploymentType,
37+ _runningServerStatus :: ServerStatus,
38+ _runningServerRepoOwner :: GhRepoOwner,
39+ _runningServerRepoName :: GhRepoName,
40+ _runningServerPackageName :: PackageName,
41+ _runningServerCreatedAt :: Maybe UTCTime,
42+ _runningServerConfigurationBuildId :: BuildId,
43+ _runningServerCommit :: CommitHash,
44+ _runningServerIpv4 :: Maybe Text,
45+ _runningServerDeployLogs :: Text
46+ }
47+ deriving stock (Eq, Show, Generic)
48+
49+instance ToJSON RunningServer where
50+ toJSON = ourToJSON
51+
52+getRunningAndRecentServersForOwners :: [GhRepoOwner] -> M [RunningServer]
53+getRunningAndRecentServersForOwners owners = do
54+ mapMaybe
55+ ( \(id, pr, branch, readyAt, endedAt, repoUser, repoName, packageName, createdAt, buildId, commit, ipv4, logs) -> do
56+ typ <- serverDeploymentType pr branch
57+ status <- serverStatus readyAt endedAt
58+ pure $ RunningServer id typ status repoUser repoName packageName createdAt buildId commit ipv4 logs
59+ )
60+ <$> DB.pgQuery
61+ [pgSQL|
62+ SELECT
63+ servers.id,
64+ servers.pull_request,
65+ builds.branch,
66+ servers.ready_at,
67+ servers.ended_at,
68+ builds.repo_user,
69+ builds.repo_name,
70+ builds.package,
71+ servers.created_at,
72+ servers.configuration_build_id,
73+ builds.git_commit,
74+ servers.ipv4,
75+ servers.deploy_logs
76+ FROM servers
77+ INNER JOIN builds
78+ ON servers.configuration_build_id = builds.id
79+ WHERE builds.repo_user = ANY(${owners})
80+ AND (servers.ended_at IS NULL OR servers.ended_at > now() - interval '24' hour)
81+ ORDER BY servers.created_at DESC
82+ |]
83+ where
84+ serverDeploymentType :: Maybe GhPullRequestId -> Maybe Branch -> Maybe DeploymentType
85+ serverDeploymentType pr branch = case (pr, branch) of
86+ (Just prId, _) -> Just $ GhPrDeployment prId
87+ (Nothing, Just branch) -> Just $ BranchDeployment branch
88+ (Nothing, Nothing) -> Nothing
89+ serverStatus :: Maybe UTCTime -> Maybe UTCTime -> Maybe ServerStatus
90+ serverStatus readyAt endedAt = case (readyAt, endedAt) of
91+ (Nothing, Nothing) -> Just Booting
92+ (Just _, Nothing) -> Just Online
93+ (Just _, Just _) -> Just Ended
94+ (Nothing, Just _) -> Nothing
95+
96+data BranchServerGroupIdentifier = BranchServerGroupIdentifier
97+ { owner :: GhRepoOwner,
98+ repo :: GhRepoName,
99+ package :: PackageName,
100+ serverTier :: ServerTier
101+ }
102+ deriving stock (Eq, Ord, Show, Generic)
103+
104+groupIdentifierToLineItemDescription :: BranchServerGroupIdentifier -> Text
105+groupIdentifierToLineItemDescription group =
106+ getGhLogin (getGhRepoOwner $ group ^. #owner)
107+ <> "/"
108+ <> getGhRepoName (group ^. #repo)
109+ <> "#"
110+ <> getPackageName (group ^. #package)
111+ <> " "
112+ <> serverTierToText (group ^. #serverTier)
113+
114+data BranchServerBillingLineItem = BranchServerBillingLineItem
115+ { group :: BranchServerGroupIdentifier,
116+ includedInPlan :: Bool,
117+ usedTime :: Duration,
118+ cost :: MonetaryCost
119+ }
120+ deriving stock (Eq, Ord, Show, Generic)
121+
122+getBranchDeploymentBillingLineItems :: Int64 -> UTCTime -> UTCTime -> GhRepoOwner -> M [BranchServerBillingLineItem]
123+getBranchDeploymentBillingLineItems numFreeServers periodStart periodEnd owner = do
124+ now <- liftIO getCurrentTime
125+ servers <-
126+ -- Note: the `ready_at` and `ended_at` in the WHERE clause is only for
127+ -- limiting the returned results, billing period calculation actually
128+ -- happens in haskell in the map below.
129+ DB.pgQuery
130+ [pgSQL|
131+ SELECT
132+ ready_at,
133+ ended_at,
134+ server_tier,
135+ builds.repo_user,
136+ builds.repo_name,
137+ builds.package
138+ FROM servers
139+ INNER JOIN builds
140+ ON servers.configuration_build_id = builds.id
141+ WHERE builds.repo_user = ${owner}
142+ AND ready_at <= ${periodEnd}
143+ AND (ended_at IS NULL OR ended_at >= ${periodStart})
144+ |]
145+ calculateBranchDeploymentBillingLineItems numFreeServers periodStart periodEnd
146+ <$> mapM
147+ ( \(readyAt :: Maybe UTCTime, endTime :: Maybe UTCTime, serverTier :: ServerTier, repoOwner :: GhRepoOwner, repoName :: GhRepoName, package :: PackageName) -> do
148+ startTime <- case readyAt of
149+ Just s -> pure s
150+ Nothing -> throw $ OtherError "getBranchDeploymentBillingLineItems: Impossible: readyAt is null"
151+ let normalizedStart = max periodStart startTime
152+ let normalizedEnd = min periodEnd $ fromMaybe now endTime
153+ let group = BranchServerGroupIdentifier repoOwner repoName package serverTier
154+ let usedTime = normalizedEnd `diffTime` normalizedStart
155+ pure (group, usedTime)
156+ )
157+ servers
158+
159+calculateBranchDeploymentBillingLineItems :: Int64 -> UTCTime -> UTCTime -> [(BranchServerGroupIdentifier, Duration)] -> [BranchServerBillingLineItem]
160+calculateBranchDeploymentBillingLineItems numFreeServers periodStart periodEnd usedTimeByGroup =
161+ let totalFreeTime = periodDuration `multiplyDuration` numFreeServers
162+ in fst
163+ $ foldl'
164+ ( \(lineItems, freeTimeRemaining) (group, usedTime) ->
165+ let (lineItemsToAdd, newFreeTimeRemaining) = generateLineItems freeTimeRemaining group usedTime
166+ in (lineItems ++ lineItemsToAdd, newFreeTimeRemaining)
167+ )
168+ ([], totalFreeTime)
169+ $ sortBy (compare `on` \(group, duration) -> (Down duration, group))
170+ $ Map.toList
171+ $ Map.fromListWith addDuration
172+ $ filter
173+ ((> emptyDuration) . snd)
174+ usedTimeByGroup
175+ where
176+ periodDuration :: Duration
177+ periodDuration = periodEnd `diffTime` periodStart
178+
179+ -- Duration returned here is the updated free time remaining
180+ generateLineItems :: Duration -> BranchServerGroupIdentifier -> Duration -> ([BranchServerBillingLineItem], Duration)
181+ generateLineItems freeTimeRemaining group usedTime
182+ | not serverIsEligibleForFreeTier || freeTimeRemaining == emptyDuration =
183+ -- There is no free time remaining, or the server is not free. Bill this server for the entire usage
184+ ( [ BranchServerBillingLineItem
185+ { group = group,
186+ includedInPlan = False,
187+ usedTime = usedTime,
188+ cost = serverCost `multiplyCost` (usedTime `divideDuration` periodDuration)
189+ }
190+ ],
191+ freeTimeRemaining
192+ )
193+ | freeTimeRemaining >= usedTime =
194+ -- We have still enough free time to cover this entire server use
195+ ( [ BranchServerBillingLineItem
196+ { group = group,
197+ includedInPlan = True,
198+ usedTime = usedTime,
199+ cost = usd 0
200+ }
201+ ],
202+ freeTimeRemaining `subtractDuration` usedTime
203+ )
204+ | otherwise =
205+ -- This server is partly covered by free time, so we split it into two line items
206+ ( [ BranchServerBillingLineItem
207+ { group = group,
208+ includedInPlan = True,
209+ usedTime = freeTimeRemaining,
210+ cost = usd 0
211+ },
212+ let unFreeUsedTime = usedTime `subtractDuration` freeTimeRemaining
213+ in BranchServerBillingLineItem
214+ { group = group,
215+ includedInPlan = False,
216+ usedTime = unFreeUsedTime,
217+ cost = serverCost `multiplyCost` (unFreeUsedTime `divideDuration` periodDuration)
218+ }
219+ ],
220+ emptyDuration
221+ )
222+ where
223+ serverIsEligibleForFreeTier :: Bool
224+ serverIsEligibleForFreeTier = group ^. #serverTier == serverTierIncludedWithPlans
225+
226+ serverCost :: MonetaryCost
227+ serverCost = serverTierToCost $ group ^. #serverTier
backend/src/Garnix/Hosting/ServerPool.hsnew file+175
@@ -0,0 +1,175 @@
1+module Garnix.Hosting.ServerPool
2+ ( createServer,
3+ initializeProvisioningPool,
4+ sshArgsFor,
5+ _checkServerPoolInterval,
6+ )
7+where
8+
9+import Cradle
10+import Data.List.Extra (enumerate)
11+import Data.Text qualified as T
12+import Garnix.DB qualified as DB
13+import Garnix.Duration
14+import Garnix.Hosting.ServerPool.Types
15+import Garnix.Monad
16+import Garnix.Monad.Async (timeoutThrowing)
17+import Garnix.Monad.NoThrow qualified as NoThrow
18+import Garnix.Monad.SubProcess (runSubProcess_)
19+import Garnix.Monad.SubProcess.Deprecated qualified as Deprecated
20+import Garnix.Prelude
21+import Garnix.Request (retrySequence)
22+import Garnix.Types
23+
24+createServer :: RepoInfo -> DeploymentType -> ServerToSpinUp -> M ServerInfo
25+createServer repoInfo deployType serverToSpinUp = do
26+ startedCreating <- liftIO getCurrentTime
27+ let pollForServer = do
28+ res <- DB.claimServerDB serverToSpinUp (ghPrDeployment deployType)
29+ case res of
30+ Just r -> pure r
31+ Nothing -> do
32+ now <- liftIO getCurrentTime
33+ when (now > addTime serverWaitTimeout startedCreating)
34+ $ throw
35+ $ ProvisioningError "Waited too long for server provisioning. Cancelled"
36+ log Informational "createServer: no server ready, waiting..."
37+ threadDelay pollForServerDuration
38+ pollForServer
39+ server <- pollForServer
40+ log Informational $ "createServer: got server" <> show (server ^. id)
41+ (ip, sshArgs) <- sshArgsFor server
42+ void
43+ ( Deprecated.runProc
44+ "ssh"
45+ ( sshArgs
46+ <> [ "root@" <> ip,
47+ "ls"
48+ ]
49+ )
50+ []
51+ )
52+ updateMetadata repoInfo deployType (serverToSpinUp ^. #build) (server ^. id) (server ^. hetznerServerId)
53+ pure server
54+
55+initializeProvisioningPool :: M ThreadId
56+initializeProvisioningPool = withTextSpan ("tag", "provisioning pool thread") $ do
57+ NoThrow.forkForever _checkServerPoolInterval $ do
58+ serverPoolConfig <- view #serverPoolConfig
59+ NoThrow.forConcurrently_ serverPoolConfig $ \(serverTier, idealPoolSize) -> do
60+ withSpan serverTier $ do
61+ count <- DB.getPreprovisionedServerCount serverTier
62+ log Informational $ "Currently provisioned " <> show serverTier <> " servers: " <> show count
63+ let missingFromPool = idealPoolSize - fromIntegral count
64+ when (missingFromPool > 0) $ do
65+ let toProvision = min missingFromPool maxNumberOfProvisioningThreads
66+ log Informational $ "Will preprovision " <> show toProvision <> " more " <> show serverTier <> " servers. " <> show missingFromPool <> " total servers needed."
67+ NoThrow.replicateConcurrently_ toProvision $ do
68+ id' <- DB.newServerInPool serverTier
69+ let candidates :: [(HetznerLocation, HetznerServerType)]
70+ candidates = do
71+ hetznerServerType <- serverTierToHetznerServerType serverTier
72+ location <- enumerate @HetznerLocation
73+ pure (location, hetznerServerType)
74+ let doIt loc hetznerServerType = do
75+ server <- provisionServer id' loc hetznerServerType <?> ("Preprovisioning server " <> show id')
76+ DB.updatePreprovisionedServer server
77+ setupServer server
78+ DB.setPreprovisionedReady (server ^. id)
79+ ( retrySequence . (`fmap` candidates) $ \(loc, hetznerServerType) ->
80+ doIt loc hetznerServerType <?> ("ServerPool: Provisioning server '" <> show serverTier <> "' in '" <> show loc <> "'")
81+ )
82+ `onError` DB.deleteServerFromPool id'
83+
84+setupServer :: PreprovisionedServer -> M ()
85+setupServer preprovisionedServer = do
86+ isInitialized <-
87+ waitTillServerIsInitialized (preprovisionedServer ^. hetznerServerId)
88+ <?> "Waiting for server to be initialized"
89+ unless isInitialized
90+ . throw
91+ . ProvisioningError
92+ $ "Server did not reach 'running' status in given time limit. Server info:"
93+ <> show preprovisionedServer
94+ isNixos <- waitTillServerIsNixos preprovisionedServer <?> "Waiting for NixOS to be installed on server"
95+ unless isNixos
96+ . throw
97+ . ProvisioningError
98+ $ "Server did not become NixOS in given time limit. Server info:"
99+ <> show preprovisionedServer
100+ return ()
101+
102+waitTillServerIsInitialized :: HetznerServerId -> M Bool
103+waitTillServerIsInitialized = mockable #waitTillServerIsInitializedMock $ loop 100
104+ where
105+ delay :: Duration
106+ delay = fromSeconds @Int 1
107+
108+ loop :: Int -> HetznerServerId -> M Bool
109+ loop n hId
110+ | n <= 0 = pure False
111+ | otherwise = do
112+ getServerStatus hId >>= \case
113+ "running" -> pure True
114+ _ -> liftIO (threadDelay delay) >> loop (n - 1) hId
115+
116+waitTillServerIsNixos :: PreprovisionedServer -> M Bool
117+waitTillServerIsNixos server = isNixos 40
118+ where
119+ delay :: Duration
120+ delay = fromSeconds @Int 20
121+
122+ isNixos :: Int -> M Bool
123+ isNixos 0 = pure False
124+ isNixos n = do
125+ (ip, sshArgs) <- sshArgsFor server
126+ let again = liftIO (threadDelay delay) >> isNixos (n - 1)
127+ ( do
128+ timeoutThrowing (fromMinutes @Int 1) (SshTimeout {command = "nixos-version"}) $ do
129+ runSubProcess_
130+ ( cmd "ssh"
131+ & addArgs (sshArgs <> ["root@" <> ip, "nixos-version"])
132+ )
133+ pure True
134+ )
135+ -- These errors happen when a connection can't be established
136+ `catchAny` const again
137+ -- These errors happen when nixos-version isn't present
138+ `catchError` const again
139+
140+maxNumberOfProvisioningThreads :: Int
141+maxNumberOfProvisioningThreads = 10
142+
143+serverWaitTimeout :: Duration
144+serverWaitTimeout = fromMinutes @Int 10
145+
146+_checkServerPoolInterval :: Duration
147+_checkServerPoolInterval = fromSeconds @Int 15
148+
149+pollForServerDuration :: Duration
150+pollForServerDuration = fromSeconds @Int 5
151+
152+sshArgsFor :: (HasIpv4Addr s Text) => s -> M (Text, [Text])
153+sshArgsFor server = do
154+ keyFiles <- view #sshUserHostingKeys
155+ let sshKeysParameters = concatMap (\f -> ["-i", cs f]) keyFiles
156+ (ip, port) <- splitPortFromIP (server ^. ipv4Addr)
157+ pure
158+ ( ip,
159+ sshKeysParameters
160+ <> [ "-o",
161+ "BatchMode=yes",
162+ "-o",
163+ "StrictHostKeychecking=no",
164+ "-o",
165+ "UserKnownHostsFile=/dev/null",
166+ "-p",
167+ port
168+ ]
169+ )
170+
171+splitPortFromIP :: Text -> M (Text, Text)
172+splitPortFromIP ip = case T.splitOn ":" ip of
173+ [_] -> pure (ip, "22")
174+ [ip', p] -> pure (ip', p)
175+ _ -> throw $ ProvisioningError ("IP not valid: " <> ip)
backend/src/Garnix/Hosting/ServerPool/Types.hsnew file+80
@@ -0,0 +1,80 @@
1+module Garnix.Hosting.ServerPool.Types where
2+
3+import Garnix.MonetaryCost
4+import Garnix.Prelude
5+
6+data ServerTier
7+ = I2x4
8+ | I4x8
9+ | I8x16
10+ | I16x32
11+ deriving (Eq, Show, Enum, Bounded, Ord)
12+
13+instance Default ServerTier where
14+ def = I2x4
15+
16+instance PGParameter "text" ServerTier where
17+ pgEncode proxy tier = pgEncode proxy (serverTierToText tier)
18+
19+instance PGColumn "text" ServerTier where
20+ pgDecode proxy value =
21+ let text :: Text = pgDecode proxy value
22+ in case lookup text (map swap serverTierTextMapping) of
23+ Just tier -> tier
24+ Nothing -> error $ "Impossible: unknown subscription type " <> cs text
25+
26+serverTierTextMapping :: [(ServerTier, Text)]
27+serverTierTextMapping = map (\tier -> (tier, serverTierToText tier)) [minBound .. maxBound]
28+
29+serverTierToText :: ServerTier -> Text
30+serverTierToText = \case
31+ I2x4 -> "i2x4"
32+ I4x8 -> "i4x8"
33+ I8x16 -> "i8x16"
34+ I16x32 -> "i16x32"
35+
36+serverTierToHetznerServerType :: ServerTier -> [HetznerServerType]
37+serverTierToHetznerServerType = \case
38+ I2x4 -> [HetznerCX23, HetznerCPX22]
39+ I4x8 -> [HetznerCX33]
40+ I8x16 -> [HetznerCX43]
41+ I16x32 -> [HetznerCX53]
42+
43+-- | See https://docs.hetzner.com/cloud/servers/overview/.
44+data HetznerServerType
45+ = HetznerCX23
46+ | HetznerCPX22
47+ | HetznerCX33
48+ | HetznerCX43
49+ | HetznerCX53
50+ deriving (Eq, Show, Enum, Bounded, Ord)
51+
52+hetznerServerTypeToName :: HetznerServerType -> Text
53+hetznerServerTypeToName = \case
54+ HetznerCX23 -> "cx23"
55+ HetznerCPX22 -> "cpx22"
56+ HetznerCX33 -> "cx33"
57+ HetznerCX43 -> "cx43"
58+ HetznerCX53 -> "cx53"
59+
60+data HetznerLocation
61+ = HetznerHelsinki
62+ | HetznerNuremberg
63+ | HetznerFalkenstein
64+ deriving (Eq, Show, Enum, Bounded, Ord)
65+
66+hetznerLocationToName :: HetznerLocation -> Text
67+hetznerLocationToName = \case
68+ HetznerNuremberg -> "nbg1"
69+ HetznerFalkenstein -> "fsn1"
70+ HetznerHelsinki -> "hel1"
71+
72+serverTierIncludedWithPlans :: ServerTier
73+serverTierIncludedWithPlans = I2x4
74+
75+serverTierToCost :: ServerTier -> MonetaryCost
76+serverTierToCost = \case
77+ I2x4 -> usd 15
78+ I4x8 -> usd 30
79+ I8x16 -> usd 60
80+ I16x32 -> usd 120
backend/src/Garnix/Incremental.hsnew file+153
@@ -0,0 +1,153 @@
1+module Garnix.Incremental
2+ ( makeNormalizedFlake,
3+ renderNormalizedFlakeWithHelpers,
4+ withIntermediatesFlake,
5+ NormalizedFlake (..),
6+ )
7+where
8+
9+import Control.Lens (At, Index, IxValue, Ixed)
10+import Cradle
11+import Data.Map qualified as Map
12+import Data.Monoid
13+import Data.String.Interpolate (i)
14+import Data.Text qualified as T
15+import Data.Text.IO qualified as T
16+import Garnix.DB qualified as DB
17+import Garnix.Monad
18+import Garnix.Nix.StorePath
19+import Garnix.Nix.Types qualified as Nix
20+import Garnix.Prelude
21+import Garnix.Types
22+import System.IO.Temp
23+
24+withIntermediatesFlake :: Build -> (Maybe FilePath -> M a) -> M a
25+withIntermediatesFlake build action = do
26+ builds <-
27+ (previousCandidates >>= DB.getIncrementalTarget build)
28+ <?> "Getting incremental build base candidates"
29+ case builds of
30+ [] -> do
31+ log Informational "No builds found for incrementalization"
32+ action Nothing
33+ _ -> do
34+ emptyDir' <- view #emptyDir
35+ flakeContents <- liftIO . renderNormalizedFlakeWithHelpers emptyDir' =<< makeNormalizedFlake builds
36+ log Informational $ "Using the following flake file for garnix-incrementalize:\n "
37+ <> flakeContents
38+ withSystemTempDirectory "incremental-build" $ \fp -> do
39+ liftIO $ T.writeFile (fp <> "/flake.nix") flakeContents
40+ action (Just fp)
41+
42+previousCandidates :: M [CommitHash]
43+previousCandidates = do
44+ workingDir <- view #workingDir
45+ (StdoutTrimmed out, e) <-
46+ run
47+ $ cmd "git"
48+ & addArgs ["rev-list", "-n", "5", "HEAD" :: String]
49+ & setWorkingDir workingDir
50+ case e of
51+ ExitSuccess -> pure $ CommitHash <$> tail (T.lines out)
52+ _ -> throw $ OtherError "Could not get rev-list for incrementalization"
53+
54+makeNormalizedFlake :: [Build] -> M NormalizedFlake
55+makeNormalizedFlake = foldM go mempty
56+ where
57+ go :: NormalizedFlake -> Build -> M NormalizedFlake
58+ go f build
59+ | build ^. packageType == TypeOverall = pure f
60+ | otherwise = do
61+ withStorePath build "intermediates" $ \storePath -> do
62+ pure
63+ $ f
64+ & at
65+ ( build ^. packageType,
66+ build ^. system,
67+ build ^. package
68+ )
69+ .~ storePath
70+
71+renderNormalizedFlakeWithHelpers :: FilePath -> NormalizedFlake -> IO Text
72+renderNormalizedFlakeWithHelpers emptyDir' (NormalizedFlake f) = cs <$> rendered
73+ where
74+ renderSingle :: (PackageType, MaybeSystem, PackageName) -> Nix.StorePath -> Text -> Text
75+ renderSingle (typ, sys, PackageName name) s prev =
76+ prev
77+ <> "\n"
78+ <> (typ ^. re asPackageType)
79+ <> "s."
80+ <> case sys of
81+ NoSystem -> ""
82+ IsSystem s -> s ^. systemTextIso <> "."
83+ <> name
84+ <> case typ of
85+ TypeNixosConfiguration -> ".config.system.build.toplevel"
86+ _ -> ""
87+ <> ".intermediates"
88+ <> " = builtins.fetchClosure { "
89+ <> " inputAddressed = true;" -- Is this worth changing?
90+ <> " fromStore = \"https://cache.garnix.io\"; "
91+ <> " fromPath = \""
92+ <> cs s
93+ <> "\";"
94+ <> " };"
95+ attrs :: Text
96+ attrs = Map.foldrWithKey renderSingle "" f
97+
98+ helpers = do
99+ pure
100+ [i|
101+ lib.withCaches = args.self.lib.withCachesFor args.self;
102+
103+ lib.withCachesFor = prev: outputs:
104+ let wantedAttrs = ["packages" "checks" "devShells"];
105+ emptyDir = "${#{emptyDir'}}";
106+ mapAttrsIfSet = fn : s : if builtins.isAttrs s then builtins.mapAttrs fn s else s;
107+ in (mapAttrsIfSet (type:
108+ mapAttrsIfSet (sys:
109+ mapAttrsIfSet (pkg: def:
110+ if builtins.elem type wantedAttrs && builtins.isFunction def
111+ then (def (prev.${type}.${sys}.${pkg}.intermediates or emptyDir))
112+ else def
113+ )))) outputs;
114+
115+ |]
116+ rendered = do
117+ h <- helpers
118+ pure
119+ [i|
120+ {
121+ outputs = args : {
122+ #{attrs}
123+ #{h}
124+ };
125+ }
126+ |]
127+
128+-- * Types
129+
130+-- | Represents a Flake that has no eval left to be done, and without any inputs.
131+-- This can be thought of as the "normal form" of a flake.
132+newtype NormalizedFlake
133+ = NormalizedFlake
134+ (Map.Map (PackageType, MaybeSystem, PackageName) Nix.StorePath)
135+ deriving newtype (Semigroup, Monoid)
136+
137+instance At NormalizedFlake where
138+ at i = co . at i . coerced
139+ where
140+ co ::
141+ Iso'
142+ NormalizedFlake
143+ (Map.Map (PackageType, MaybeSystem, PackageName) Nix.StorePath)
144+ co = coerced
145+
146+instance Ixed NormalizedFlake
147+
148+type instance Index NormalizedFlake = (PackageType, MaybeSystem, PackageName)
149+
150+type instance IxValue NormalizedFlake = Nix.StorePath
151+
152+newtype Output = Output Text
153+ deriving newtype (Eq, Ord, IsString)
backend/src/Garnix/Log.hsnew file+118
@@ -0,0 +1,118 @@
1+module Garnix.Log where
2+
3+import Garnix.Hosting.ServerPool.Types
4+import Garnix.Prelude
5+import Garnix.Types
6+import GitHub.App.Auth (InstallationAuth)
7+
8+class Loggable a where
9+ asLog :: a -> [(Text, Text)]
10+
11+instance Loggable PackageType where
12+ asLog n = [("package_type", showPretty n)]
13+
14+instance Loggable PackageName where
15+ asLog n = [("package", prj n)]
16+
17+instance Loggable Branch where
18+ asLog n = [("branch", prj n)]
19+
20+instance Loggable RepoPublicity where
21+ asLog (RepoIsPublic n) = [("public", show n)]
22+
23+instance Loggable PrFromFork where
24+ asLog (PrFromFork n) = [("fork", n)]
25+
26+instance (Loggable a) => Loggable (Maybe a) where
27+ asLog = maybe [] asLog
28+
29+instance Loggable MaybeSystem where
30+ asLog n = case n of
31+ IsSystem s -> [("system", s ^. systemTextIso)]
32+ NoSystem -> []
33+
34+instance Loggable CommitHash where
35+ asLog (CommitHash n) = [("commit", n)]
36+
37+instance Loggable GhRepoOwner where
38+ asLog (GhRepoOwner (GhLogin n)) = [("gh_owner", n)]
39+
40+instance Loggable GhRepoName where
41+ asLog (GhRepoName n) = [("gh_repo", n)]
42+
43+instance Loggable GhPullRequestId where
44+ asLog (GhPullRequestId n) = [("gh_pr", show n)]
45+
46+instance Loggable UserId where
47+ asLog (UserId n) = [("user", show n)]
48+
49+instance Loggable GhLogin where
50+ asLog (GhLogin n) = [("req_user", n)]
51+
52+instance Loggable InstallationAuth where
53+ asLog _ = []
54+
55+instance Loggable BuildId where
56+ asLog id = [("buildId", getHashId hash <> "(" <> show (hash ^. hashIdInt) <> ")")]
57+ where
58+ hash = getBuildId id
59+
60+instance Loggable CommitInfo where
61+ asLog info = asLog (info ^. _CommitInfo)
62+
63+instance Loggable RepoInfo where
64+ asLog info = asLog (info ^. _RepoInfo)
65+
66+instance Loggable PackageInfo where
67+ asLog info = asLog (info ^. _PackageInfo)
68+
69+instance Loggable GhToken where
70+ asLog _ = []
71+
72+instance Loggable ServerTier where
73+ asLog serverTier = [("server_tier", show serverTier)]
74+
75+instance
76+ (Loggable a, Loggable b, Loggable c) =>
77+ Loggable (a, b, c)
78+ where
79+ asLog (a, b, c) =
80+ asLog a
81+ <> asLog b
82+ <> asLog c
83+
84+instance
85+ (Loggable a, Loggable b, Loggable c, Loggable d) =>
86+ Loggable (a, b, c, d)
87+ where
88+ asLog (a, b, c, d) =
89+ asLog a
90+ <> asLog b
91+ <> asLog c
92+ <> asLog d
93+
94+instance
95+ (Loggable a, Loggable b, Loggable c, Loggable d, Loggable e, Loggable f) =>
96+ Loggable (a, b, c, d, e, f)
97+ where
98+ asLog (a, b, c, d, e, f) =
99+ asLog a
100+ <> asLog b
101+ <> asLog c
102+ <> asLog d
103+ <> asLog e
104+ <> asLog f
105+
106+instance
107+ (Loggable a, Loggable b, Loggable c, Loggable d, Loggable e, Loggable f, Loggable g, Loggable h) =>
108+ Loggable (a, b, c, d, e, f, g, h)
109+ where
110+ asLog (a, b, c, d, e, f, g, h) =
111+ asLog a
112+ <> asLog b
113+ <> asLog c
114+ <> asLog d
115+ <> asLog e
116+ <> asLog f
117+ <> asLog g
118+ <> asLog h
backend/src/Garnix/Modules.hsnew file+45
@@ -0,0 +1,45 @@
1+module Garnix.Modules
2+ ( publish,
3+ )
4+where
5+
6+import Data.Row ((.+), (.==))
7+import Garnix.BuildLogs.Types (mkLogLine)
8+import Garnix.DB qualified as DB
9+import Garnix.DB.ModuleValues qualified as DB
10+import Garnix.Modules.Schema qualified as ModuleSchema
11+import Garnix.Monad
12+import Garnix.Prelude
13+import Garnix.Reporters.Utils (withRunReporter)
14+import Garnix.Types
15+import Garnix.YamlConfig qualified as Config
16+
17+-- | Assumes Env ^. #workingDir is correctly set, i.e., we're running within `withCheckout`.
18+publish :: Reporter -> Config.GarnixConfig -> CommitInfo -> M ()
19+publish reporter config commitInfo = withTextSpan ("modules_publish", show publishEnabled) $ do
20+ case (publishEnabled, commitInfo ^. branch) of
21+ (True, Just "main") -> do
22+ run <- DB.newRun "Garnix module publish" commitInfo
23+ withRunReporter reporter (ReportRun run) $ \runReporter -> do
24+ case commitInfo ^. repoInfo . ghRepoOwner of
25+ "garnix-io" -> do
26+ schema <- view #workingDir >>= ModuleSchema.readModuleSchema
27+ let repo = commitInfo ^. repoInfo
28+ repoName = repo ^. ghRepoName . to getGhRepoName
29+ moduleName = ModuleSchema.repoNameToModuleName repoName
30+ DB.insertLatestVersion
31+ $ (#name .== moduleName)
32+ .+ (#repo_user .== repo ^. ghRepoOwner)
33+ .+ (#repo_name .== repo ^. ghRepoName)
34+ .+ (#git_commit .== commitInfo ^. commit)
35+ .+ (#schema .== toJSON schema)
36+ .+ (#description .== ModuleSchema.description schema)
37+ reportLogs runReporter $ mkLogLine $ "Module " <> moduleName <> " updated successfully!"
38+ reportComplete runReporter RunReportStatusSuccess
39+ otherOrg ->
40+ throw $ OtherError $ "Publishing modules is not enabled for " <> getGhLogin (getGhRepoOwner otherOrg) <> "."
41+ _ ->
42+ log Informational "Modules: skipping publish"
43+ where
44+ publishEnabled :: Bool
45+ publishEnabled = config ^. Config.moduleSection . #publish
backend/src/Garnix/Modules/Schema.hsnew file+163
@@ -0,0 +1,163 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+
3+module Garnix.Modules.Schema where
4+
5+import Cradle
6+import Data.Aeson hiding ((.?=))
7+import Data.Aeson.Lens
8+import Data.ByteString (ByteString)
9+import Data.FileEmbed
10+import Data.Map.Strict (Map)
11+import Data.Map.Strict qualified as Map
12+import Data.String.Interpolate (i)
13+import Data.Text qualified as T
14+import Garnix.DB.ModuleValues qualified as ModuleValues
15+import Garnix.Monad
16+import Garnix.NixConfig (addNixConfigEnvironment)
17+import Garnix.Prelude
18+import Garnix.Sandbox
19+import Garnix.Types hiding (description)
20+import Text.Casing qualified as Casing
21+
22+data ModuleSchema = ModuleSchema
23+ { description :: Maybe Text,
24+ example :: Maybe Text,
25+ default_ :: Maybe ModuleValues.NixValue,
26+ name :: Maybe Text,
27+ typ :: ModuleSchemaType
28+ }
29+ deriving (Show, Eq)
30+
31+instance ToJSON ModuleSchema where
32+ toJSON (ModuleSchema {description, example, default_, name, typ}) =
33+ object
34+ ( ("description" .?= description)
35+ <> ("example" .?= example)
36+ <> ("default" .?= default_)
37+ <> ("name" .?= name)
38+ <> ["typ" .= typ]
39+ )
40+
41+(.?=) :: (ToJSON a) => Key -> Maybe a -> [(Key, Value)]
42+name .?= value = case value of
43+ Nothing -> []
44+ Just value -> [(name, toJSON value)]
45+
46+instance FromJSON ModuleSchema where
47+ parseJSON = withObject "ModuleSchema" $ \o -> do
48+ typ <- o .: "typ"
49+ description <- o .:? "description"
50+ example <- o .:? "example"
51+ default_ <- o .:? "default"
52+ name <- o .:? "name"
53+ pure $ ModuleSchema description example default_ name typ
54+
55+data ModuleSchemaType
56+ = Secret
57+ | Path
58+ | Str
59+ | NonEmptyStr
60+ | Bool
61+ | Int
62+ | UnsignedInt16
63+ | Enum [Text]
64+ | Package
65+ | Submodule (Map Text ModuleSchema)
66+ | AttrsOf ModuleSchemaType
67+ | ListOf ModuleSchemaType
68+ | NullOr ModuleSchemaType
69+ deriving (Show, Eq)
70+
71+instance ToJSON ModuleSchemaType where
72+ toJSON = \case
73+ Secret -> object ["tag" .= ("encryptedSecret" :: Text)]
74+ Path -> object ["tag" .= ("path" :: Text)]
75+ Str -> object ["tag" .= ("str" :: Text)]
76+ NonEmptyStr -> object ["tag" .= ("nonEmptyStr" :: Text)]
77+ Garnix.Modules.Schema.Bool -> object ["tag" .= ("bool" :: Text)]
78+ Garnix.Modules.Schema.Int -> object ["tag" .= ("int" :: Text)]
79+ UnsignedInt16 -> object ["tag" .= ("unsignedInt16" :: Text)]
80+ Garnix.Modules.Schema.Enum variants -> object ["tag" .= ("enum" :: Text), "variants" .= variants]
81+ Garnix.Modules.Schema.Package -> object ["tag" .= ("package" :: Text)]
82+ Submodule fields -> object ["tag" .= ("submodule" :: Text), "fields" .= fields]
83+ AttrsOf elementType -> object ["tag" .= ("attrsOf" :: Text), "fieldType" .= elementType]
84+ ListOf elementType -> object ["tag" .= ("listOf" :: Text), "elementType" .= elementType]
85+ NullOr innerType -> object ["tag" .= ("nullOr" :: Text), "innerType" .= innerType]
86+
87+instance FromJSON ModuleSchemaType where
88+ parseJSON = withObject "ModuleSchemaType" $ \o -> do
89+ tag <- o .: "tag"
90+ case tag of
91+ "encryptedSecret" -> pure Secret
92+ "path" -> pure Path
93+ "str" -> pure Str
94+ "nonEmptyStr" -> pure NonEmptyStr
95+ "bool" -> pure Garnix.Modules.Schema.Bool
96+ "int" -> pure Garnix.Modules.Schema.Int
97+ "unsignedInt16" -> pure UnsignedInt16
98+ "enum" -> Enum <$> o .: "variants"
99+ "package" -> pure Garnix.Modules.Schema.Package
100+ "submodule" -> Submodule <$> o .: "fields"
101+ "attrsOf" -> AttrsOf <$> o .: "fieldType"
102+ "listOf" -> ListOf <$> o .: "elementType"
103+ "nullOr" -> NullOr <$> o .: "innerType"
104+ _ -> fail $ "unknown tag: " <> tag
105+
106+readModuleSchema :: FilePath -> M ModuleSchema
107+readModuleSchema path = do
108+ nixConfig <- view #userNixConfig
109+ (StdoutUntrimmed output, StderrRaw stderr, exitCode) <-
110+ (>>= run)
111+ $ cmd "nix"
112+ & addArgs
113+ [ "eval",
114+ path <> "#garnixModules.default",
115+ "--json",
116+ "--apply",
117+ [i| (#{moduleToSchemaNix}) { nixpkgsLib = #{nixpkgsLib}; } |]
118+ ]
119+ & setWorkingDir path
120+ & addNixConfigEnvironment nixConfig
121+ & pure
122+ & inNixSandbox [] Nothing
123+ when (exitCode /= ExitSuccess) $ do
124+ throw $ OtherError $ cs stderr
125+ schema <- aesonDecode "extracted module schema" parseJSON output
126+ StdoutUntrimmed output <-
127+ (>>= run)
128+ $ cmd "nix"
129+ & addArgs ["flake", "metadata", "--json" :: Text]
130+ & setWorkingDir path
131+ & addNixConfigEnvironment nixConfig
132+ & pure
133+ & inNixSandbox [] Nothing
134+ let moduleDescription = output ^? key "description" . _String
135+ pure $ schema {description = moduleDescription}
136+
137+repoNameToModuleName :: Text -> Text
138+repoNameToModuleName name =
139+ let nameFromFile = fromMaybe name $ T.stripSuffix "-module" name
140+ in case nameFromFile of
141+ "nodejs" -> "NodeJS"
142+ "postgresql" -> "PostgreSQL"
143+ "rss-bridge" -> "RSS-Bridge"
144+ otherName -> cs . Casing.pascal . cs $ otherName
145+
146+moduleToSchemaNix :: ByteString
147+moduleToSchemaNix = $(embedFileRelative "src/Garnix/Modules/module-to-schema.nix")
148+
149+nixpkgsLib :: Text
150+nixpkgsLib =
151+ cs
152+ [i|
153+ (let nixpkgsLib = builtins.fetchGit (builtins.fromJSON ''#{encode args}'');
154+ in import "${nixpkgsLib}/lib")
155+ |]
156+ where
157+ args :: Map Text Text
158+ args =
159+ Map.fromList
160+ [ ("url", "https://github.com/nix-community/nixpkgs.lib"),
161+ ("ref", "master"),
162+ ("rev", "f4dc9a6c02e5e14d91d158522f69f6ab4194eb5b")
163+ ]
backend/src/Garnix/Modules/module-to-schema.nixnew file+103
@@ -0,0 +1,103 @@
1+{ nixpkgsLib }: module:
2+let
3+ lib = nixpkgsLib;
4+
5+ deepMerge = list: lib.foldl lib.recursiveUpdate { } list;
6+
7+ optionToSchema = option:
8+ if option._type != "option"
9+ then builtins.abort "not an option"
10+ else {
11+ typ = typeToSchema option.type;
12+ description = if option ? description then option.description else null;
13+ example = if option ? example then toString option.example else null;
14+ default = if option ? default then defaultToNixValue option.default else null;
15+ name = if option ? name then option.name else null;
16+ };
17+
18+ defaultToNixValue = value:
19+ let type = builtins.typeOf value; in
20+ if type == "list" then
21+ { tag = type; value = lib.map defaultToNixValue value; }
22+ else if type == "null" then
23+ { tag = type; }
24+ else if lib.elem type [ "string" "int" "bool" "path" ] then
25+ { tag = type; inherit value; }
26+ else builtins.abort ("unsupported default type: " + type + " (value: " + builtins.toJSON value + ")");
27+
28+ typeToSchema = type:
29+ if type.name == "encryptedSecret" then
30+ { tag = "encryptedSecret"; }
31+ else if type.name == "path" then
32+ { tag = "path"; }
33+ else if type.name == "str" then
34+ { tag = "str"; }
35+ else if type.name == "nonEmptyStr" then
36+ { tag = "nonEmptyStr"; }
37+ else if type.name == "bool" then
38+ { tag = "bool"; }
39+ else if type.name == "int" then
40+ { tag = "int"; }
41+ else if type.name == "unsignedInt16" then
42+ { tag = "unsignedInt16"; }
43+ else if type.name == "enum" then
44+ {
45+ tag = "enum";
46+ # This will likely break when 25.04 hits, because of this PR:
47+ # https://github.com/NixOS/nixpkgs/commit/f407f6f57ec12cfe1c5bf2de531cd8c3d601332d#diff-64168148acd9f2147ef733b1498b8821c24f2e3f32354b0e147dd421d71274f3R1019
48+ # We'll need to change this to `type.functor.payload.values`.
49+ variants = type.functor.payload;
50+ }
51+ else if type.name == "package" then
52+ { tag = "package"; }
53+ else if type.name == "submodule" then
54+ {
55+ tag = "submodule";
56+ fields = moduleListToSchema type.getSubModules;
57+ }
58+ else if type.name == "attrsOf" then
59+ {
60+ tag = "attrsOf";
61+ fieldType = typeToSchema type.nestedTypes.elemType;
62+ }
63+ else if type.name == "listOf" then
64+ {
65+ tag = "listOf";
66+ elementType = typeToSchema type.nestedTypes.elemType;
67+ }
68+ else if type.name == "nullOr" then
69+ {
70+ tag = "nullOr";
71+ innerType = typeToSchema type.nestedTypes.elemType;
72+ }
73+ else
74+ builtins.abort ("unsupported option type: " + type.name);
75+
76+ moduleListToSchema = list: lib.mapAttrs
77+ (key: value: optionToSchema value)
78+ ((deepMerge (map moduleToSet list)).options or { });
79+
80+ moduleToSet = module:
81+ if builtins.typeOf module == "set" then module
82+ else if builtins.typeOf module == "lambda" then
83+ let
84+ arguments = lib.mergeAttrsList
85+ (map (name: { "${name}" = mkArgument name; }) requiredArguments);
86+ requiredArguments = lib.attrNames (builtins.functionArgs module);
87+ mkArgument = name:
88+ if name == "lib" then lib
89+ else if name == "pkgs" then builtins.abort "pkgs"
90+ else if name == "config" then { }
91+ else
92+ builtins.abort ("module argument not supported: " + name);
93+ in
94+ module arguments
95+ else
96+ builtins.abort ("unsupported module type: " + builtins.typeOf module);
97+in
98+{
99+ typ = {
100+ tag = "submodule";
101+ fields = moduleListToSchema [ module ];
102+ };
103+}
backend/src/Garnix/Monad.hsnew file+726
@@ -0,0 +1,726 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+{-# LANGUAGE UndecidableInstances #-}
3+
4+module Garnix.Monad
5+ ( module Garnix.Monad,
6+ Severity (..),
7+ )
8+where
9+
10+import Amazonka.Env qualified as Amazonka (Env)
11+import Amazonka.S3 qualified as Amazonka
12+import Control.Concurrent (MVar, modifyMVar_, newMVar, readMVar)
13+import Control.Exception.Safe qualified as SafeException
14+import Control.Lens (IndexedTraversal')
15+import Control.Lens.Regex.Text qualified as RE
16+import Control.Monad.Base (MonadBase)
17+import Data.Aeson (Value, eitherDecode')
18+import Data.Aeson.Types (Parser, parseEither)
19+import Data.ByteString (ByteString)
20+import Data.ByteString.Lazy qualified as BSL
21+import Data.Containers.ListUtils (nubOrd)
22+import Data.Set qualified as Set
23+import Data.Text qualified as T
24+import Data.Text.Format.Numbers (PrettyCfg (PrettyCfg), prettyF)
25+import Data.Time (Day)
26+import Data.Time.Clock.System (getSystemTime, systemToUTCTime)
27+import Data.UUID qualified
28+import Data.UUID.V4 qualified
29+import Garnix.Async (Promise)
30+import Garnix.Build.Types (EvaluationResult)
31+import Garnix.BuildLogs.Types (LogLine)
32+import Garnix.DB.FeatureFlags.Types (FeatureFlagConfig)
33+import Garnix.Duration
34+import Garnix.GithubInterface.Types
35+import Garnix.Hosting.ServerPool.Types
36+import Garnix.Log
37+import Garnix.Monad.ForkT
38+import Garnix.Monad.Memoization (MemoTable)
39+import Garnix.Monad.Metrics (Metrics, incrementEvent)
40+import Garnix.Monad.Pool (Pool)
41+import Garnix.Nix.Types (StoreHash)
42+import Garnix.Nix.Types qualified as Nix
43+import Garnix.Prelude
44+import Garnix.StripeLib.Types qualified as StripeLib
45+import Garnix.Types hiding (ghRunId, statusCode)
46+import GitHub qualified as GH
47+import GitHub.App.Auth (InstallationAuth)
48+import GitHub.App.Auth qualified as GHA
49+import GitHub.Data (Id)
50+import GitHub.Data.Apps (App)
51+import GitHub.Data.Installations qualified as GHA
52+import Network.HTTP.Client (Manager)
53+import Network.HTTP.Types (statusCode)
54+import Network.Wai qualified as Wai
55+import Network.Wai.Header (contentLength)
56+import Network.Wreq qualified as Wreq
57+import Servant.Auth.Server (CookieSettings, JWTSettings)
58+import System.Directory (canonicalizePath)
59+import System.Log.FastLogger as FastLogger
60+import Text.Read (readMaybe)
61+
62+data Env = Env
63+ { testFeatures :: Set TestFeature,
64+ githubAppAuth :: GHA.AppAuth,
65+ githubAppName :: Text,
66+ githubAppId :: Id App,
67+ githubWebhookSecret :: ByteString,
68+ manager :: Manager,
69+ githubClientSecret :: Text,
70+ githubClientId :: Text,
71+ buildLogsReportingPort :: Maybe Int,
72+ githubInterface :: GithubInterface,
73+ hetznerInterface :: HetznerInterface,
74+ serverPoolConfig :: [(ServerTier, Int)],
75+ -- | A thread-safe version of `CWD`
76+ workingDir :: FilePath,
77+ nixXdgCacheDir :: Maybe FilePath,
78+ userNixConfig :: NixConfig,
79+ cookieSettings :: CookieSettings,
80+ jwtSettings :: JWTSettings,
81+ dbConn :: DatabaseConnection,
82+ baseUrl :: Text,
83+ sshUserHostingKeys :: [FilePath],
84+ s3CacheEnv :: S3CacheEnv,
85+ action :: ActionEnv,
86+ repoSecretsEncryptionKeyPath :: RepoSecretsEncryptionKeyPath,
87+ repoSecretsEncryptionPubKey :: RepoSecretsEncryptionPubKey,
88+ logger :: LogItem -> IO (),
89+ buildLogsDir :: FilePath,
90+ hetznerToken :: StrictByteString,
91+ opensearchQueryUrl :: String,
92+ opensearchPassword :: ByteString,
93+ nixEvalPool :: Garnix.Monad.Pool.Pool GhRepoOwner,
94+ s3UploadPool :: Garnix.Monad.Pool.Pool GhRepoOwner,
95+ stripe :: StripeEnv,
96+ mocks :: Maybe EnvMocks,
97+ emptyDir :: FilePath,
98+ spanCtx :: [(Text, Text)],
99+ metrics :: Metrics,
100+ hostname :: Text,
101+ githubLogDebounceDuration :: Duration,
102+ featureFlagConfig :: FeatureFlagConfig,
103+ fodCheckPool :: Garnix.Monad.Pool.Pool ()
104+ }
105+ deriving stock (Generic)
106+
107+data TestFeature
108+ = DevApi
109+ | OpenSearchMocks
110+ | StripeMocks
111+ | CacheUploadMocks
112+ | FodCheckMocks
113+ deriving stock (Eq, Show, Read, Ord, Generic, Enum, Bounded)
114+
115+localDevelopment :: Set TestFeature
116+localDevelopment = Set.fromList [minBound .. maxBound]
117+
118+parseTestFeature :: String -> Either Text TestFeature
119+parseTestFeature feature = case readMaybe feature of
120+ Just feature -> Right feature
121+ Nothing ->
122+ Left
123+ $ "unknown test feature passed to --enable: "
124+ <> cs feature
125+ <> ". Possible values: "
126+ <> T.intercalate ", " (map show [minBound .. maxBound :: TestFeature])
127+
128+data S3CacheEnv = S3CacheEnv
129+ { amazonkaEnv :: Amazonka.Env,
130+ publicBucket :: Amazonka.BucketName,
131+ publicBaseUrl :: Text,
132+ privateBucket :: Amazonka.BucketName,
133+ cachePrivKeyFile :: FilePath,
134+ cachePrivKeyName :: Text,
135+ expiration :: Duration,
136+ maxUploadSize :: Integer,
137+ isInNixosCacheMemoTable :: MVar (MemoTable StoreHash Bool)
138+ }
139+ deriving (Generic)
140+
141+data ActionEnv = ActionEnv
142+ { runnerHost :: Text,
143+ runnerSshKey :: Text,
144+ timeoutDuration :: Duration
145+ }
146+ deriving (Generic)
147+
148+data StripeEnv = StripeEnv
149+ { publishableKey :: Text,
150+ secretKey :: Text,
151+ webhookSecret :: Text
152+ }
153+ deriving stock (Generic)
154+
155+data EnvMocks = EnvMocks
156+ { executeDeployPlanMock ::
157+ Maybe
158+ (Mock (Reporter, CommitInfo, DeployPlan, DeploymentType) [ServerInfo]),
159+ waitTillServerIsInitializedMock :: Maybe (Mock HetznerServerId Bool),
160+ buildFlakeMock ::
161+ Maybe
162+ (Mock (Reporter, CommitInfo) (Promise ())),
163+ storeLogLineMock :: Maybe (Mock (OpenSearchId, LogLine) ()),
164+ queryOpenSearchMock ::
165+ Maybe
166+ (Mock (OpenSearchId, [Day], Maybe UTCTime, Int) [OpenSearchMessage]),
167+ startServerMock ::
168+ Maybe
169+ (Mock (Reporter, CommitInfo, DeploymentType, ServerToSpinUp) ServerInfo),
170+ setupServerMock ::
171+ Maybe
172+ (Mock (RepoInfo, Build, ServerInfo) (ServerInfo, Text)),
173+ makeOpenSearchMsearchRequestMock ::
174+ Maybe
175+ (Mock (Value, Value) BSL.ByteString),
176+ createCustomerMock :: Maybe (Mock (GhRepoOwner, StripeLib.Name, Email) StripeLib.CustomerDto),
177+ createSubscriptionMock :: Maybe (Mock (CustomerId, StripeLib.PriceId, Text, Text) StripeLib.SubscriptionDto),
178+ createInvoiceItemMock :: Maybe (Mock (CustomerId, InvoiceId, Text, StripeLib.UnitAmount, Int64) ()),
179+ listSubscriptionsMock :: Maybe (Mock CustomerId StripeLib.SubscriptionListDto),
180+ cancelSubscriptionMock :: Maybe (Mock SubscriptionId ()),
181+ getPriceMock :: Maybe (Mock StripeLib.PriceId StripeLib.PriceDto),
182+ getBuildPlanMock :: Maybe (Mock ByteString Nix.Plan),
183+ buildPkgMock :: Maybe (Mock (Maybe FodChecker, RunReporter, BuildKind, FlakeDir, RepoConfig, ProductPlan, Build) Build),
184+ s3CacheUploadMock :: Maybe (Mock (RunReporter, GhRepoOwner, GhRepoName, EvaluationResult, RepoPublicity) ()),
185+ fodCheckMock :: Maybe (Mock (Maybe FodChecker, Nix.DrvPath) ()),
186+ rebuildFodMock :: Maybe (Mock (System, Nix.DrvPath) (Either Text Text))
187+ }
188+ deriving (Generic)
189+
190+emptyMocks :: EnvMocks
191+emptyMocks =
192+ EnvMocks
193+ { executeDeployPlanMock = Nothing,
194+ waitTillServerIsInitializedMock = Nothing,
195+ buildFlakeMock = Nothing,
196+ storeLogLineMock = Nothing,
197+ queryOpenSearchMock = Nothing,
198+ startServerMock = Nothing,
199+ makeOpenSearchMsearchRequestMock = Nothing,
200+ createCustomerMock = Nothing,
201+ createSubscriptionMock = Nothing,
202+ createInvoiceItemMock = Nothing,
203+ listSubscriptionsMock = Nothing,
204+ cancelSubscriptionMock = Nothing,
205+ getPriceMock = Nothing,
206+ getBuildPlanMock = Nothing,
207+ setupServerMock = Nothing,
208+ buildPkgMock = Nothing,
209+ s3CacheUploadMock = Nothing,
210+ fodCheckMock = Nothing,
211+ rebuildFodMock = Nothing
212+ }
213+
214+data Mock arg result = Mock
215+ { mockImplementation :: arg -> M result,
216+ calls :: MVar [arg]
217+ }
218+
219+relativeUrlConverter :: M (Text -> Text)
220+relativeUrlConverter = do
221+ burl <- asks baseUrl
222+ pure $ \end ->
223+ if
224+ | "/" `T.isPrefixOf` end && "/" `T.isSuffixOf` burl -> burl <> T.drop 1 end
225+ | "/" `T.isPrefixOf` end || "/" `T.isSuffixOf` burl -> burl <> end
226+ | otherwise -> burl <> "/" <> end
227+
228+newtype M a = M {runM' :: ExceptT ErrorWithContext (ReaderT Env (ForkT IO)) a}
229+ deriving newtype
230+ ( Functor,
231+ Applicative,
232+ Monad,
233+ MonadError ErrorWithContext,
234+ MonadIO,
235+ MonadBase IO,
236+ MonadThrow,
237+ MonadCatch,
238+ MonadMask,
239+ MonadReader Env,
240+ MonadBaseControl IO,
241+ HasForkT
242+ )
243+
244+runM :: Env -> M a -> IO (Either ErrorWithContext a)
245+runM env (M action) = runForkT (runReaderT (runExceptT action) env)
246+
247+newtype FlakeDir = FlakeDir {__unsafeGetFlakeDir :: FilePath}
248+ deriving newtype (Eq, Show, Generic)
249+
250+newtype ReportSummary = ReportSummary {getReportSummary :: Text}
251+ deriving stock (Show, Eq)
252+
253+data RunReporter = RunReporter
254+ { reportLogs :: LogLine -> M (),
255+ reportComplete :: RunReportStatus -> M (),
256+ ghRunId :: Maybe GhRunId
257+ }
258+
259+instance Semigroup RunReporter where
260+ a <> b =
261+ RunReporter
262+ { reportLogs = \logs -> reportLogs a logs >> reportLogs b logs,
263+ reportComplete = \status -> reportComplete a status >> reportComplete b status,
264+ ghRunId = ghRunId a <|> ghRunId b
265+ }
266+
267+instance Monoid RunReporter where
268+ mempty =
269+ RunReporter
270+ { reportLogs = \_ -> pure (),
271+ reportComplete = \_ -> pure (),
272+ ghRunId = Nothing
273+ }
274+
275+data ReportType
276+ = ReportBuild {_reportTypeName :: Text, _reportTypeBuild :: Build}
277+ | ReportRun {_reportTypeRun :: Run}
278+ | MetaCheck
279+ deriving stock (Show, Eq)
280+
281+reportName :: ReportType -> Text
282+reportName = \case
283+ ReportBuild name _ -> name
284+ ReportRun run -> run ^. name
285+ MetaCheck -> "All Garnix checks"
286+
287+newtype Reporter = Reporter {createNewRun :: ReportType -> M RunReporter}
288+ deriving stock (Show)
289+
290+instance Semigroup Reporter where
291+ a <> b =
292+ Reporter
293+ { createNewRun = \reportType ->
294+ (<>) <$> createNewRun a reportType <*> createNewRun b reportType
295+ }
296+
297+instance Monoid Reporter where
298+ mempty = Reporter {createNewRun = \_ -> pure mempty}
299+
300+data FodChecker = FodChecker
301+ { runReporter :: RunReporter,
302+ totalSkipped :: MVar Int,
303+ totalVerified :: MVar Int,
304+ promises :: MVar (Maybe [Promise (Either [(Nix.DrvPath, Text)] ())]),
305+ startedOrDone :: MVar (Set Nix.DrvPath)
306+ }
307+ deriving (Generic)
308+
309+-- * Github interface
310+
311+data GhCollaborators
312+ = GhCollaborators [GhLogin]
313+ | RepoNotFound
314+ deriving (Show)
315+
316+newtype RemoteUrl = RemoteUrl Text
317+
318+realRemoteUrl :: RemoteUrl -> Text
319+realRemoteUrl (RemoteUrl url) = url
320+
321+data GithubInterface = GithubInterface
322+ { _githubInterfaceGetInstallation :: (HasCallStack) => GH.Id GHA.Installation -> M GHA.InstallationAuth,
323+ _githubInterfaceGetInstallations :: (HasCallStack) => GhToken -> M [GH.Id GHA.Installation],
324+ _githubInterfaceGetGarnixInstallationId :: (HasCallStack) => GhRepoOwner -> GhRepoName -> M (Maybe Integer),
325+ _githubInterfaceGetAccessToken :: (HasCallStack) => GHA.InstallationAuth -> M GhToken,
326+ _githubInterfaceGetDefaultBranch :: (HasCallStack) => Maybe GHA.InstallationAuth -> GhRepoOwner -> GhRepoName -> M (Maybe Branch),
327+ _githubInterfaceGetHeadCommit :: (HasCallStack) => GhToken -> GhRepoOwner -> GhRepoName -> Branch -> M CommitHash,
328+ _githubInterfaceNewBuildReport :: (HasCallStack) => RepoInfo -> GhRunReport -> M GhRunId,
329+ _githubInterfaceUpdateBuildReport :: (HasCallStack) => GhRunId -> GhRunReport -> RepoInfo -> M (),
330+ _githubInterfaceDoesRepoFileExist :: (HasCallStack) => CommitInfo -> FilePath -> M DoesFileExist,
331+ _githubInterfaceGetRemote :: (HasCallStack) => CommitInfo -> M RemoteUrl,
332+ _githubInterfaceGetRepoCollaborators :: (HasCallStack) => InstallationAuth -> GhRepoOwner -> GhRepoName -> M GhCollaborators,
333+ _githubInterfaceGetRepoPublicity :: (HasCallStack) => InstallationAuth -> GhRepoOwner -> GhRepoName -> M RepoPublicity,
334+ _githubInterfaceGetInstalledOrgs :: (HasCallStack) => GhToken -> M [GhUserOrgMembership],
335+ _githubInterfaceGetReposInInstallationAccessibleTo :: (HasCallStack) => GH.Id GHA.Installation -> GhToken -> M [Text],
336+ _githubInterfaceOpenGithubPullRequest :: (HasCallStack) => GhRepoOwner -> GhRepoName -> PullRequest -> M PullRequestResult
337+ }
338+
339+data RunReportStatus
340+ = RunReportStatusInProgress
341+ | RunReportStatusSuccess
342+ | RunReportStatusFailure
343+ | RunReportStatusTimeout
344+ | RunReportStatusCancelled
345+ deriving stock (Eq, Show)
346+
347+data GhRunReport = GhRunReport
348+ { _ghRunReportName :: Text,
349+ _ghRunReportCommit :: CommitHash,
350+ _ghRunReportUrl :: Maybe Text,
351+ _ghRunReportStatus :: RunReportStatus,
352+ _ghRunReportTitle :: Text,
353+ _ghRunReportSummary :: Text,
354+ _ghRunReportLogs :: RawLogs
355+ }
356+ deriving stock (Eq, Show)
357+
358+data DeploymentStatus = DeploymentInProgress | DeploymentSuccess | DeploymentFailure
359+
360+data PullRequest = PullRequest
361+ { _pullRequestTitle :: Text,
362+ _pullRequestBody :: Text,
363+ _pullRequestHeadBranch :: Branch,
364+ _pullRequestBaseBranch :: Branch
365+ }
366+
367+-- * HetznerInterface
368+
369+data HetznerInterface = HetznerInterface
370+ { _hetznerInterfaceProvisionServer :: PreprovisionedServerId -> HetznerLocation -> HetznerServerType -> M PreprovisionedServer,
371+ _hetznerInterfaceUpdateMetadata :: RepoInfo -> DeploymentType -> Build -> ServerId -> HetznerServerId -> M (),
372+ _hetznerInterfaceDeleteServer :: HetznerServerId -> M (),
373+ _hetznerInterfaceGetServerStatus :: HetznerServerId -> M Text
374+ }
375+
376+makeFields ''PullRequest
377+makeFields ''EnvMocks
378+makeFields ''GhRunReport
379+
380+-- Accessors
381+
382+getInstallation :: GH.Id GHA.Installation -> M GHA.InstallationAuth
383+getInstallation inst = do
384+ gh <- view #githubInterface
385+ _githubInterfaceGetInstallation gh inst
386+
387+getInstallations :: GhToken -> M [GH.Id GHA.Installation]
388+getInstallations token = do
389+ gh <- view #githubInterface
390+ _githubInterfaceGetInstallations gh token
391+
392+getGarnixInstallationId :: GhRepoOwner -> GhRepoName -> M (Maybe Integer)
393+getGarnixInstallationId owner name = do
394+ gh <- view #githubInterface
395+ _githubInterfaceGetGarnixInstallationId gh owner name
396+
397+getAccessToken :: GHA.InstallationAuth -> M GhToken
398+getAccessToken iAuth = do
399+ gh <- view #githubInterface
400+ _githubInterfaceGetAccessToken gh iAuth
401+
402+getDefaultBranch :: Maybe GHA.InstallationAuth -> GhRepoOwner -> GhRepoName -> M (Maybe Branch)
403+getDefaultBranch miAuth owner repo = do
404+ gh <- view #githubInterface
405+ _githubInterfaceGetDefaultBranch gh miAuth owner repo
406+
407+getHeadCommit :: GhToken -> GhRepoOwner -> GhRepoName -> Branch -> M CommitHash
408+getHeadCommit token owner repo branch = do
409+ gh <- view #githubInterface
410+ _githubInterfaceGetHeadCommit gh token owner repo branch
411+
412+newBuildReport :: RepoInfo -> GhRunReport -> M GhRunId
413+newBuildReport repoInfo build' = do
414+ gh <- view #githubInterface
415+ _githubInterfaceNewBuildReport gh repoInfo build'
416+
417+updateBuildReport :: GhRunId -> GhRunReport -> RepoInfo -> M ()
418+updateBuildReport runId' runReport repoInfo = do
419+ gh <- view #githubInterface
420+ _githubInterfaceUpdateBuildReport gh runId' runReport repoInfo
421+
422+getRemote :: (HasCallStack) => CommitInfo -> M RemoteUrl
423+getRemote commitInfo = do
424+ gh <- view #githubInterface
425+ _githubInterfaceGetRemote gh commitInfo
426+
427+getRepoCollaborators :: (HasCallStack) => InstallationAuth -> GhRepoOwner -> GhRepoName -> M GhCollaborators
428+getRepoCollaborators iAuth owner repo = do
429+ gh <- view #githubInterface
430+ _githubInterfaceGetRepoCollaborators gh iAuth owner repo
431+
432+doesRepoFileExist :: (HasCallStack) => CommitInfo -> FilePath -> M DoesFileExist
433+doesRepoFileExist commitInfo path = do
434+ gh <- view #githubInterface
435+ _githubInterfaceDoesRepoFileExist gh commitInfo path
436+
437+getRepoPublicity :: (HasCallStack) => InstallationAuth -> GhRepoOwner -> GhRepoName -> M RepoPublicity
438+getRepoPublicity iAuth owner name = do
439+ gh <- view #githubInterface
440+ _githubInterfaceGetRepoPublicity gh iAuth owner name
441+
442+getInstalledOrgs :: (HasCallStack) => GhToken -> M [GhUserOrgMembership]
443+getInstalledOrgs tok = do
444+ gh <- view #githubInterface
445+ _githubInterfaceGetInstalledOrgs gh tok
446+
447+getReposInInstallationAccessibleTo :: (HasCallStack) => GH.Id GHA.Installation -> GhToken -> M [Text]
448+getReposInInstallationAccessibleTo installation token = do
449+ gh <- view #githubInterface
450+ _githubInterfaceGetReposInInstallationAccessibleTo gh installation token
451+
452+openGithubPullRequest :: (HasCallStack) => GhRepoOwner -> GhRepoName -> PullRequest -> M PullRequestResult
453+openGithubPullRequest owner name pr = do
454+ gh <- view #githubInterface
455+ _githubInterfaceOpenGithubPullRequest gh owner name pr
456+
457+withWreqOptions :: (Wreq.Options -> IO a) -> M a
458+withWreqOptions action = do
459+ manager <- view #manager
460+ let options = Wreq.defaults & Wreq.manager .~ Right manager
461+ liftIO $ action options
462+
463+provisionServer :: PreprovisionedServerId -> HetznerLocation -> HetznerServerType -> M PreprovisionedServer
464+provisionServer sId loc typ = do
465+ iface <- view #hetznerInterface
466+ _hetznerInterfaceProvisionServer iface sId loc typ
467+
468+updateMetadata :: RepoInfo -> DeploymentType -> Build -> ServerId -> HetznerServerId -> M ()
469+updateMetadata repoInfo deploymentType build serverId hetznerServerId = do
470+ iface <- view #hetznerInterface
471+ _hetznerInterfaceUpdateMetadata iface repoInfo deploymentType build serverId hetznerServerId
472+
473+deleteServer :: HetznerServerId -> M ()
474+deleteServer sId = do
475+ iface <- view #hetznerInterface
476+ _hetznerInterfaceDeleteServer iface sId
477+
478+getServerStatus :: HetznerServerId -> M Text
479+getServerStatus sId = do
480+ iface <- view #hetznerInterface
481+ _hetznerInterfaceGetServerStatus iface sId
482+
483+getNixXdgCacheDir :: M String
484+getNixXdgCacheDir =
485+ view #nixXdgCacheDir
486+ >>= maybe (throw $ OtherError "Local cache not properly set up.") pure
487+
488+newtype RequestTraceId = RequestTraceId {getRequestTraceId :: Data.UUID.UUID}
489+
490+instance Loggable RequestTraceId where
491+ asLog (RequestTraceId uuid) = [("request_trace_id", Data.UUID.toText uuid)]
492+
493+-- * mocking
494+
495+newMock :: (a -> M b) -> IO (Mock a b)
496+newMock f = do
497+ Mock f <$> newMVar []
498+
499+mockable ::
500+ Lens' EnvMocks (Maybe (Mock arg result)) ->
501+ (arg -> M result) ->
502+ (arg -> M result)
503+mockable lens prodImplementation arg = do
504+ mocks <- view #mocks
505+ case mocks >>= (^. lens) of
506+ Nothing -> prodImplementation arg
507+ Just (Mock mock calls) -> do
508+ liftIO $ modifyMVar_ calls $ \acc -> pure (acc ++ [arg])
509+ mock arg
510+
511+withMock ::
512+ Lens' EnvMocks (Maybe (Mock arg result)) ->
513+ (arg -> M result) ->
514+ M a ->
515+ M a
516+withMock lens mock action = do
517+ mocks <- view #mocks
518+ case mocks of
519+ Nothing -> log Critical "trying to mock during production"
520+ Just _ -> pure ()
521+ calls <- liftIO $ newMVar []
522+ local (setMock calls) action
523+ where
524+ setMock calls env =
525+ env & #mocks . _Just . lens ?~ Mock mock calls
526+
527+withUnmock :: Lens' EnvMocks (Maybe (Mock arg result)) -> M a -> M a
528+withUnmock lens action = do
529+ local unsetMock action
530+ where
531+ unsetMock env = case mocks env of
532+ Nothing -> env
533+ Just mocks ->
534+ env {mocks = Just (mocks & lens .~ Nothing)}
535+
536+withMockReturning ::
537+ Lens' EnvMocks (Maybe (Mock arg result)) ->
538+ result ->
539+ M a ->
540+ M a
541+withMockReturning lens result =
542+ withMock lens $ const $ pure result
543+
544+getMockCalls ::
545+ Lens' EnvMocks (Maybe (Mock arg result)) ->
546+ M [arg]
547+getMockCalls lens = do
548+ mocks <- view #mocks
549+ case mocks >>= (^. lens) of
550+ Just mock -> liftIO $ readMVar $ calls mock
551+ Nothing -> throw $ OtherError "getMockCalls called for unmocked function"
552+
553+-- * logging
554+
555+withDefaultLogger :: ((LogItem -> IO ()) -> IO a) -> IO a
556+withDefaultLogger action =
557+ bracket
558+ (FastLogger.newFastLogger1 $ FastLogger.LogStdout FastLogger.defaultBufSize)
559+ snd
560+ (\(logger, _) -> action (\x -> logger $ FastLogger.toLogStr x <> "\n"))
561+
562+log :: Severity -> Text -> M ()
563+log sev txt = do
564+ spans <- asks spanCtx
565+ logger <- asks logger
566+ liftIO $ logger (LogItem sev (nubOrd spans) (limitMessage txt))
567+ case sev of
568+ Critical -> incrementEvent #logsCritical
569+ Error -> incrementEvent #logsError
570+ Warning -> incrementEvent #logsWarning
571+ _ -> pure ()
572+ where
573+ limit = 20000
574+ limitMessage text =
575+ if T.length text > limit
576+ then T.take limit text <> "...[snip]"
577+ else text
578+
579+logRequestsMiddleware :: Env -> (RequestTraceId -> Wai.Application) -> Wai.Application
580+logRequestsMiddleware env innerApp request respond = do
581+ requestTraceId <- RequestTraceId <$> liftIO Data.UUID.V4.nextRandom
582+ innerApp requestTraceId request $ \response -> do
583+ result <- respond response
584+ void $ runM env $ do
585+ let spans =
586+ [ ("method", cs $ Wai.requestMethod request),
587+ ("path", cs $ Wai.rawPathInfo request),
588+ ("http_version", show $ Wai.httpVersion request),
589+ ("status", show $ statusCode $ Wai.responseStatus response),
590+ ("response_content_length", maybe "-" show $ contentLength $ Wai.responseHeaders response),
591+ ("referrer", maybe "-" cs $ Wai.requestHeaderReferer request),
592+ ("user_agent", maybe "-" cs $ Wai.requestHeaderUserAgent request)
593+ ]
594+ withSpan requestTraceId
595+ $ withTextSpans spans
596+ $ do
597+ log Informational "http request"
598+ pure result
599+
600+withSpan :: (Loggable l) => l -> M a -> M a
601+withSpan toAdd = withTextSpans (asLog toAdd)
602+
603+withTextSpan :: (Text, Text) -> M a -> M a
604+withTextSpan toAdd = withTextSpans [toAdd]
605+
606+withTextSpans :: [(Text, Text)] -> M a -> M a
607+withTextSpans toAdd = local $ \env -> env {spanCtx = map addSpanPrefix toAdd <> spanCtx env}
608+ where
609+ addSpanPrefix :: (Text, Text) -> (Text, Text)
610+ addSpanPrefix = first ("span_" <>)
611+
612+logThrownErrors :: M a -> M a
613+logThrownErrors action = action `catchError` (\e -> logError e >> rethrow e)
614+
615+logError :: ErrorWithContext -> M ()
616+logError error = withRawTextSpans (spans error) $ do
617+ log (error ^. #severity) $ showDebug error
618+ where
619+ withRawTextSpans :: [(Text, Text)] -> M a -> M a
620+ withRawTextSpans rawSpans = local $ \env -> env {spanCtx = rawSpans <> spanCtx env}
621+
622+logSomeException :: SomeException -> M ()
623+logSomeException error =
624+ log Error $ "runtime exception: " <> show error
625+
626+(<?>) :: M a -> Text -> M a
627+action <?> msg = do
628+ log Notice msg
629+ result <-
630+ action `whenErrorEither` \e -> do
631+ log Warning (msg <> " - FAILED: " <> either show showDebug e)
632+ log Notice (msg <> " - DONE")
633+ pure result
634+
635+withMessage :: Text -> M a -> M a
636+withMessage = flip (<?>)
637+
638+logDuration :: Text -> M a -> M a
639+logDuration message action = do
640+ before <- liftIO getSystemTime
641+ result <- action
642+ after <- liftIO getSystemTime
643+ let seconds = nominalDiffTimeToSeconds $ diffUTCTime (systemToUTCTime after) (systemToUTCTime before)
644+ log Informational $ "logDuration: " <> message <> ", seconds: " <> prettyF (PrettyCfg 2 Nothing '.') seconds
645+ pure result
646+
647+-- * error handling
648+
649+throwWithSeverity :: (HasCallStack) => Severity -> Error -> M a
650+throwWithSeverity severity e = do
651+ spans <- asks spanCtx
652+ throwError $ ErrorWithContext {callstack = callStack, spans, severity, err = e}
653+
654+throw :: (HasCallStack) => Error -> M a
655+throw = throwWithSeverity Error
656+
657+shortcut :: (HasCallStack) => Error -> M a
658+shortcut = throwWithSeverity Informational
659+
660+rethrow :: ErrorWithContext -> M a
661+rethrow = throwError
662+
663+catchIfErrorMatches :: M a -> IndexedTraversal' Int T.Text RE.Match -> (NonEmpty [Text] -> M a) -> M a
664+catchIfErrorMatches action regex fallback = do
665+ action
666+ `catchError` ( \e -> case err e of
667+ RunProcessError {..} -> case stdErr ^.. regex . RE.groups of
668+ [] -> throwError e
669+ m : rest -> fallback (m :| rest)
670+ _ -> throwError e
671+ )
672+
673+catchEither :: M a -> (Either SomeException ErrorWithContext -> M a) -> M a
674+catchEither action handler = (action `catchError` (handler . Right)) `catchAny` (handler . Left)
675+
676+rethrowEither :: Either SomeException ErrorWithContext -> M a
677+rethrowEither = \case
678+ Right e -> rethrow e
679+ Left e -> SafeException.throwIO e
680+
681+tryEither :: M a -> M (Either (Either SomeException ErrorWithContext) a)
682+tryEither action = catchEither (Right <$> action) (pure . Left)
683+
684+-- | Executes a given clean-up function on exceptions and errors.
685+-- Runs the clean-up functions also for async exceptions.
686+whenErrorEither :: M a -> (Either SomeException ErrorWithContext -> M ()) -> M a
687+whenErrorEither action onError =
688+ ( action
689+ `SafeException.withException` (onError . Left)
690+ )
691+ `catchError` ( \e -> do
692+ onError $ Right e
693+ throwError e
694+ )
695+
696+-- Catches both IO and MonadError errors, logs them at Critical, but otherwise
697+-- ignores them
698+ignoringAllErrors :: M a -> M ()
699+ignoringAllErrors action =
700+ (void action `catchError` logIt) `SafeException.catchAny` logIt
701+ where
702+ logIt x = log Critical $ "Got an error (ignored): " <> show x
703+
704+aesonDecode :: (HasCallStack) => Text -> (Value -> Parser a) -> Text -> M a
705+aesonDecode description parser json = do
706+ case eitherDecode' (cs json) >>= Data.Aeson.Types.parseEither parser of
707+ Left e ->
708+ throw
709+ $ DecodeError
710+ { original = json,
711+ message =
712+ "Could not decode "
713+ <> description
714+ <> ". Error was: "
715+ <> cs e
716+ }
717+ Right parsed -> pure parsed
718+
719+safeGetAbsoluteFlakeDir :: FlakeDir -> M FilePath
720+safeGetAbsoluteFlakeDir (FlakeDir flakeDir) = do
721+ curDir <- view #workingDir
722+ result <- liftIO $ canonicalizePath (curDir </> flakeDir)
723+ if result == curDir || (curDir <> "/") `isPrefixOf` result
724+ then pure result
725+ else do
726+ throw $ OtherError $ "'" <> cs flakeDir <> "' is not a path within the repo"
backend/src/Garnix/Monad/Async.hsnew file+68
@@ -0,0 +1,68 @@
1+module Garnix.Monad.Async where
2+
3+import Control.Concurrent (forkFinally, forkIO, newEmptyMVar, newMVar, putMVar, readMVar)
4+import Control.Exception (throwIO)
5+import Control.Monad.Trans.Control (control)
6+import Garnix.Async
7+import Garnix.Duration
8+import Garnix.Monad (M, log, logError, runM, throw)
9+import Garnix.Prelude
10+import Garnix.Types (Error, Severity (..))
11+
12+spawn :: M a -> M (Promise a)
13+spawn action = do
14+ env <- ask
15+ mvar <- liftIO newEmptyMVar
16+ let go r = putMVar mvar $ case r of
17+ Left exception -> PromiseException exception
18+ Right (Left err) -> PromiseThrew err
19+ Right (Right val) -> PromiseSucceeded val
20+ _ <- liftIO $ forkFinally (runM env action) go
21+ pure $ Promise mvar
22+
23+resolve :: Promise a -> M a
24+resolve p =
25+ liftIO (resolveIO p) >>= \case
26+ PromiseException e -> liftIO $ throwIO e
27+ PromiseThrew e -> throwError e
28+ PromiseSucceeded a -> pure a
29+
30+emptyPromise :: M (Promise ())
31+emptyPromise = do
32+ Promise <$> liftIO (newMVar $ PromiseSucceeded ())
33+
34+logPromiseErrors :: (Show a) => Promise a -> M ()
35+logPromiseErrors (Promise mvar) = do
36+ control $ \runInBase -> do
37+ void $ forkIO $ do
38+ result <- readMVar mvar
39+ void $ runInBase $ case result of
40+ PromiseException exception -> log Error (show exception)
41+ PromiseThrew e -> logError e
42+ PromiseSucceeded _ -> pure ()
43+ pure $ Right ()
44+
45+timeoutThrowing :: Duration -> Error -> M a -> M a
46+timeoutThrowing delay error action = do
47+ result <- timeout delay action
48+ case result of
49+ Just a -> pure a
50+ Nothing -> Garnix.Monad.throw error
51+
52+joinAll_ :: [Promise a] -> M (Promise ())
53+joinAll_ rs = liftIO $ do
54+ mvar <- liftIO newEmptyMVar
55+ _ <- forkFinally (mapM resolveIO rs) (putMVar mvar . either PromiseException sequence_)
56+ pure $ Promise mvar
57+
58+joinAll :: [Promise a] -> M (Promise [a])
59+joinAll promises = liftIO $ do
60+ mvar <- liftIO newEmptyMVar
61+ _ <- forkFinally (mapM resolveIO promises) (putMVar mvar . either PromiseException sequence)
62+ pure $ Promise mvar
63+
64+-- | Like async's forConcurrently, but does not cancel if any action throws
65+doAllConcurrently :: [a] -> (a -> M b) -> M (Promise ())
66+doAllConcurrently v act = do
67+ promises <- forM v $ spawn . act
68+ joinAll_ promises
backend/src/Garnix/Monad/Bubbling.hsnew file+22
@@ -0,0 +1,22 @@
1+module Garnix.Monad.Bubbling where
2+
3+import Control.Exception.Safe qualified as SafeException
4+import Garnix.Monad
5+import Garnix.Prelude
6+
7+withBubbling ::
8+ (Typeable e, Show e) =>
9+ ((forall a. Either e a -> M a) -> M o) ->
10+ M (Either e o)
11+withBubbling action = do
12+ let bubble = \case
13+ Right a -> pure a
14+ Left error -> SafeException.throwIO (BubblingError error)
15+ SafeException.handle
16+ (\(BubblingError error) -> pure $ Left error)
17+ (Right <$> action bubble)
18+
19+data BubblingError e = BubblingError e
20+ deriving stock (Show)
21+
22+instance (Typeable e, Show e) => Exception (BubblingError e)
backend/src/Garnix/Monad/Concurrency.hsnew file+17
@@ -0,0 +1,17 @@
1+module Garnix.Monad.Concurrency where
2+
3+import Garnix.Monad
4+import Garnix.Monad.ForkT (safeFork)
5+import Garnix.Prelude
6+
7+forkM :: M () -> M ()
8+forkM action = do
9+ _ <- safeFork $ do
10+ result <- tryEither action
11+ case result of
12+ Left (Left exception) -> do
13+ logSomeException exception
14+ Left (Right error) -> do
15+ logError error
16+ Right () -> pure ()
17+ pure ()
backend/src/Garnix/Monad/ForkT.hsnew file+107
@@ -0,0 +1,107 @@
1+{-# LANGUAGE UndecidableInstances #-}
2+
3+module Garnix.Monad.ForkT
4+ ( ForkT,
5+ HasForkT,
6+ runForkT,
7+ safeFork,
8+ safeAcquire,
9+ safeSystemTempDirectory,
10+ safeSystemTempFile,
11+ )
12+where
13+
14+import Control.Concurrent.Lifted (MVar, modifyMVar_, newEmptyMVar, newMVar, putMVar, readMVar, takeMVar)
15+import Control.Exception.Safe qualified as Safe
16+import Control.Monad.Base
17+import Control.Monad.Trans.Control
18+import Garnix.Prelude
19+import System.Directory (removeDirectoryRecursive, removeFile)
20+import System.IO (hClose, openTempFile)
21+import System.IO.Temp (createTempDirectory, getCanonicalTemporaryDirectory)
22+
23+data ForkState = ForkState
24+ { threads :: [MVar ()],
25+ releasers :: [IO ()]
26+ }
27+ deriving stock (Generic)
28+
29+instance Default ForkState where
30+ def = ForkState [] []
31+
32+newtype ForkT m a = ForkT (ReaderT (MVar ForkState) m a)
33+ deriving newtype
34+ ( Functor,
35+ Applicative,
36+ Monad,
37+ MonadIO,
38+ MonadTrans,
39+ MonadTransControl,
40+ MonadBaseControl base,
41+ MonadThrow,
42+ MonadCatch,
43+ MonadMask
44+ )
45+
46+runForkT :: (MonadBase IO m) => ForkT m a -> m a
47+runForkT (ForkT inner) = do
48+ mvar <- newMVar def
49+ a <- runReaderT inner mvar
50+ ForkState {threads, releasers} <- readMVar mvar
51+ forM_ threads $ \thread -> do
52+ takeMVar thread
53+ results :: [Either SomeException ()] <- forM releasers $ \releaser -> do
54+ liftBase $ Safe.try releaser
55+ forM_ results $ \result -> do
56+ case result of
57+ Left exception -> liftBase $ Safe.throwIO exception
58+ Right () -> pure ()
59+ pure a
60+
61+class (MonadBaseControl IO m, MonadMask m) => HasForkT m where
62+ forkTAsk :: m (MVar ForkState)
63+
64+instance {-# OVERLAPS #-} (MonadMask m, MonadBaseControl IO m) => HasForkT (ForkT m) where
65+ forkTAsk :: ForkT m (MVar ForkState)
66+ forkTAsk = ForkT ask
67+
68+instance
69+ {-# OVERLAPS #-}
70+ (MonadBaseControl IO (t m), MonadMask (t m), MonadTrans t, Monad m, HasForkT m) =>
71+ HasForkT (t m)
72+ where
73+ forkTAsk = lift forkTAsk
74+
75+safeFork :: (HasForkT m) => m () -> m ThreadId
76+safeFork action = do
77+ threadEnded <- newEmptyMVar
78+ threadId <- fork (action `finally` putMVar threadEnded ())
79+ forkState <- forkTAsk
80+ modifyMVar_ forkState (pure . (#threads %~ (threadEnded :)))
81+ pure threadId
82+
83+instance (MonadBase base m) => MonadBase base (ForkT m) where
84+ liftBase = ForkT . liftBase
85+
86+safeAcquire :: (HasForkT m) => m resource -> (resource -> IO ()) -> m resource
87+safeAcquire acquire release = do
88+ resource <- acquire
89+ forkState <- forkTAsk
90+ modifyMVar_ forkState (pure . (#releasers %~ (release resource :)))
91+ pure resource
92+
93+-- * convenience helpers
94+
95+safeSystemTempDirectory :: (HasForkT m) => Text -> m FilePath
96+safeSystemTempDirectory snippet = do
97+ systemTempDir <- liftBase getCanonicalTemporaryDirectory
98+ safeAcquire
99+ (liftBase $ createTempDirectory systemTempDir (cs snippet))
100+ removeDirectoryRecursive
101+
102+safeSystemTempFile :: (HasForkT m) => String -> m (FilePath, Handle)
103+safeSystemTempFile snippet = do
104+ systemTempDir <- liftBase getCanonicalTemporaryDirectory
105+ safeAcquire
106+ (liftBase $ openTempFile systemTempDir snippet)
107+ (\(path, handle) -> liftBase (hClose handle >> removeFile path))
backend/src/Garnix/Monad/Memoization.hsnew file+35
@@ -0,0 +1,35 @@
1+module Garnix.Monad.Memoization where
2+
3+import Control.Concurrent.Lifted (MVar, modifyMVar)
4+import Control.Lens
5+import Data.HashTable.IO qualified as HashTables
6+import Data.HashTable.IO qualified as HashTables.HashTables
7+import Data.Hashable (Hashable)
8+import Garnix.Prelude
9+
10+type MemoTable key result = HashTables.HashTables.LinearHashTable key result
11+
12+memoize ::
13+ forall context m input output.
14+ (MonadReader context m, Hashable input, MonadIO m, MonadBaseControl IO m) =>
15+ Getting (MVar (MemoTable input output)) context (MVar (MemoTable input output)) ->
16+ input ->
17+ m output ->
18+ m output
19+memoize cacheLens key action = do
20+ cached <- withCache $ \cache -> do
21+ liftIO $ HashTables.lookup cache key
22+ case cached of
23+ Just result -> pure result
24+ Nothing -> do
25+ result <- action
26+ withCache $ \cache -> do
27+ liftIO $ HashTables.insert cache key result
28+ pure result
29+ where
30+ withCache :: (MemoTable input output -> m a) -> m a
31+ withCache action = do
32+ cacheMVar <- view cacheLens
33+ modifyMVar cacheMVar $ \cache -> do
34+ result <- action cache
35+ pure (cache, result)
backend/src/Garnix/Monad/Metrics.hsnew file+202
@@ -0,0 +1,202 @@
1+module Garnix.Monad.Metrics where
2+
3+import Data.Generics.Product (HasField')
4+import Garnix.Prelude
5+import Network.Wai.Handler.Warp (Port, run)
6+import System.Metrics.Prometheus.Concurrent.Registry
7+import System.Metrics.Prometheus.Http.Scrape qualified as Prom
8+import System.Metrics.Prometheus.Metric.Counter (Counter, inc)
9+import System.Metrics.Prometheus.Metric.Gauge (Gauge)
10+import System.Metrics.Prometheus.Metric.Histogram (Histogram, observe)
11+import System.Metrics.Prometheus.MetricId (fromList)
12+
13+-- For consistency, all time-related Doubles should be seconds
14+data Metrics = Metrics
15+ { evalQueueLen :: Gauge,
16+ evalQueueWaitTime :: Histogram,
17+ -- | Including queue wait time
18+ evalDrvPathTime :: Histogram,
19+ -- | Including queue wait time
20+ getAttrsToBuildTime :: Histogram,
21+ s3QueueLen :: Gauge,
22+ s3QueueWaitTime :: Histogram,
23+ gitCloneTime :: Histogram,
24+ cachePushTime :: Histogram,
25+ cachePushSuccess :: Counter,
26+ cachePushFailure :: Counter,
27+ packageBuildsAttempted :: Counter,
28+ dbQueryTime :: Histogram,
29+ dbQueries :: Counter,
30+ logsCritical :: Counter,
31+ logsError :: Counter,
32+ logsWarning :: Counter,
33+ s3CacheFallbacksToOldCache :: Counter,
34+ s3CacheUploads :: Counter,
35+ s3CacheNarfilesServed :: Counter,
36+ fodCheckTime :: Histogram,
37+ fodCheckBatchSize :: Histogram,
38+ fodCheckQueueLen :: Gauge,
39+ fodCheckQueueWaitTime :: Histogram,
40+ registry :: Registry
41+ }
42+ deriving (Generic)
43+
44+timingAs :: (MonadIO m, MonadReader e m, HasField' "metrics" e Metrics) => Lens' Metrics Histogram -> m x -> m x
45+timingAs l action = do
46+ h <- view (#metrics . l)
47+ start <- liftIO getCurrentTime
48+ result <- action
49+ end <- liftIO getCurrentTime
50+ -- We fork to not hold up the thread (or crash it)
51+ void
52+ . liftIO
53+ $ fork
54+ $ observe
55+ (fromRational . toRational $ nominalDiffTimeToSeconds $ diffUTCTime end start)
56+ h
57+ pure result
58+
59+incrementEvent :: (MonadIO m, MonadReader e m, HasField' "metrics" e Metrics) => Lens' Metrics Counter -> m ()
60+incrementEvent l = do
61+ h <- view (#metrics . l)
62+ void . liftIO $ fork $ inc h
63+
64+registerMetrics :: IO Metrics
65+registerMetrics = do
66+ registry <- new
67+ evalQueueLen <-
68+ registerGauge
69+ "garnix_server_eval_queue_len"
70+ mempty
71+ registry
72+ evalQueueWaitTime <-
73+ registerHistogram
74+ "garnix_server_eval_queue_wait_time"
75+ mempty
76+ [0.01, 0.1, 0.5, 1, 2, 4, 6, 10, 30, 60, 120, 600]
77+ registry
78+ evalDrvPathTime <-
79+ registerHistogram
80+ "garnix_server_eval_drv_path_time"
81+ mempty
82+ [0.2, 0.5, 1, 5, 10, 30, 60, 120, 360, 600, 900]
83+ registry
84+ getAttrsToBuildTime <-
85+ registerHistogram
86+ "garnix_server_get_attrs_to_build_time"
87+ mempty
88+ [0.01, 0.1, 0.5, 1, 2, 4, 6, 10, 30, 60, 120, 600]
89+ registry
90+ s3QueueLen <-
91+ registerGauge
92+ "garnix_server_s3_queue_len"
93+ mempty
94+ registry
95+ s3QueueWaitTime <-
96+ registerHistogram
97+ "garnix_server_s3_queue_wait_time"
98+ mempty
99+ [0.2, 0.5, 1, 5, 10, 30, 60, 120, 360, 600, 900, 1500, 3000, 10000]
100+ registry
101+ gitCloneTime <-
102+ registerHistogram
103+ "garnix_server_git_clone_time"
104+ mempty
105+ [0.01, 0.1, 0.5, 1, 2, 4, 6, 10, 30, 60, 120, 300]
106+ registry
107+ cachePushTime <-
108+ registerHistogram
109+ "garnix_server_cache_push_time"
110+ mempty
111+ [0.2, 0.5, 1, 2, 5, 10, 30, 60, 120, 300, 600]
112+ registry
113+ cachePushSuccess <-
114+ registerCounter
115+ "garnix_server_cache_push_success"
116+ mempty
117+ registry
118+ cachePushFailure <-
119+ registerCounter
120+ "garnix_server_cache_push_failure"
121+ mempty
122+ registry
123+ packageBuildsAttempted <-
124+ registerCounter
125+ "garnix_server_package_builds_attempted"
126+ mempty
127+ registry
128+ dbQueries <-
129+ registerCounter
130+ "garnix_server_db_queries"
131+ mempty
132+ registry
133+ dbQueryTime <-
134+ registerHistogram
135+ "garnix_server_db_query_time"
136+ mempty
137+ [0.05, 0.1, 0.5, 1, 2, 4, 6, 10, 30, 60, 120, 300]
138+ registry
139+ let errorCountName = "garnix_server_log_errors_total"
140+ severity = "severity"
141+ logsCritical <-
142+ registerCounter
143+ errorCountName
144+ (fromList [(severity, "critical")])
145+ registry
146+ logsError <-
147+ registerCounter
148+ errorCountName
149+ (fromList [(severity, "error")])
150+ registry
151+ logsWarning <-
152+ registerCounter
153+ errorCountName
154+ (fromList [(severity, "warning")])
155+ registry
156+ s3CacheFallbacksToOldCache <-
157+ registerCounter
158+ "garnix_s3_cache_fallback_to_old_cache"
159+ mempty
160+ registry
161+ s3CacheUploads <-
162+ registerCounter
163+ "garnix_s3_cache_uploads"
164+ mempty
165+ registry
166+ s3CacheNarfilesServed <-
167+ registerCounter
168+ "garnix_s3_cache_narfile_served"
169+ mempty
170+ registry
171+ fodCheckTime <-
172+ registerHistogram
173+ "garnix_server_fod_check_time"
174+ mempty
175+ [0.2, 0.5, 1, 5, 10, 30, 60, 120, 360, 600, 900]
176+ registry
177+ fodCheckBatchSize <-
178+ registerHistogram
179+ "garnix_server_fod_check_batch_size"
180+ mempty
181+ [1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024]
182+ registry
183+ fodCheckQueueLen <-
184+ registerGauge
185+ "garnix_server_fod_check_queue_len"
186+ mempty
187+ registry
188+ fodCheckQueueWaitTime <-
189+ registerHistogram
190+ "garnix_server_fod_check_queue_wait_time"
191+ mempty
192+ [0.2, 0.5, 1, 5, 10, 30, 60, 120, 360, 600, 900, 1500, 3000, 10000]
193+ registry
194+ pure $ Metrics {..}
195+
196+serveMetrics :: Port -> Metrics -> IO ()
197+serveMetrics port m = do
198+ void
199+ $ fork
200+ $ run port
201+ $ Prom.prometheusApp []
202+ $ sample (registry m)
backend/src/Garnix/Monad/NoThrow.hsnew file+29
@@ -0,0 +1,29 @@
1+module Garnix.Monad.NoThrow where
2+
3+import Control.Concurrent.Async.Lifted qualified
4+import Control.Exception.Safe qualified as SafeException
5+import Garnix.Duration
6+import Garnix.Monad
7+import Garnix.Prelude
8+
9+forkForever :: Duration -> M () -> M ThreadId
10+forkForever delay action =
11+ fork . forever $ do
12+ action
13+ `catchError` logError
14+ `SafeException.catchAny` logSomeException
15+ threadDelay delay
16+
17+replicateConcurrently_ :: Int -> M a -> M ()
18+replicateConcurrently_ n action = do
19+ Control.Concurrent.Async.Lifted.replicateConcurrently_ n $ do
20+ void action
21+ `catchError` logError
22+ `SafeException.catchAny` logSomeException
23+
24+forConcurrently_ :: (Foldable f) => f a -> (a -> M b) -> M ()
25+forConcurrently_ as action = do
26+ Control.Concurrent.Async.Lifted.forConcurrently_ as $ \a -> do
27+ void (action a)
28+ `catchError` logError
29+ `SafeException.catchAny` logSomeException
backend/src/Garnix/Monad/Polling.hsnew file+27
@@ -0,0 +1,27 @@
1+module Garnix.Monad.Polling where
2+
3+import Garnix.Duration
4+import Garnix.Monad (M, throw)
5+import Garnix.Prelude
6+import Garnix.Types (Error (OtherError))
7+
8+data PollingConfig = PollingConfig
9+ { interval :: Duration,
10+ cutOff :: Duration
11+ }
12+
13+withPolling :: PollingConfig -> M (Maybe a) -> M a
14+withPolling cfg action = do
15+ startPoll <- liftIO getCurrentTime
16+ inner startPoll
17+ where
18+ inner startPoll = do
19+ result <- action
20+ case result of
21+ Nothing -> do
22+ now <- liftIO getCurrentTime
23+ when (diffTime now startPoll > cutOff cfg) $ do
24+ throw $ OtherError "withPolling: Timed out waiting for action to complete"
25+ threadDelay $ interval cfg
26+ inner startPoll
27+ Just value -> pure value
backend/src/Garnix/Monad/Pool.hsnew file+121
@@ -0,0 +1,121 @@
1+module Garnix.Monad.Pool (Pool, newPool, withPoolM, withPool) where
2+
3+import Control.Concurrent.Lifted (MVar, modifyMVar, newEmptyMVar, newMVar, putMVar, takeMVar)
4+import Data.Generics.Product (HasField')
5+import Data.Sequence (Seq ((:<|)), (|>))
6+import Data.Sequence qualified as Seq
7+import Garnix.Monad.Metrics
8+import Garnix.Prelude
9+import System.Metrics.Prometheus.Metric.Gauge (Gauge, set)
10+import System.Metrics.Prometheus.Metric.Histogram (Histogram)
11+
12+newtype FairQSem a
13+ = FairQSem (MVar (Queue a))
14+
15+data Queue a
16+ = SlotsAvailable
17+ { count :: Int
18+ }
19+ | BackedUp
20+ { scheduled :: Seq (PerKeyQueue a)
21+ }
22+
23+data PerKeyQueue a = PerKeyQueue
24+ { key :: a,
25+ queue :: Seq (MVar ())
26+ }
27+
28+newQSem :: Gauge -> Int -> IO (FairQSem a)
29+newQSem gauge initial = do
30+ let q = SlotsAvailable initial
31+ updateGauge gauge q
32+ sem <- newMVar q
33+ return (FairQSem sem)
34+
35+modifyAndUpdateGauge :: Gauge -> FairQSem a -> (Queue a -> IO (Queue a, output)) -> IO output
36+modifyAndUpdateGauge gauge (FairQSem q) action = do
37+ modifyMVar q $ \queue -> do
38+ (newQueue, output) <- action queue
39+ updateGauge gauge newQueue
40+ pure (newQueue, output)
41+
42+updateGauge :: Gauge -> Queue a -> IO ()
43+updateGauge g q = do
44+ flip set g $ realToFrac $ case q of
45+ SlotsAvailable n -> -n
46+ BackedUp seq -> foldl' (+) 0 $ fmap (length . queue) seq
47+
48+-- | Like the traditional 'waitQSem', but takes an extra argument, the key for the
49+-- fairness property. After a threads with key k is woken up, another one with
50+-- the same key will only be woken up after all waiting keys have had one thread
51+-- served.
52+--
53+-- In this way, FairQSem is round-robin on the waiting keys.
54+waitQSem :: (Show a, Eq a) => Gauge -> a -> FairQSem a -> IO ()
55+waitQSem gauge v q = do
56+ res <- schedule
57+ case res of
58+ Nothing -> pure ()
59+ Just me -> takeMVar me
60+ where
61+ schedule :: IO (Maybe (MVar ()))
62+ schedule = modifyAndUpdateGauge gauge q $ \queue -> do
63+ case queue of
64+ BackedUp {scheduled} -> do
65+ me <- newEmptyMVar
66+ pure (BackedUp {scheduled = signMeUp me scheduled}, Just me)
67+ SlotsAvailable {count = 0} -> do
68+ me <- newEmptyMVar
69+ pure (BackedUp {scheduled = signMeUp me Seq.Empty}, Just me)
70+ SlotsAvailable {count} -> do
71+ pure (SlotsAvailable (count - 1), Nothing)
72+ signMeUp me s = case Seq.spanl (\PerKeyQueue {key} -> v /= key) s of
73+ (aheadOfMe, PerKeyQueue {queue} :<| behindMe) ->
74+ aheadOfMe <> (PerKeyQueue v (queue |> me) :<| behindMe)
75+ (aheadOfMe, Seq.Empty) -> aheadOfMe |> PerKeyQueue v (Seq.singleton me)
76+
77+signalQSem :: (Show a) => Gauge -> FairQSem a -> IO ()
78+signalQSem gauge q = do
79+ modifyAndUpdateGauge gauge q $ \queue -> do
80+ (,()) <$> case queue of
81+ SlotsAvailable {count} -> pure $ SlotsAvailable {count = count + 1}
82+ BackedUp {scheduled} -> do
83+ case scheduled of
84+ Seq.Empty -> pure $ SlotsAvailable {count = 1}
85+ -- We 'do' the first thing in the queue, *and then* shift all of
86+ -- the items in the key with the same key as that thing to the back.
87+ (PerKeyQueue {key, queue = upNext :<| rest}) :<| others -> do
88+ putMVar upNext ()
89+ let remaining =
90+ if Seq.null rest
91+ then others
92+ else others |> PerKeyQueue key rest
93+ pure $ BackedUp {scheduled = remaining}
94+ (PerKeyQueue {queue = Seq.Empty}) :<| _ -> error "impossible"
95+
96+data Pool a = Pool
97+ { _qsem :: FairQSem a,
98+ _timerMetric :: Lens' Metrics Histogram,
99+ _lenMetric :: Lens' Metrics Gauge
100+ }
101+
102+newPool :: (MonadIO m) => Int -> Metrics -> Lens' Metrics Histogram -> Lens' Metrics Gauge -> m (Pool a)
103+newPool limit metrics timerMetric lenMetric = do
104+ qsem <- liftIO $ newQSem (metrics ^. lenMetric) limit
105+ pure $ Pool qsem timerMetric lenMetric
106+
107+withPoolM :: (MonadMask m, MonadReader s m, HasField' "metrics" s Metrics, Eq a, Show a, MonadIO m) => (s -> Pool a) -> a -> m b -> m b
108+withPoolM poolLens key action = do
109+ pool <- asks poolLens
110+ withPool pool key action
111+
112+withPool :: (MonadMask m, MonadReader s m, HasField' "metrics" s Metrics, Eq a, Show a, MonadIO m) => Pool a -> a -> m b -> m b
113+withPool (Pool qsem timerMetricLens lenMetricLens) key action = do
114+ bracket_ acquire release action
115+ where
116+ acquire = do
117+ lenMetric <- view (#metrics . lenMetricLens)
118+ timingAs timerMetricLens $ liftIO $ waitQSem lenMetric key qsem
119+ release = do
120+ lenMetric <- view (#metrics . lenMetricLens)
121+ liftIO $ signalQSem lenMetric qsem
backend/src/Garnix/Monad/SubProcess.hsnew file+115
@@ -0,0 +1,115 @@
1+module Garnix.Monad.SubProcess
2+ ( withUtf8LinesStream,
3+ runGitProcess,
4+ runSubProcess,
5+ runSubProcess_,
6+ )
7+where
8+
9+import Control.Concurrent.Async.Lifted (wait, withAsync)
10+import Control.Exception (throwIO)
11+import Control.Monad.Trans.Control
12+import Cradle qualified
13+import Cradle.ProcessConfiguration qualified as Cradle
14+import Foreign.C.Error (Errno (..), ePIPE)
15+import GHC.IO.Exception (IOErrorType (ResourceVanished), IOException (IOError), ioe_errno, ioe_type)
16+import Garnix.Monad
17+import Garnix.Prelude
18+import Garnix.SafeUnix (safeCreatePipe)
19+import Garnix.Types
20+import Streaming.Prelude qualified as S
21+import System.IO (hClose, hSetEncoding, utf8)
22+
23+-- * streaming helpers
24+
25+withUtf8LinesStream ::
26+ forall m a.
27+ (MonadBaseControl IO m, MonadMask m, MonadIO m) =>
28+ (String -> m ()) ->
29+ (Handle -> m a) ->
30+ m a
31+withUtf8LinesStream consumer action = do
32+ bracket
33+ ( liftIO $ do
34+ (a, b) <- safeCreatePipe
35+ hSetBuffering a LineBuffering
36+ hSetBuffering b LineBuffering
37+ hSetEncoding a utf8
38+ hSetEncoding b utf8
39+ return (a, b)
40+ )
41+ -- This cleanup logic follows https://hackage.haskell.org/package/process-1.6.18.0/docs/src/System.Process.html#cleanupProcess
42+ ( \(readEnd, writeEnd) -> liftIO $ do
43+ ignoreSigPipe $ hClose writeEnd
44+ hClose readEnd
45+ )
46+ ( \(readEnd, writeEnd) -> do
47+ withAsync (S.mapM_ consumer (S.fromHandle readEnd)) $ \streamWriter -> do
48+ output <- action writeEnd
49+ liftIO $ hClose writeEnd
50+ wait streamWriter :: m ()
51+ pure output
52+ )
53+ where
54+ ignoreSigPipe :: IO () -> IO ()
55+ ignoreSigPipe = handle $ \e -> case e of
56+ IOError
57+ { ioe_type = ResourceVanished,
58+ ioe_errno = Just ioe
59+ }
60+ | Errno ioe == ePIPE -> return ()
61+ _ -> throwIO e
62+
63+runGitProcess :: [Text] -> M ()
64+runGitProcess args = do
65+ dir <- view #workingDir
66+ result <-
67+ Cradle.run
68+ $ Cradle.cmd "git"
69+ & Cradle.modifyEnvVar "GIT_AUTHOR_NAME" (const $ pure "garnix-bot")
70+ & Cradle.modifyEnvVar "GIT_AUTHOR_EMAIL" (const $ pure "contact@garnix.io")
71+ & Cradle.modifyEnvVar "GIT_COMMITTER_NAME" (const $ pure "garnix-bot")
72+ & Cradle.modifyEnvVar "GIT_COMMITTER_EMAIL" (const $ pure "contact@garnix.io")
73+ & Cradle.addArgs args
74+ & Cradle.setWorkingDir dir
75+ case result of
76+ (Cradle.ExitFailure code, Cradle.StdoutRaw out, Cradle.StderrRaw err) -> do
77+ log Warning
78+ $ "SubProcess.runGitProcess error: '"
79+ <> show args
80+ <> "': stdout ("
81+ <> cs out
82+ <> ") stderr ("
83+ <> cs err
84+ <> ")"
85+ throw RunProcessError {command = "git", arguments = args, stdErr = cs err, stdOut = cs out, exitCode = code}
86+ (Cradle.ExitSuccess, _, _) ->
87+ log Informational $ "SubProcess.runGitProcess: successfully ran git command: " <> show args
88+
89+runSubProcess :: (Cradle.Output o) => Cradle.ProcessConfiguration -> M o
90+runSubProcess config = do
91+ (output, exitCode, Cradle.StdoutRaw stdout, Cradle.StderrRaw stderr) <-
92+ Cradle.run config
93+ case exitCode of
94+ Cradle.ExitFailure exitCode -> do
95+ spans <- view #spanCtx
96+ let e =
97+ ErrorWithContext
98+ { callstack = callStack,
99+ spans,
100+ severity = Error,
101+ err =
102+ RunProcessError
103+ { command = cs $ Cradle.executable config,
104+ arguments = map cs $ Cradle.arguments config,
105+ stdOut = cs stdout,
106+ stdErr = cs stderr,
107+ exitCode
108+ }
109+ }
110+ log Warning $ "Command " <> cs (Cradle.executable config) <> " failed with " <> show e
111+ throwError e
112+ Cradle.ExitSuccess -> pure output
113+
114+runSubProcess_ :: Cradle.ProcessConfiguration -> M ()
115+runSubProcess_ = runSubProcess
backend/src/Garnix/Monad/SubProcess/Deprecated.hsnew file+58
@@ -0,0 +1,58 @@
1+module Garnix.Monad.SubProcess.Deprecated (runProc) where
2+
3+import Control.Exception (ErrorCall (ErrorCall), throwIO)
4+import Garnix.Monad
5+import Garnix.Prelude
6+import Garnix.Types
7+import System.Environment (getEnvironment)
8+import System.Process qualified as Proc
9+import System.Process.Typed
10+
11+runProc ::
12+ (HasCallStack) =>
13+ Text ->
14+ [Text] ->
15+ [(Text, Text)] ->
16+ M Text
17+runProc cmd args addedEnv = do
18+ dir <- view #workingDir
19+ go dir
20+ <?> ( "Running command"
21+ <> cs (show (cmd, args))
22+ <> " in dir "
23+ <> cs dir
24+ )
25+ where
26+ go dir = do
27+ environment <- liftIO $ addToParentEnv addedEnv
28+ (exitCode', out, err) <-
29+ -- See [Async process exceptions]
30+ liftIO
31+ $ Proc.readCreateProcessWithExitCode
32+ ( ( Proc.proc
33+ (cs cmd)
34+ (cs <$> args)
35+ )
36+ { Proc.cwd = Just dir,
37+ Proc.std_in = Proc.NoStream,
38+ Proc.std_out = Proc.CreatePipe,
39+ Proc.std_err = Proc.CreatePipe,
40+ Proc.env = Just environment
41+ }
42+ )
43+ ""
44+ case exitCode' of
45+ ExitSuccess -> pure $ cs out
46+ ExitFailure no -> do
47+ let error = RunProcessError cmd args (cs err) (cs out) no
48+ log Warning $ "Command " <> cs (show (cmd, args)) <> " failed with " <> show error
49+ throw error
50+
51+addToParentEnv :: [(Text, Text)] -> IO [(String, String)]
52+addToParentEnv addedEnv = do
53+ parentEnv <- getEnvironment
54+ let parentKeys :: [Text] = map (cs . fst) parentEnv
55+ forM_ addedEnv $ \(added, _) -> do
56+ when (added `elem` parentKeys) $ do
57+ throwIO $ ErrorCall $ cs $ "runProcIOEnv: overwriting environment variable: " <> added
58+ pure $ map (bimap cs cs) addedEnv ++ parentEnv
backend/src/Garnix/MonetaryCost.hsnew file+41
@@ -0,0 +1,41 @@
1+module Garnix.MonetaryCost
2+ ( MonetaryCost,
3+ usd,
4+ formatCost,
5+ roundFromUsd,
6+ getCostInCents,
7+ addCost,
8+ multiplyCost,
9+ divCost,
10+ )
11+where
12+
13+import Garnix.Prelude
14+
15+newtype MonetaryCost = MonetaryCost {getCostInCents :: Int32}
16+ deriving stock (Eq, Show, Generic, Ord)
17+ deriving newtype (ToJSON, FromJSON)
18+
19+formatCost :: MonetaryCost -> Text
20+formatCost (MonetaryCost costInCents) = "$" <> show dollars <> "." <> centsFormatted
21+ where
22+ (dollars, cents) = costInCents `divMod` 100
23+ centsFormatted
24+ | 0 <= cents && cents <= 9 = "0" <> show cents
25+ | 10 <= cents && cents <= 99 = show cents
26+ | otherwise = error "Impossible cents value"
27+
28+usd :: Int32 -> MonetaryCost
29+usd dollars = MonetaryCost $ dollars * 100
30+
31+roundFromUsd :: Double -> MonetaryCost
32+roundFromUsd dollars = MonetaryCost $ round $ dollars * 100
33+
34+addCost :: MonetaryCost -> MonetaryCost -> MonetaryCost
35+addCost (MonetaryCost a) (MonetaryCost b) = MonetaryCost (a + b)
36+
37+multiplyCost :: (Real a) => MonetaryCost -> a -> MonetaryCost
38+multiplyCost (MonetaryCost costInCents) multiplier = roundFromUsd $ fromIntegral costInCents * realToFrac multiplier / 100.0
39+
40+divCost :: MonetaryCost -> MonetaryCost -> Int32
41+a `divCost` b = getCostInCents a `quot` getCostInCents b
backend/src/Garnix/Nix/PathInfo.hsnew file+53
@@ -0,0 +1,53 @@
1+module Garnix.Nix.PathInfo where
2+
3+import Control.Lens
4+import Cradle
5+import Data.Aeson ((.:))
6+import Data.Aeson qualified as Aeson
7+import Data.Map.Strict qualified as Map
8+import Data.Text qualified as T
9+import Garnix.Monad
10+import Garnix.Monad.SubProcess (runSubProcess)
11+import Garnix.Nix.Types
12+import Garnix.Nix.Types qualified as Nix
13+import Garnix.NixConfig (addNixConfigEnvironment)
14+import Garnix.Prelude
15+import Garnix.Types
16+
17+-- | Doesn't contain *all* the fields from `nix path-info`, but the ones we needed so far.
18+data PathInfo = PathInfo
19+ { signatures :: [Text],
20+ references :: [StorePath]
21+ }
22+ deriving stock (Generic)
23+
24+instance FromJSON PathInfo where
25+ parseJSON = Aeson.withObject "nix path-info output" $ \o -> do
26+ signatures <- o .: "signatures"
27+ references <- o .: "references"
28+ parsedReferences <- forM references $ \reference -> do
29+ case Nix.parseStorePath (reference :: Text) of
30+ Right ref -> pure ref
31+ Left error -> fail $ cs error
32+ pure
33+ $ PathInfo
34+ { signatures,
35+ references = parsedReferences
36+ }
37+
38+getPathInfo :: StorePath -> M PathInfo
39+getPathInfo storePath = do
40+ nixConfig <- view #userNixConfig
41+ StdoutRaw output <-
42+ runSubProcess $ cmd "nix"
43+ & addArgs
44+ ["path-info", "--json", cs storePath :: Text]
45+ & addNixConfigEnvironment nixConfig
46+ parsed :: Map.Map Text PathInfo <- aesonDecode "nix path-info output" parseJSON (cs output)
47+ case Map.lookup (cs storePath) parsed of
48+ Just pathInfo -> pure pathInfo
49+ Nothing -> throw $ OtherError "Error parsing `nix path-info` output: cannot find store path"
50+
51+signaturesForCacheKey :: PathInfo -> Text -> [Text]
52+signaturesForCacheKey pathInfo cacheKeyName = do
53+ filter ((cacheKeyName <> ":") `T.isPrefixOf`) (pathInfo ^. #signatures)
backend/src/Garnix/Nix/Plan.hsnew file+51
@@ -0,0 +1,51 @@
1+module Garnix.Nix.Plan
2+ ( getPlanOf,
3+ )
4+where
5+
6+import Data.Attoparsec.Text.Lazy hiding (take)
7+import Data.ByteString (ByteString)
8+import Data.Char
9+import Data.Text qualified as T
10+import Data.Text.Lazy.IO (readFile)
11+import Data.Tuple.Extra
12+import Garnix.Monad
13+import Garnix.Nix.Types
14+import Garnix.Prelude hiding (readFile)
15+import Garnix.Types hiding (head)
16+import Nix.Derivation qualified
17+
18+mapToStorePaths :: (ConvertibleStrings a Text) => [a] -> M [StorePath]
19+mapToStorePaths paths = forM paths $ either (throw . OtherError) pure . parseStorePath
20+
21+mapToDrvPaths :: [Text] -> M [DrvPath]
22+mapToDrvPaths paths = map DrvPath <$> mapToStorePaths paths
23+
24+getDrvOutputPaths :: DrvPath -> M [StorePath]
25+getDrvOutputPaths drvFile = do
26+ drvText <- liftIO $ readFile $ cs drvFile
27+ case parse Nix.Derivation.parseDerivation drvText of
28+ Fail _ _ err -> throw $ FailedToParseDrvFile (cs drvFile) $ cs err
29+ Done _ parsed -> mapToStorePaths $ map Nix.Derivation.path $ toList $ Nix.Derivation.outputs parsed
30+
31+getPlanOf :: ByteString -> M Plan
32+getPlanOf = mockable #getBuildPlanMock $ \input -> do
33+ (toBuild, remaining) <-
34+ firstM mapToDrvPaths
35+ . span ("/nix/store/" `T.isPrefixOf`)
36+ . fmap (T.dropWhile isSpace)
37+ . drop 1
38+ . dropWhile (not . isStartOfDerivationPaths)
39+ . T.lines
40+ . cs
41+ $ input
42+ when (not (null remaining) && not ("will be fetched" `T.isInfixOf` head remaining))
43+ $ log Critical "Expected `nix build --dry-run` to output a list of derivations to be built followed by optionally a list of derivations to be fetched"
44+ outputHashes <- forM toBuild $ \drvPath -> (drvPath,) <$> getDrvOutputPaths drvPath
45+ pure $ Plan outputHashes
46+
47+isStartOfDerivationPaths :: Text -> Bool
48+isStartOfDerivationPaths line
49+ | "derivations will be built:" `T.isSuffixOf` line = True
50+ | "derivation will be built:" `T.isSuffixOf` line = True
51+ | otherwise = False
backend/src/Garnix/Nix/StorePath.hsnew file+113
@@ -0,0 +1,113 @@
1+module Garnix.Nix.StorePath
2+ ( withStorePath,
3+ getClosure,
4+
5+ -- * exported for tests
6+ _getOutputs,
7+ _withGcRoot,
8+ )
9+where
10+
11+import Cradle
12+import Data.Map.Strict (Map)
13+import Data.Map.Strict qualified as Map
14+import Data.Row (Rec, type (.==))
15+import Data.Text qualified as T
16+import Garnix.Monad
17+import Garnix.Monad.SubProcess (runSubProcess, runSubProcess_)
18+import Garnix.Nix.Types qualified as Nix
19+import Garnix.NixConfig (addNixConfigEnvironment)
20+import Garnix.Prelude
21+import Garnix.Types
22+import System.Directory (removeFile)
23+import System.IO.Temp (withSystemTempDirectory)
24+
25+type NixDerivationShowOutput =
26+ ( Rec
27+ ( "derivations"
28+ .== Map
29+ Text
30+ ( Rec
31+ ("outputs" .== Map Text (Rec ("path" .== Text)))
32+ )
33+ )
34+ )
35+
36+_getOutputs :: Nix.DrvPath -> M (Map Text Nix.StorePath)
37+_getOutputs drvPath = do
38+ nixConfig <- view #userNixConfig
39+ (StdoutTrimmed output, StderrRaw _, exit) <-
40+ run
41+ $ cmd "nix"
42+ & addArgs ["derivation", "show", cs drvPath :: Text]
43+ & addNixConfigEnvironment nixConfig
44+ case exit of
45+ ExitSuccess -> do
46+ parsed :: NixDerivationShowOutput <- aesonDecode "nix derivation show output" parseJSON output
47+ let raw :: Map Text Text =
48+ fmap (("/nix/store/" <>) . (^. #path))
49+ $ Map.fromList
50+ $ mconcat
51+ $ map (\x -> Map.toList (x ^. #outputs))
52+ $ Map.elems (parsed ^. #derivations)
53+ forM raw $ \storePath -> do
54+ case Nix.parseStorePath storePath of
55+ Right storePath -> pure storePath
56+ Left error -> throw $ OtherError error
57+ ExitFailure _ -> do
58+ throw $ OtherError "Could not get outputs from derivation"
59+
60+_withGcRoot :: Nix.StorePath -> (FilePath -> M a) -> M a
61+_withGcRoot storePath action =
62+ withSystemTempDirectory "garnix-gc" $ \tempDir -> do
63+ let rootPath = tempDir </> cs (Nix.getHash storePath)
64+ bracket (acquire rootPath) release action
65+ where
66+ acquire rootPath = do
67+ workingDir <- view #workingDir
68+ nixConfig <- view #userNixConfig
69+ runSubProcess_
70+ $ cmd "nix-store"
71+ & addArgs ["--add-root", cs rootPath, "--indirect", "--realize", cs storePath :: Text]
72+ & setWorkingDir workingDir
73+ & addNixConfigEnvironment nixConfig
74+ & silenceStdout
75+ pure rootPath
76+ release = liftIO . removeFile . cs
77+
78+withStorePath :: Build -> Text -> (Maybe Nix.StorePath -> M a) -> M a
79+withStorePath build output action = case outputsForBuild build >>= Nix.getOutputByName output of
80+ Just storePath -> _withGcRoot storePath $ const $ action $ Just storePath
81+ Nothing -> do
82+ case (build ^. drvPath :: Maybe FilePath) of
83+ Nothing -> action Nothing
84+ Just drvPath -> do
85+ log Warning $ "Build missing store path for " <> output <> ", but did have drvPath. Falling back to old logic"
86+ drvStorePath <- case Nix.parseStorePath drvPath of
87+ Left error -> do
88+ log Critical error
89+ throw $ OtherError error
90+ Right path -> pure path
91+ _withGcRoot drvStorePath $ \_ -> do
92+ storePathsMap <- _getOutputs $ Nix.DrvPath drvStorePath
93+ case Map.lookup output storePathsMap of
94+ Nothing -> action Nothing
95+ Just storePath -> _withGcRoot storePath $ const $ action $ Just storePath
96+
97+-- | Returns nothing if the passed store path is not valid (for example if passed a store path of a failed build)
98+getClosure :: Nix.StorePath -> M (Maybe [Nix.StorePath])
99+getClosure drvPath = do
100+ result <-
101+ try
102+ $ runSubProcess
103+ $ cmd "nix-store"
104+ & addArgs ["--query", "--requisites", cs drvPath :: Text]
105+ case result of
106+ Right (StdoutRaw output) -> do
107+ paths <- forM (T.lines $ cs output) $ \line -> case Nix.parseStorePath line of
108+ Left _ -> throw $ OtherError $ "cannot parse 'nix-store --query' output: " <> cs output
109+ Right storePath -> pure storePath
110+ pure $ Just paths
111+ Left (ErrorWithContext {err = RunProcessError {stdErr}})
112+ | "error: path '" `T.isPrefixOf` stdErr && "' is not valid\n" `T.isSuffixOf` stdErr -> pure Nothing
113+ Left e -> throwError e
backend/src/Garnix/Nix/Types.hsnew file+117
@@ -0,0 +1,117 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+
3+module Garnix.Nix.Types
4+ ( DrvPath (..),
5+ StoreHash (..),
6+ Plan (..),
7+ planDrvPaths,
8+ planOutputs,
9+ planDrvPathsAndOutputs,
10+ StorePath (..),
11+ parseStorePath,
12+ parseDrvPath,
13+ getStorePath,
14+ getRelativeStorePath,
15+ AppExecPath (..),
16+ BuildOutputs (..),
17+ getOutputByName,
18+ )
19+where
20+
21+import Data.Either.Extra
22+import Data.Hashable (Hashable)
23+import Data.Map qualified as Map
24+import Data.String.Conversions (LBS)
25+import Data.Text qualified as T
26+import Garnix.Prelude
27+import Prelude qualified (show)
28+
29+newtype StoreHash = StoreHash {getStoreHash :: Text}
30+ deriving stock (Eq, Show, Generic, Ord)
31+ deriving newtype (PGParameter "text", PGColumn "text", Hashable)
32+
33+instance ConvertibleStrings StoreHash Text where
34+ convertString :: StoreHash -> Text
35+ convertString = getStoreHash
36+
37+instance ConvertibleStrings StoreHash String where
38+ convertString :: StoreHash -> String
39+ convertString = cs . getStoreHash
40+
41+newtype DrvPath = DrvPath {getDrvPath :: StorePath}
42+ deriving newtype (Eq, Ord)
43+ deriving stock (Generic)
44+
45+instance Show DrvPath where
46+ show (DrvPath storePath) = "DrvPath " <> cs (show (getStorePath storePath))
47+
48+instance ConvertibleStrings DrvPath Text where
49+ convertString :: DrvPath -> Text
50+ convertString = cs . getDrvPath
51+
52+instance ConvertibleStrings DrvPath FilePath where
53+ convertString :: DrvPath -> FilePath
54+ convertString = cs . getDrvPath
55+
56+data StorePath = StorePath {getHash :: StoreHash, getName :: Text}
57+ deriving stock (Eq, Generic, Ord)
58+
59+instance Show StorePath where
60+ show storePath = "StorePath " <> cs (show (getStorePath storePath))
61+
62+instance ConvertibleStrings StorePath Text where
63+ convertString :: StorePath -> Text
64+ convertString = getStorePath
65+
66+instance ConvertibleStrings StorePath String where
67+ convertString :: StorePath -> String
68+ convertString = cs . getStorePath
69+
70+instance ConvertibleStrings StorePath LBS where
71+ convertString :: StorePath -> LBS
72+ convertString = cs . getStorePath
73+
74+getRelativeStorePath :: StorePath -> Text
75+getRelativeStorePath storePath = cs (getHash storePath) <> "-" <> getName storePath
76+
77+getStorePath :: StorePath -> Text
78+getStorePath storePath = "/nix/store/" <> getRelativeStorePath storePath
79+
80+parseStorePath :: (ConvertibleStrings a Text) => a -> Either Text StorePath
81+parseStorePath path = do
82+ let pathT = cs path
83+ handleInvalid = maybeToEither $ pathT <> " is not a valid nix store path"
84+ withoutNixStorePrefix <- handleInvalid $ T.stripPrefix "/nix/store/" pathT
85+ let hash = StoreHash $ T.take 32 withoutNixStorePrefix
86+ name <- handleInvalid (T.stripPrefix (cs hash <> "-") withoutNixStorePrefix)
87+ Right $ StorePath hash name
88+
89+parseDrvPath :: (ConvertibleStrings a Text) => a -> Either Text DrvPath
90+parseDrvPath path = do
91+ storePath <- parseStorePath path
92+ when (not (".drv" `T.isSuffixOf` cs storePath)) $ do
93+ throwError $ "not a drv path: " <> cs path
94+ pure $ DrvPath storePath
95+
96+newtype Plan = Plan [(DrvPath, [StorePath])]
97+ deriving stock (Show, Eq)
98+
99+planDrvPaths :: Plan -> [DrvPath]
100+planDrvPaths (Plan plan) = fmap fst plan
101+
102+planOutputs :: Plan -> [StorePath]
103+planOutputs (Plan plan) = concatMap snd plan
104+
105+planDrvPathsAndOutputs :: Plan -> [StorePath]
106+planDrvPathsAndOutputs plan = fmap getDrvPath (planDrvPaths plan) <> planOutputs plan
107+
108+newtype AppExecPath = AppExecPath {getAppExecPath :: Text}
109+ deriving newtype (Show)
110+
111+newtype BuildOutputs = BuildOutputs {getBuildOutputs :: Map.Map Text StorePath}
112+ deriving stock (Show, Eq)
113+
114+getOutputByName :: Text -> BuildOutputs -> Maybe StorePath
115+getOutputByName outputName (BuildOutputs map) = Map.lookup outputName map
116+
117+makeFields ''Plan
backend/src/Garnix/NixConfig.hsnew file+38
@@ -0,0 +1,38 @@
1+module Garnix.NixConfig
2+ ( defaultNixConfig,
3+ fromNetRcFile,
4+ getNetRcFileSetting,
5+ addNixConfigEnvironment,
6+ nixConfDefaults,
7+ )
8+where
9+
10+import Cradle
11+import Data.Map.Strict qualified as Map
12+import Garnix.Prelude
13+import Garnix.Types (NetRcFile (..), NixConfig (..))
14+
15+defaultNixConfig :: NixConfig
16+defaultNixConfig =
17+ NixConfig $ Map.insert "experimental-features" (unwords ["nix-command", "flakes", "pipe-operators"]) mempty
18+
19+fromNetRcFile :: NetRcFile -> NixConfig
20+fromNetRcFile file = NixConfig $ Map.insert "netrc-file" (getNetRcFile file) mempty
21+
22+getNetRcFileSetting :: NixConfig -> Maybe NetRcFile
23+getNetRcFileSetting (NixConfig m) = NetRcFile <$> Map.lookup "netrc-file" m
24+
25+formatConfig :: NixConfig -> String
26+formatConfig (NixConfig config) =
27+ intercalate "\n" $ map (\(key, value) -> key <> " = " <> value) $ Map.assocs config
28+
29+addNixConfigEnvironment :: NixConfig -> ProcessConfiguration -> ProcessConfiguration
30+addNixConfigEnvironment config =
31+ modifyEnvVar
32+ "NIX_CONFIG"
33+ $ \case
34+ Nothing -> Just $ formatConfig config
35+ Just existing -> Just $ existing <> "\n" <> formatConfig config
36+
37+nixConfDefaults :: ProcessConfiguration -> ProcessConfiguration
38+nixConfDefaults = addArgs ["--extra-experimental-features", "nix-command flakes" :: Text]
backend/src/Garnix/Orchestrator.hsnew file+103
@@ -0,0 +1,103 @@
1+{-# LANGUAGE DuplicateRecordFields #-}
2+
3+module Garnix.Orchestrator
4+ ( handlePullRequest,
5+ handleCommit,
6+ handleRerun,
7+ RerunEvent (..),
8+ )
9+where
10+
11+import Garnix.Async (Promise)
12+import Garnix.Build (buildFlake, rerunBuild)
13+import Garnix.Build.Checkout qualified as Build.Checkout
14+import Garnix.Build.Helpers (withInternalCacheToken)
15+import Garnix.DB qualified as DB
16+import Garnix.Hosting.Deploy (rolloutNewServerVersion)
17+import Garnix.Monad
18+import Garnix.Monad.Async (emptyPromise, resolve, spawn)
19+import Garnix.Prelude
20+import Garnix.Reporters.GithubReporter (mkGithubReporter)
21+import Garnix.Reporters.OpenSearchReporter (openSearchReporter)
22+import Garnix.Types hiding (ghRunId)
23+import Garnix.Types qualified as Types
24+import GitHub.App.Auth qualified as GH
25+
26+data RerunEvent = RerunEvent
27+ { reqUser :: GhLogin,
28+ ghRunId :: GhRunId,
29+ installAuth :: GH.InstallationAuth,
30+ token :: GhToken,
31+ repoIsPublic :: RepoPublicity
32+ }
33+ deriving stock (Generic)
34+
35+handlePullRequest :: (HasCallStack) => Reporter -> CommitInfo -> GhPullRequestId -> M (Promise ())
36+handlePullRequest reporter commitInfo prId = do
37+ assertIsAllowedToBuild (commitInfo ^. repoInfo . ghRepoOwner) (commitInfo ^. repoInfo . ghRepoName)
38+
39+ withSpan commitInfo $ spawn $ do
40+ -- We assume the build is already running if it's NOT a fork.
41+ when (isJust $ commitInfo ^. prFromFork) $ do
42+ buildFlake reporter commitInfo >>= resolve
43+
44+ -- If it's not a fork, we're already attempting to deploy in `buildFlake`.
45+ when (isNothing $ commitInfo ^. prFromFork) $ do
46+ deployPrServers prId
47+ where
48+ deployPrServers :: GhPullRequestId -> M ()
49+ deployPrServers prId = do
50+ Build.Checkout.withCheckout commitInfo $ withSpan prId $ do
51+ withInternalCacheToken (commitInfo ^. Types.reqUser) $ do
52+ void (rolloutNewServerVersion reporter commitInfo $ GhPrDeployment prId)
53+
54+handleCommit :: (HasCallStack) => Reporter -> Bool -> CommitInfo -> M (Promise ())
55+handleCommit reporter allowDuplicateRun commitInfo = do
56+ withSpan commitInfo $ do
57+ assertIsAllowedToBuild (commitInfo ^. repoInfo . ghRepoOwner) (commitInfo ^. repoInfo . ghRepoName)
58+ pushResult <- case commitInfo ^. branch of
59+ Nothing -> do
60+ log Informational "handleCommit: CommitInfo is missing branch. Not registering push"
61+ pure Nothing
62+ Just branch -> do
63+ Just
64+ <$> DB.registerPush
65+ (commitInfo ^. repoInfo . ghRepoOwner)
66+ (commitInfo ^. repoInfo . ghRepoName)
67+ (commitInfo ^. commit)
68+ branch
69+ case (allowDuplicateRun, pushResult) of
70+ (False, Just DB.AlreadyPushed) -> do
71+ log Informational "handleCommit: This repoOwner, repoName, commit, branch combination has already been pushed before. Skipping build"
72+ emptyPromise
73+ (False, Nothing) -> do
74+ log Informational "handleCommit: CommitInfo is missing branch, but allowDuplicateRun is set. Skipping build"
75+ emptyPromise
76+ (False, Just DB.NewPush) -> do
77+ buildFlake reporter commitInfo <?> "Build flake"
78+ (True, _) -> do
79+ buildFlake reporter commitInfo <?> "Build flake"
80+
81+handleRerun :: (HasCallStack) => RerunEvent -> M ()
82+handleRerun ev = do
83+ hostname <- view #hostname
84+ build' <- DB.makeNewBuildForGithubRunId (ev ^. #reqUser) (ev ^. #ghRunId) hostname
85+ withSpan (build' ^. id) $ do
86+ let commitInfo =
87+ CommitInfo
88+ { _commitInfoReqUser = ev ^. #reqUser,
89+ _commitInfoRepoPublicity = ev ^. #repoIsPublic,
90+ _commitInfoRepoInfo = RepoInfo (ev ^. #installAuth) (ev ^. #token) (build' ^. repoUser) (build' ^. repoName),
91+ _commitInfoBranch = build' ^. branch,
92+ _commitInfoPrFromFork = build' ^. prFromFork,
93+ _commitInfoCommit = build' ^. gitCommit
94+ }
95+ let reporter = openSearchReporter <> mkGithubReporter (commitInfo ^. repoInfo) (commitInfo ^. commit)
96+ assertIsAllowedToBuild (build' ^. repoUser) (build' ^. repoName)
97+ withSpan commitInfo $ rerunBuild reporter build' commitInfo
98+
99+assertIsAllowedToBuild :: GhRepoOwner -> GhRepoName -> M ()
100+assertIsAllowedToBuild owner repo = do
101+ isDenied <- DB.isDenylisted owner repo
102+ when isDenied $ do
103+ throw IsDeniedAccess
backend/src/Garnix/ParseHttpBasicAuth.hsnew file+15
@@ -0,0 +1,15 @@
1+module Garnix.ParseHttpBasicAuth where
2+
3+import Data.ByteString.Base64
4+import Data.Text qualified as T
5+import Garnix.Prelude
6+
7+parseBasicAuth :: Text -> Either String (Text, Text)
8+parseBasicAuth authHeader = do
9+ b64 <- case T.stripPrefix "Basic " authHeader of
10+ Just x -> Right x
11+ Nothing -> Left "Auth request missing `Basic ` prefix"
12+ credentials <- decode $ cs b64
13+ case T.splitOn ":" $ cs credentials of
14+ user : password | not (null password) -> Right (user, T.intercalate ":" password)
15+ _ -> Left "No `:` in decoded payload"
backend/src/Garnix/Password.hsnew file+23
@@ -0,0 +1,23 @@
1+module Garnix.Password
2+ ( HashedPassword,
3+ hashPassword,
4+ validatePassword,
5+ )
6+where
7+
8+import Data.ByteString (ByteString)
9+import Garnix.Prelude
10+import "crypton" Crypto.KDF.BCrypt qualified
11+
12+newtype HashedPassword = HashedPassword ByteString
13+ deriving stock (Eq, Show, Generic)
14+ deriving newtype (PGParameter "text", PGColumn "text")
15+
16+hashPassword :: (MonadIO m) => Text -> m HashedPassword
17+hashPassword plainTextPassword = do
18+ hashed :: ByteString <- liftIO $ Crypto.KDF.BCrypt.hashPassword 8 (cs plainTextPassword :: ByteString)
19+ pure $ HashedPassword hashed
20+
21+validatePassword :: HashedPassword -> Text -> Bool
22+validatePassword (HashedPassword hashedPassword) plainTextPassword =
23+ Crypto.KDF.BCrypt.validatePassword (cs plainTextPassword :: ByteString) hashedPassword
backend/src/Garnix/Prelude.hsnew file+431
@@ -0,0 +1,431 @@
1+{-# OPTIONS_GHC -fno-warn-orphans #-}
2+
3+module Garnix.Prelude
4+ ( module M,
5+ identity,
6+ curry3,
7+ curry4,
8+ curry5,
9+ curry6,
10+ curry7,
11+ location,
12+ putText,
13+ putTextLn,
14+ show,
15+ showPretty,
16+ pShow,
17+ pPrint,
18+ dbg,
19+ dbgM,
20+ dbgM_,
21+ error,
22+ slidingWindow',
23+ try,
24+ whenError,
25+ withError,
26+ whenIs,
27+ whenM,
28+ unlessM,
29+ ourToJSON,
30+ ourToEncoding,
31+ ourParseJSON,
32+ HashId,
33+ hashIdText,
34+ hashIdInt,
35+ getHashId,
36+ isValidSubdomainString,
37+ xdgCacheHome,
38+ randomElement,
39+ randomBase64,
40+ uniq,
41+ )
42+where
43+
44+import Control.Applicative as M
45+import Control.Concurrent.Async.Lifted as M (forConcurrently, forConcurrently_)
46+import Control.Concurrent.Lifted as M (ThreadId, fork, killThread)
47+import Control.Exception.Safe as M (catch, catchAny)
48+import Control.Lens (_head)
49+import Control.Lens as M
50+ ( Iso,
51+ Iso',
52+ Lens,
53+ Lens',
54+ Prism,
55+ Prism',
56+ at,
57+ coerced,
58+ contramap,
59+ each,
60+ filtered,
61+ from,
62+ iso,
63+ lens,
64+ makeFields,
65+ makePrisms,
66+ mapped,
67+ prism,
68+ re,
69+ review,
70+ to,
71+ view,
72+ views,
73+ (%~),
74+ (&),
75+ (.~),
76+ (?~),
77+ (^.),
78+ (^..),
79+ (^?),
80+ _1,
81+ _2,
82+ _3,
83+ _Just,
84+ _Left,
85+ _Right,
86+ )
87+import Control.Monad as M (filterM, foldM, forever, guard, join, replicateM, replicateM_, unless, when, (<=<), (>=>))
88+import Control.Monad.Catch as M hiding (catch, try)
89+import Control.Monad.Except as M hiding (withError)
90+import Control.Monad.Reader as M
91+import Control.Monad.State as M
92+import Control.Monad.Trans.Control as M (MonadBaseControl, liftBaseOp)
93+import Control.Monad.Trans.Resource as M (ResourceT, runResourceT)
94+import Data.Aeson as M (FromJSON (..), FromJSONKey (..), ToJSON (..), ToJSONKey (..))
95+import Data.Aeson qualified as Aeson
96+import Data.Aeson.QQ as M (aesonQQ)
97+import Data.Aeson.Types qualified as Aeson
98+import Data.Bifunctor as M (Bifunctor, bimap, first, second)
99+import Data.Char (toLower)
100+import Data.Coerce as M (coerce)
101+import Data.Default.Class as M
102+import Data.Either as M (isLeft, isRight)
103+import Data.Foldable as M
104+import Data.Function as M (on)
105+import Data.Functor as M (void, ($>))
106+import Data.Generics.Labels ()
107+import Data.Int as M (Int16, Int32, Int64)
108+import Data.List as M hiding (delete)
109+import Data.List.NonEmpty as M (NonEmpty ((:|)))
110+import Data.Maybe as M (catMaybes, fromMaybe, isJust, isNothing)
111+import Data.Proxy as M (Proxy (Proxy))
112+import Data.Row.Aeson ()
113+import Data.Sequence as M (Seq)
114+import Data.Set as M (Set)
115+import Data.String.Conversions as M (ConvertibleStrings (convertString), LazyByteString, LazyText, StrictByteString, StrictText, cs)
116+import Data.Text as M (Text)
117+import Data.Text qualified as T
118+import Data.Text.IO qualified as T
119+import Data.Time.Clock as M (NominalDiffTime, UTCTime, addUTCTime, diffUTCTime, getCurrentTime, nominalDiffTimeToSeconds, secondsToNominalDiffTime)
120+import Data.Traversable as M (forM)
121+import Data.Tuple as M (swap)
122+import Data.Typeable as M (Typeable)
123+import Data.Vector as M (Vector)
124+import Database.PostgreSQL.Typed as M (PGConnection)
125+import Database.PostgreSQL.Typed.HDBC as M (Connection, withPGConnection)
126+import Database.PostgreSQL.Typed.Types as M (PGColumn (..), PGParameter (..), PGType (..), PGTypeID (..))
127+import Debug.Trace (trace)
128+import GHC.Exts as M (Constraint, IsString (..))
129+import GHC.Generics as M (Generic, Rep)
130+import GHC.OverloadedLabels as M (IsLabel (..))
131+import GHC.Stack as M (CallStack, HasCallStack, SrcLoc, callStack, getCallStack, prettyCallStack)
132+import GHC.TypeLits as M (Symbol)
133+import Generics.Eot (HasEot, datatype, datatypeName)
134+import Iso.Deriving as M (As (..), As1 (..), Inject (..), Isomorphic, Project (..))
135+import Network.Wai (Application)
136+import Prettyprinter as M (Pretty (..), pretty, vsep, (<+>))
137+import Prettyprinter as Pretty
138+import Prettyprinter.Render.Text as Pretty
139+import Servant as M
140+ ( Capture,
141+ Delete,
142+ FromHttpApiData (..),
143+ Get,
144+ Header,
145+ Headers,
146+ JSON,
147+ NoContent (..),
148+ Patch,
149+ Post,
150+ QueryParam,
151+ ReqBody,
152+ ToHttpApiData (..),
153+ addHeader,
154+ (:>),
155+ )
156+import Servant qualified
157+import Servant.API.Generic as M ((:-))
158+import Servant.Auth.Server as M (FromJWT, ToJWT)
159+import Servant.Auth.Server.Internal.AddSetCookie (AddSetCookieApi, AddSetCookies (..), Nat (S))
160+import Servant.RawM (RawM')
161+import Servant.Server.Generic as M (AsServerT)
162+import Streaming as M (Of, Stream)
163+import Streaming.ByteString.Char8 as M (ByteStream)
164+import Streaming.Prelude qualified as S
165+import System.Environment as M (lookupEnv)
166+import System.FilePath.Posix as M (takeDirectory, (</>))
167+import System.IO as M (BufferMode (..), Handle, hSetBuffering, stderr, stdout)
168+import System.Random (randomRIO)
169+import Text.Show.Pretty qualified
170+import Web.Hashids qualified as Hashids
171+import "base64-bytestring" Data.ByteString.Base64 qualified as Base64
172+import "crypton" Crypto.Random.Entropy qualified as Random
173+import Prelude as M hiding (error, id, log, show)
174+import Prelude qualified
175+
176+identity :: a -> a
177+identity x = x
178+
179+curry3 :: ((a, b, c) -> output) -> a -> b -> c -> output
180+curry3 fun a b c = fun (a, b, c)
181+
182+curry4 :: ((a, b, c, d) -> output) -> a -> b -> c -> d -> output
183+curry4 fun a b c d = fun (a, b, c, d)
184+
185+curry5 :: ((a, b, c, d, e) -> output) -> a -> b -> c -> d -> e -> output
186+curry5 fun a b c d e = fun (a, b, c, d, e)
187+
188+curry6 :: ((a, b, c, d, e, f) -> output) -> a -> b -> c -> d -> e -> f -> output
189+curry6 fun a b c d e f = fun (a, b, c, d, e, f)
190+
191+curry7 :: ((a, b, c, d, e, f, g) -> output) -> a -> b -> c -> d -> e -> f -> g -> output
192+curry7 fun a b c d e f g = fun (a, b, c, d, e, f, g)
193+
194+location :: (HasCallStack) => SrcLoc
195+location =
196+ case getCallStack callStack of
197+ (_, srcLoc) : _ -> srcLoc
198+ _ -> error "impossible"
199+
200+putText :: (MonadIO m) => Text -> m ()
201+putText = liftIO . T.putStr
202+
203+putTextLn :: (MonadIO m) => Text -> m ()
204+putTextLn = liftIO . T.putStrLn
205+
206+show :: (Show a) => a -> Text
207+show = cs . Prelude.show
208+
209+showPretty :: (Pretty a) => a -> Text
210+showPretty = Pretty.renderStrict . Pretty.layoutPretty Pretty.defaultLayoutOptions . pretty
211+
212+pShow :: (Show a) => a -> Text
213+pShow = cs . Text.Show.Pretty.ppShow
214+
215+pPrint :: (Show a, MonadIO m) => a -> m ()
216+pPrint = liftIO . Text.Show.Pretty.pPrint
217+
218+dbg :: (Show a) => a -> a
219+dbg a = trace ("dbg: " <> Text.Show.Pretty.ppShow a) a
220+
221+dbgM :: (MonadIO m, Show a) => a -> m a
222+dbgM a = dbgM_ a $> a
223+
224+dbgM_ :: (MonadIO m, Show a) => a -> m ()
225+dbgM_ a = liftIO $ T.hPutStrLn stderr ("dbgM: " <> cs (Text.Show.Pretty.ppShow a))
226+
227+error :: (HasCallStack) => Text -> a
228+error = Prelude.error . cs
229+
230+try :: (MonadError e m) => m a -> m (Either e a)
231+try a = (Right <$> a) `catchError` (pure . Left)
232+
233+whenError :: (MonadError e m) => m a -> (e -> m ()) -> m a
234+whenError action note = action `catchError` (\e -> note e >> throwError e)
235+
236+withError :: (MonadError e m) => (e -> e) -> m a -> m a
237+withError f action = try action >>= either (throwError . f) pure
238+
239+-- | Similar to `when`, but takes a value, and a prism into the value, instead.
240+whenIs :: (Applicative m) => Prism' a b -> a -> (b -> m ()) -> m ()
241+whenIs prism_ val act = case val ^? prism_ of
242+ Nothing -> pure ()
243+ Just v -> act v
244+
245+-- | Monadic version of @when@
246+whenM :: (Monad m) => m Bool -> m () -> m ()
247+whenM mb thing = mb >>= \b -> when b thing
248+
249+-- | Monadic version of @unless@
250+unlessM :: (Monad m) => m Bool -> m () -> m ()
251+unlessM condM acc = condM >>= \cond -> unless cond acc
252+
253+-- * HashIds
254+
255+hashIdContext :: Hashids.HashidsContext
256+hashIdContext = Hashids.hashidsMinimum "the smell of petroleum pervades throughout" 8
257+
258+-- | A hashid is an opaque ID generated from an int.
259+--
260+-- This should only be constructed from the DB, so we don't export the
261+-- constructor.
262+newtype HashId = HashId {getHashId :: Text}
263+ deriving stock (Eq, Show)
264+ deriving newtype (ToJSON, FromJSON, Pretty, ToHttpApiData)
265+
266+instance Servant.FromHttpApiData HashId where
267+ parseUrlPiece piece = case Hashids.decode hashIdContext (cs piece) of
268+ [_] -> Right $ HashId piece
269+ _ -> Left $ "Invalid hash id: " <> piece
270+
271+instance PGParameter "bigint" HashId where
272+ pgEncode t (HashId bs) = case Hashids.decode hashIdContext (cs bs) of
273+ [i] -> pgEncode t (fromIntegral i :: Int64)
274+ err -> error $ "expected single integer: " <> show err
275+
276+instance PGColumn "bigint" HashId where
277+ pgDecode t i =
278+ HashId $ cs $ Hashids.encode hashIdContext (fromIntegral (pgDecode t i :: Int64))
279+
280+hashIdText :: Prism' Text HashId
281+hashIdText = prism there back
282+ where
283+ there = getHashId
284+ back t = case Hashids.decode hashIdContext (cs t) of
285+ [_] -> pure $ HashId t
286+ _ -> Left "Expected a single integer"
287+
288+hashIdInt :: Iso' HashId Int
289+hashIdInt = iso hashIdToInt intToHashId
290+ where
291+ intToHashId int = HashId $ cs $ Hashids.encode hashIdContext int
292+ hashIdToInt hashId = case Hashids.decode hashIdContext $ cs $ getHashId hashId of
293+ [int] -> int
294+ _ -> error "HashId should always decode into single element integers"
295+
296+-- * Aeson helpers
297+
298+ourOpts :: forall a. (HasEot a) => Proxy a -> Aeson.Options
299+ourOpts p =
300+ Aeson.defaultOptions
301+ { Aeson.fieldLabelModifier = \field ->
302+ let prefix = ("_" <> (datatypeName (datatype p) & _head %~ toLower))
303+ in case stripPrefix prefix field of
304+ Just stripped -> Aeson.camelTo2 '_' stripped
305+ Nothing -> error $ "ourOpts: cannot strip " <> cs prefix <> " from " <> cs field,
306+ Aeson.omitNothingFields = True
307+ }
308+
309+ourToEncoding ::
310+ forall a.
311+ (Generic a, Aeson.GToJSON' Aeson.Encoding Aeson.Zero (Rep a), HasEot a) =>
312+ a ->
313+ Aeson.Encoding
314+ourToEncoding = Aeson.genericToEncoding (ourOpts p)
315+ where
316+ p :: Proxy a
317+ p = Proxy
318+
319+ourToJSON ::
320+ forall a.
321+ (Generic a, Aeson.GToJSON' Aeson.Value Aeson.Zero (Rep a), HasEot a) =>
322+ a ->
323+ Aeson.Value
324+ourToJSON = Aeson.genericToJSON (ourOpts p)
325+ where
326+ p :: Proxy a
327+ p = Proxy
328+
329+ourParseJSON ::
330+ forall a.
331+ (Generic a, Aeson.GFromJSON Aeson.Zero (Rep a), HasEot a) =>
332+ Aeson.Value ->
333+ Aeson.Parser a
334+ourParseJSON = Aeson.genericParseJSON (ourOpts p)
335+ where
336+ p :: Proxy a
337+ p = Proxy
338+
339+-- * Streaming
340+
341+-- | A sliding window like streaming's slidingWindow, but doesn't wait to min
342+-- value to start yielding.
343+slidingWindow' ::
344+ forall a b m.
345+ (Monad m) =>
346+ Int ->
347+ Stream (Of [a]) m b ->
348+ Stream (Of [a]) m b
349+slidingWindow' n = window mempty
350+ where
351+ lastN :: [a] -> (Int, [a])
352+ lastN a =
353+ let l = length a
354+ in if l > n then (n, drop (l - n) a) else (l, a)
355+
356+ window :: (Monad m) => [a] -> Stream (Of [a]) m b -> Stream (Of [a]) m b
357+ window !sequ str = do
358+ e <- lift (S.next str)
359+ case e of
360+ Left r -> return r
361+ Right (a, rest) -> do
362+ let (_, a') = lastN (sequ <> a)
363+ S.yield a'
364+ window a' rest
365+
366+instance (Isomorphic a b, IsString a) => IsString (As a b) where
367+ fromString x = As $ inj (fromString x :: a)
368+
369+instance (Isomorphic a b, Aeson.FromJSON a) => Aeson.FromJSON (As a b) where
370+ parseJSON x = fmap (As . inj) (Aeson.parseJSON x :: Aeson.Parser a)
371+
372+instance (Isomorphic a b, Aeson.ToJSON a) => Aeson.ToJSON (As a b) where
373+ toJSON (As x) = Aeson.toJSON (prj x :: a)
374+
375+instance (Isomorphic a b, FromHttpApiData a) => FromHttpApiData (As a b) where
376+ parseUrlPiece x = fmap (As . inj) (parseUrlPiece x :: Either Text a)
377+
378+instance (Isomorphic a b, ToHttpApiData a) => ToHttpApiData (As a b) where
379+ toUrlPiece (As x) = toUrlPiece (prj x :: a)
380+
381+instance (Isomorphic a b, Aeson.ToJSONKey a) => Aeson.ToJSONKey (As a b) where
382+ toJSONKey = contramap (\(As b) -> prj b) (Aeson.toJSONKey :: Aeson.ToJSONKeyFunction a)
383+ toJSONKeyList = contramap (fmap (\(As b) -> prj b)) (Aeson.toJSONKeyList :: Aeson.ToJSONKeyFunction [a])
384+
385+instance (Isomorphic a b, Aeson.FromJSONKey a) => Aeson.FromJSONKey (As a b) where
386+ fromJSONKey = fmap (As . inj) (Aeson.fromJSONKey :: Aeson.FromJSONKeyFunction a)
387+ fromJSONKeyList = fmap (fmap (As . inj)) (Aeson.fromJSONKeyList :: Aeson.FromJSONKeyFunction [a])
388+
389+instance (Isomorphic a b, PGParameter t a) => PGParameter t (As a b) where
390+ pgEncode i (As b) = pgEncode i (prj b :: a)
391+
392+instance (Isomorphic a b, PGColumn t a) => PGColumn t (As a b) where
393+ pgDecode i x = As $ inj (pgDecode i x :: a)
394+
395+-- Type instance for RawM for SetCookie API
396+type instance AddSetCookieApi (RawM' a) = RawM' a
397+
398+-- Provide an @AddSetCookies@ instance for functors of Application.
399+instance
400+ (Functor m) =>
401+ AddSetCookies ('S n) (m Application) (m Application)
402+ where
403+ addSetCookies cookies = fmap $ addSetCookies cookies
404+
405+isValidSubdomainString :: Text -> Bool
406+isValidSubdomainString x =
407+ T.all (`elem` allowed) x
408+ && not ("-" `T.isPrefixOf` x)
409+ && not ("-" `T.isSuffixOf` x)
410+ where
411+ allowed = ['A' .. 'Z'] ++ ['0' .. '9'] ++ ['a' .. 'z'] ++ ['-']
412+
413+xdgCacheHome :: String
414+xdgCacheHome = "XDG_CACHE_HOME"
415+
416+randomElement :: (MonadIO m) => [a] -> m a
417+randomElement list = do
418+ index <- randomRIO (0, length list - 1)
419+ pure $ list !! index
420+
421+randomBase64 :: (MonadIO m) => Int -> m Text
422+randomBase64 size = cs . Base64.encode <$> liftIO (Random.getEntropy size)
423+
424+uniq :: (Eq a) => [a] -> [a]
425+uniq = \case
426+ [] -> []
427+ [a] -> [a]
428+ a : b : rest ->
429+ if a == b
430+ then uniq (b : rest)
431+ else a : uniq (b : rest)
backend/src/Garnix/Reporters/GithubReporter.hsnew file+97
@@ -0,0 +1,97 @@
1+module Garnix.Reporters.GithubReporter (mkGithubReporter) where
2+
3+import Control.Concurrent.Lifted (modifyMVar_, newMVar, readMVar)
4+import Control.Debounce
5+import Data.Text qualified as T
6+import Garnix.BuildLogs.Types (LogLine (..))
7+import Garnix.Duration
8+import Garnix.Monad
9+import Garnix.Prelude
10+import Garnix.Types
11+import Network.URI (URI, parseRelativeReference)
12+
13+mkGithubReporter :: RepoInfo -> CommitHash -> Reporter
14+mkGithubReporter repoInfo commit =
15+ Reporter
16+ { createNewRun = \reportType -> do
17+ let name = reportName reportType
18+ url <- getRelativeUrl reportType
19+ let initialReport = mkReport name url commit "" RunReportStatusInProgress
20+ ghRunId <- newBuildReport repoInfo initialReport
21+ logsMVar <- newMVar (RunReportStatusInProgress, Nothing)
22+ lastSentLogsMVar <- newMVar Nothing
23+ let addNewline t = if "\n" `T.isSuffixOf` t then t else t <> "\n"
24+ let appendLogs status logs = do
25+ modifyMVar_ logsMVar $ \(curStatus, curLogs) -> do
26+ pure (fromMaybe curStatus status, Just $ mconcat $ map addNewline $ catMaybes [curLogs, logs])
27+ let sendLogs = do
28+ (status, logs) <- readMVar logsMVar
29+ lastSent <- readMVar lastSentLogsMVar
30+ when (lastSent /= Just (status, logs)) $ do
31+ modifyMVar_ lastSentLogsMVar $ const $ pure $ Just (status, logs)
32+ let report = mkReport name url commit (fromMaybe "" logs) status
33+ void $ ignoringAllErrors $ updateBuildReport ghRunId report repoInfo
34+ debouncedSendLogs <- do
35+ debounceDuration <- view #githubLogDebounceDuration
36+ if debounceDuration == emptyDuration
37+ then pure sendLogs
38+ else do
39+ env <- ask
40+ liftIO
41+ <$> liftIO
42+ ( mkDebounce
43+ defaultDebounceSettings
44+ { debounceAction = void $ runM env sendLogs,
45+ debounceFreq = toMicroseconds debounceDuration,
46+ debounceEdge = trailingEdge
47+ }
48+ )
49+ pure
50+ $ RunReporter
51+ { reportLogs = \(LogLine package _phase log) -> do
52+ appendLogs Nothing $ Just $ prefixLogLineWithPackageName package log
53+ debouncedSendLogs,
54+ reportComplete = \status -> do
55+ appendLogs (Just status) Nothing
56+ sendLogs,
57+ ghRunId = Just ghRunId
58+ }
59+ }
60+
61+prefixLogLineWithPackageName :: Maybe PackageName -> Text -> Text
62+prefixLogLineWithPackageName mPkgName logLine = prefix <> logLine
63+ where
64+ prefix = maybe "" ((<> "> ") . getPackageName) mPkgName
65+
66+getRelativeUrl :: ReportType -> M (Maybe URI)
67+getRelativeUrl buildOrRun = do
68+ let path = case buildOrRun of
69+ ReportBuild _name build -> Just $ "/build/" <> build ^. id . to getBuildId . re hashIdText
70+ ReportRun run -> Just $ "/run/" <> run ^. id . to getRunId . re hashIdText
71+ MetaCheck -> Nothing
72+ case path of
73+ Nothing -> pure Nothing
74+ Just path -> do
75+ case parseRelativeReference (cs path) of
76+ Just uri -> pure $ Just uri
77+ Nothing -> throw $ OtherError $ "Failed to parse build URI from " <> path
78+
79+mkReport :: Text -> Maybe URI -> CommitHash -> Text -> RunReportStatus -> GhRunReport
80+mkReport name url commit logs reportStatus =
81+ GhRunReport
82+ { _ghRunReportName = name,
83+ _ghRunReportCommit = commit,
84+ _ghRunReportUrl = show <$> url,
85+ _ghRunReportStatus = reportStatus,
86+ _ghRunReportTitle = name,
87+ _ghRunReportSummary = getReportSummary $ checkRunSummary name reportStatus,
88+ _ghRunReportLogs = RawLogs logs
89+ }
90+
91+checkRunSummary :: Text -> RunReportStatus -> ReportSummary
92+checkRunSummary runName = \case
93+ RunReportStatusInProgress -> ReportSummary runName
94+ RunReportStatusSuccess -> ReportSummary $ runName <> " succeeded"
95+ RunReportStatusFailure -> ReportSummary $ runName <> " failed"
96+ RunReportStatusTimeout -> ReportSummary $ runName <> " timed out"
97+ RunReportStatusCancelled -> ReportSummary $ runName <> " cancelled"
backend/src/Garnix/Reporters/OpenSearchReporter.hsnew file+45
@@ -0,0 +1,45 @@
1+module Garnix.Reporters.OpenSearchReporter
2+ ( openSearchReporter,
3+ )
4+where
5+
6+import Garnix.DB qualified as DB
7+import Garnix.Monad
8+import Garnix.Prelude
9+import Garnix.Types
10+import Garnix.UserLogs
11+
12+openSearchReporter :: Reporter
13+openSearchReporter =
14+ Reporter
15+ { createNewRun = \reportType -> do
16+ pure $ case reportType of
17+ MetaCheck -> do
18+ RunReporter
19+ { reportLogs = \_ -> pure (),
20+ reportComplete = \_ -> pure (),
21+ ghRunId = Nothing
22+ }
23+ ReportRun run -> do
24+ RunReporter
25+ { reportLogs = \logLine -> do
26+ storeRunLogLine run logLine,
27+ reportComplete = \runReportStatus -> do
28+ let status = case runReportStatus of
29+ RunReportStatusSuccess -> Just Success
30+ RunReportStatusInProgress -> Nothing
31+ RunReportStatusFailure -> Just Failure
32+ RunReportStatusTimeout -> Just Timeout
33+ RunReportStatusCancelled -> Just Cancelled
34+ DB.setRunStatus (run ^. id) status,
35+ ghRunId = Nothing
36+ }
37+ ReportBuild _name build -> do
38+ RunReporter
39+ { reportLogs = \logLine -> do
40+ storeBuildLogLine build logLine,
41+ reportComplete = \_runReportStatus -> do
42+ pure (),
43+ ghRunId = Nothing
44+ }
45+ }
backend/src/Garnix/Reporters/Utils.hsnew file+60
@@ -0,0 +1,60 @@
1+module Garnix.Reporters.Utils
2+ ( withRunReporter,
3+ runWithRunReporter,
4+ runWithRunReporterNoStdout,
5+ runWithRunReporter_,
6+ )
7+where
8+
9+import Control.Concurrent.Async.Lifted qualified as Async
10+import Cradle qualified
11+import Cradle.ProcessConfiguration (ProcessConfiguration (..), addHandle)
12+import Data.Function (applyWhen)
13+import Garnix.BuildLogs.Types (mkLogLine)
14+import Garnix.Monad
15+import Garnix.Prelude
16+import Garnix.SafeUnix (safeCreatePipe)
17+import Garnix.Types
18+import Streaming.ByteString.Char8 qualified as S
19+import Streaming.Prelude qualified as S hiding (fromHandle)
20+import System.IO qualified
21+
22+withRunReporter :: Reporter -> ReportType -> (RunReporter -> M a) -> M a
23+withRunReporter report reportType action = do
24+ runReporter <- createNewRun report reportType
25+ catchEither (action runReporter) $ \e -> do
26+ let message = case e of
27+ Right e -> show $ pretty $ err e
28+ Left e -> show e
29+ reportLogs runReporter (mkLogLine message)
30+ reportComplete runReporter RunReportStatusFailure
31+ rethrowEither e
32+
33+-- | Sends both `stdout` and `stderr` to the given `RunReporter`. Also disables
34+-- sending the `stdout` and `stderr` to the parent's output streams.
35+runWithRunReporter :: (Cradle.Output o) => RunReporter -> Cradle.ProcessConfiguration -> M o
36+runWithRunReporter = runWithRunReporter' True
37+
38+runWithRunReporterNoStdout :: (Cradle.Output o) => RunReporter -> Cradle.ProcessConfiguration -> M o
39+runWithRunReporterNoStdout = runWithRunReporter' False
40+
41+runWithRunReporter' :: (Cradle.Output o) => Bool -> RunReporter -> Cradle.ProcessConfiguration -> M o
42+runWithRunReporter' sendStdout runReporter config = do
43+ (readHandle, writeHandle) <- liftIO safeCreatePipe
44+ let pipeThread = do
45+ let lines = S.mapped S.toLazy $ S.lines $ S.fromHandle readHandle
46+ flip S.mapM_ lines $ \line -> do
47+ reportLogs runReporter (mkLogLine $ cs line)
48+ Async.withAsync pipeThread $ \pipeThread -> do
49+ output <-
50+ Cradle.run
51+ $ applyWhen
52+ sendStdout
53+ (\c -> c {stdoutConfig = Cradle.ProcessConfiguration.addHandle writeHandle (stdoutConfig config)})
54+ $ config {stderrConfig = Cradle.ProcessConfiguration.addHandle writeHandle (stderrConfig config)}
55+ liftIO $ System.IO.hClose writeHandle
56+ Async.wait pipeThread
57+ pure output
58+
59+runWithRunReporter_ :: RunReporter -> Cradle.ProcessConfiguration -> M ()
60+runWithRunReporter_ runReporter cmd = runWithRunReporter runReporter cmd
backend/src/Garnix/Request.hsnew file+46
@@ -0,0 +1,46 @@
1+module Garnix.Request
2+ ( retryingFor,
3+ retryingWithPolicy,
4+ retrySequence,
5+ )
6+where
7+
8+import Control.Retry
9+import Data.Maybe (listToMaybe)
10+import Garnix.Duration
11+import Garnix.Monad
12+import Garnix.Prelude
13+import Garnix.Types
14+
15+-- | Retrying with reasonable defaults. Exponential backoff, with a
16+-- *cumulative* cap of the given seconds.
17+retryingFor :: Duration -> M a -> M a
18+retryingFor secs action =
19+ retryingWithPolicy policy action
20+ where
21+ policy :: RetryPolicy
22+ policy =
23+ limitRetriesByCumulativeDelay (toMicroseconds secs)
24+ $ exponentialBackoff 100
25+
26+retryingWithPolicy :: RetryPolicyM M -> M a -> M a
27+retryingWithPolicy policy action =
28+ retryOnError
29+ policy
30+ (\_ _ -> pure True)
31+ (\_ -> action `catchAny` (throw . OtherError . show))
32+
33+-- | Try all actions in sequence.
34+retrySequence :: forall a. [M a] -> M a
35+retrySequence actions = do
36+ retryOnError policy (\_ _ -> pure True) go
37+ where
38+ policy :: RetryPolicy
39+ policy = limitRetries $ length actions - 1
40+
41+ go :: RetryStatus -> M a
42+ go RetryStatus {..} =
43+ fromMaybe
44+ (throw $ OtherError "retries policy failed")
45+ . listToMaybe
46+ $ drop rsIterNumber actions
backend/src/Garnix/S3Cache.hsnew file+217
@@ -0,0 +1,217 @@
1+module Garnix.S3Cache (upload, toNarFilePath) where
2+
3+import Amazonka qualified
4+import Amazonka.S3 qualified as Amazonka
5+import Control.Lens
6+import Control.Retry (RetryPolicyM, fullJitterBackoff, limitRetries, limitRetriesByCumulativeDelay)
7+import Cradle
8+import Data.ByteString.Builder qualified as ByteString
9+import Data.Containers.ListUtils (nubOrd)
10+import Data.Text qualified as T
11+import Garnix.API.Cache.Types
12+import Garnix.Build.Types (EvaluationResult)
13+import Garnix.BuildLogs.Types hiding (log)
14+import Garnix.DB qualified as DB
15+import Garnix.Duration
16+import Garnix.Monad
17+import Garnix.Monad.Memoization (memoize)
18+import Garnix.Monad.Metrics (incrementEvent, timingAs)
19+import Garnix.Monad.Pool
20+import Garnix.Monad.SubProcess (runSubProcess, runSubProcess_)
21+import Garnix.Nix.PathInfo (getPathInfo)
22+import Garnix.Nix.PathInfo qualified as Nix
23+import Garnix.Nix.StorePath qualified as Nix
24+import Garnix.Nix.Types
25+import Garnix.Nix.Types qualified as Nix
26+import Garnix.NixConfig (addNixConfigEnvironment)
27+import Garnix.Prelude
28+import Garnix.Request (retryingWithPolicy)
29+import Garnix.Types
30+import System.Directory (doesDirectoryExist, doesFileExist, getFileSize, listDirectory, pathIsSymbolicLink)
31+import System.IO (withBinaryFile)
32+import System.IO qualified as IO
33+import System.IO.Temp (withSystemTempDirectory)
34+
35+upload :: RunReporter -> GhRepoOwner -> GhRepoName -> EvaluationResult -> RepoPublicity -> M ()
36+upload = curry5 $ mockable #s3CacheUploadMock $ \(runReporter, repoOwner, repoName, evalResult, repoPublicity) -> do
37+ withTextSpan ("phase", "s3-cache-upload") $ do
38+ withSpan (Garnix.S3Cache.getPackageName (evalResult ^. #derivation)) $ do
39+ storePathClosure <- nubOrd . mconcat . catMaybes <$> forM (evalResult ^. #toUpload) Nix.getClosure
40+ notInNixosCache <- filterM (fmap not . isInNixosCache) storePathClosure
41+ notInS3Cache <- DB.claimS3CachedStorePaths notInNixosCache
42+ forM_ (notInNixosCache \\ notInS3Cache) $ \storePath -> do
43+ DB.tagCacheUploadForS3Cache repoOwner repoName $ getHash storePath
44+ forConcurrently_ notInS3Cache $ \storePath -> do
45+ dirSize <- liftIO $ getDirSize $ cs $ getStorePath storePath
46+ limit <- view (#s3CacheEnv . #maxUploadSize)
47+ if dirSize > limit
48+ then do
49+ log Notice $ getStorePath storePath <> " too big (" <> show dirSize <> "), not uploading"
50+ reportLogs runReporter
51+ $ mkLogLine
52+ ( getStorePath storePath
53+ <> " is "
54+ <> show dirSize
55+ <> " bytes, the limit is "
56+ <> show limit
57+ <> ". Not uploading to the garnix binary cache."
58+ )
59+ else do
60+ uploadStorePath repoOwner repoName storePath repoPublicity <?> "uploading to s3-cache"
61+ reportLogs runReporter $ mkLogLine ("Uploaded " <> getStorePath storePath <> " to the garnix binary cache.")
62+
63+getPackageName :: DrvPath -> PackageName
64+getPackageName drvPath =
65+ case T.stripSuffix ".drv" (cs (getName $ getDrvPath drvPath)) of
66+ Just name -> PackageName name
67+ Nothing -> "<unknown>"
68+
69+getDirSize :: FilePath -> IO Integer
70+getDirSize path = do
71+ isSymLink <- pathIsSymbolicLink path
72+ if isSymLink
73+ then pure 0
74+ else do
75+ isDir <- doesDirectoryExist path
76+ if isDir
77+ then do
78+ entries <- listDirectory path
79+ sum <$> mapM (getDirSize . (path </>)) entries
80+ else do
81+ isFile <- doesFileExist path
82+ if isFile then getFileSize path else pure 0
83+
84+uploadStorePath :: GhRepoOwner -> GhRepoName -> StorePath -> RepoPublicity -> M ()
85+uploadStorePath repoOwner repoName storePath repoPublicity = do
86+ nixConfig <- view #userNixConfig
87+ withPoolM s3UploadPool repoOwner
88+ $ withBinaryFileInTempDir
89+ $ \(narFilePath, narFileHandle) -> do
90+ runSubProcess_
91+ $ cmd "nix"
92+ & addArgs ["nar", "pack", cs storePath :: Text]
93+ & addStdoutHandle narFileHandle
94+ & addNixConfigEnvironment nixConfig
95+ liftIO $ IO.hClose narFileHandle
96+ narSize <- fromIntegral <$> Amazonka.getFileSize narFilePath
97+ narHash <- getFileHash narFilePath
98+ compressedNarFilePath <- compress narFilePath
99+ bucket <-
100+ if isRepoPublic repoPublicity
101+ then view $ #s3CacheEnv . #publicBucket
102+ else view $ #s3CacheEnv . #privateBucket
103+ body <- Amazonka.toBody <$> Amazonka.hashedFile compressedNarFilePath
104+ let policy :: RetryPolicyM M
105+ policy =
106+ limitRetriesByCumulativeDelay
107+ (toMicroseconds (fromMinutes @Int 30))
108+ (limitRetries 5 <> fullJitterBackoff (toMicroseconds (fromMilliSeconds @Int 100)))
109+ void
110+ $ timingAs #cachePushTime
111+ $ retryingWithPolicy policy
112+ $ sendWithLogging
113+ $ Amazonka.newPutObject
114+ bucket
115+ (Amazonka.ObjectKey $ toNarFilePath storePath XZ)
116+ body
117+ fileHash <- getFileHash compressedNarFilePath
118+ (sig, pathInfo) <- signStorePath storePath
119+ let references = T.unwords $ fmap Nix.getRelativeStorePath (pathInfo ^. #references)
120+ fileSize <- fromIntegral <$> Amazonka.getFileSize compressedNarFilePath
121+ DB.finalizeS3CacheUpload
122+ $ DB.S3CacheStoreHash
123+ { DB.hash = getHash storePath,
124+ DB.packageName = getName storePath,
125+ narHash,
126+ narSize,
127+ public = isRepoPublic repoPublicity,
128+ sig,
129+ references,
130+ fileSize,
131+ fileHash
132+ }
133+ DB.tagCacheUploadForS3Cache repoOwner repoName $ getHash storePath
134+ incrementEvent #s3CacheUploads
135+
136+getFileHash :: FilePath -> M Text
137+getFileHash file = do
138+ StdoutTrimmed hash <-
139+ runSubProcess
140+ $ cmd "nix-hash"
141+ & addArgs
142+ ["--base32", "--type", "sha256", "--flat", file]
143+ pure hash
144+
145+withBinaryFileInTempDir :: ((FilePath, Handle) -> M a) -> M a
146+withBinaryFileInTempDir action = do
147+ withSystemTempDirectory "garnix-narfile" $ \tempDir -> do
148+ let file = tempDir </> "file"
149+ liftBaseOp (withBinaryFile file IO.WriteMode) $ \handle -> do
150+ action (file, handle)
151+
152+compress :: FilePath -> M FilePath
153+compress file = do
154+ run_ $ cmd "xz"
155+ & addArgs [file]
156+ & setWorkingDir (takeDirectory file)
157+ pure (file <> ".xz")
158+
159+nixosCacheKeyName :: Text
160+nixosCacheKeyName = "cache.nixos.org-1"
161+
162+isInNixosCache :: StorePath -> M Bool
163+isInNixosCache storePath = memoize (#s3CacheEnv . #isInNixosCacheMemoTable) (getHash storePath) $ do
164+ pathInfo <- getPathInfo storePath
165+ let nixosSignatures = Nix.signaturesForCacheKey pathInfo nixosCacheKeyName
166+ pure $ not $ null nixosSignatures
167+
168+signStorePath :: StorePath -> M (Text, Nix.PathInfo)
169+signStorePath storePath = do
170+ cachePrivKeyFile <- view $ #s3CacheEnv . #cachePrivKeyFile
171+ cachePrivKeyName <- view $ #s3CacheEnv . #cachePrivKeyName
172+ nixConfig <- view #userNixConfig
173+ runSubProcess_
174+ $ cmd "nix"
175+ & addArgs ["store", "sign", "-k", cs cachePrivKeyFile, cs storePath :: Text]
176+ & addNixConfigEnvironment nixConfig
177+ pathInfo <- getPathInfo storePath
178+ let garnixSignatures = Nix.signaturesForCacheKey pathInfo cachePrivKeyName
179+ case garnixSignatures of
180+ sig : _ -> pure (sig, pathInfo)
181+ _ -> throw $ OtherError "Error parsing `nix path-info` output: no signature for cachePrivKey"
182+
183+toNarFilePath :: StorePath -> Compression -> Text
184+toNarFilePath storePath compression =
185+ getRelativeStorePath storePath <> ".nar." <> case compression of
186+ XZ -> "xz"
187+
188+sendWithLogging ::
189+ (Amazonka.AWSRequest request, Typeable request, Typeable (Amazonka.AWSResponse request)) =>
190+ request ->
191+ M (Amazonka.AWSResponse request)
192+sendWithLogging request = do
193+ withTextSpan ("tag", "amazonka-log") $ do
194+ logger <- view #logger
195+ spans <- view #spanCtx
196+ s3Env <-
197+ view (#s3CacheEnv . #amazonkaEnv)
198+ <&> #logger .~ amazonkaLogger spans logger
199+ response <-
200+ liftIO
201+ $ runResourceT
202+ $ Amazonka.sendEither s3Env request
203+ case response of
204+ Left error -> throw $ OtherError $ show error
205+ Right response -> pure response
206+ where
207+ amazonkaLogger :: [(Text, Text)] -> (LogItem -> IO ()) -> Amazonka.Logger
208+ amazonkaLogger spans logger logLevel message = do
209+ when (Amazonka.Info >= logLevel) $ do
210+ logger $ LogItem (convertLogLevel logLevel) spans $ cs $ ByteString.toLazyByteString message
211+
212+ convertLogLevel :: Amazonka.LogLevel -> Severity
213+ convertLogLevel = \case
214+ Amazonka.Info -> Informational
215+ Amazonka.Error -> Error
216+ Amazonka.Debug -> Informational
217+ Amazonka.Trace -> Informational
backend/src/Garnix/SafeUnix.hsnew file+29
@@ -0,0 +1,29 @@
1+module Garnix.SafeUnix (safeCreatePipe) where
2+
3+import Foreign
4+import Foreign.C
5+import Garnix.Prelude
6+import System.Posix.IO (fdToHandle)
7+import System.Posix.Types (Fd (..))
8+
9+-- O_CLOEXEC from fcntl.h. See `man 2 open` and `man 2 pipe`.
10+oCLOEXEC :: CInt
11+oCLOEXEC = 524288
12+
13+safeCreatePipe :: IO (Handle, Handle)
14+safeCreatePipe = do
15+ (readfd, writefd) <- safeCreatePipeFd
16+ readh <- fdToHandle readfd
17+ writeh <- fdToHandle writefd
18+ return (readh, writeh)
19+
20+safeCreatePipeFd :: IO (Fd, Fd)
21+safeCreatePipeFd =
22+ allocaArray 2 $ \p_fd -> do
23+ throwErrnoIfMinus1_ "safeCreatePipe" (c_pipe2 p_fd oCLOEXEC)
24+ rfd <- Fd <$> peekElemOff p_fd 0
25+ wfd <- Fd <$> peekElemOff p_fd 1
26+ return (rfd, wfd)
27+
28+foreign import ccall unsafe "pipe2"
29+ c_pipe2 :: Ptr CInt -> CInt -> IO CInt
backend/src/Garnix/Sandbox.hsnew file+102
@@ -0,0 +1,102 @@
1+module Garnix.Sandbox (inNixSandbox, SandboxAccessType (..)) where
2+
3+import Cradle
4+import Cradle.ProcessConfiguration
5+import Data.List
6+import Garnix.Monad hiding (workingDir)
7+import Garnix.NixConfig (getNetRcFileSetting)
8+import Garnix.Prelude
9+import Garnix.Types (NetRcFile (..))
10+import System.Directory (createDirectoryIfMissing, getCurrentDirectory)
11+import System.Environment (getEnv)
12+
13+-- A sandbox appropriate for nix commands.
14+inNixSandbox :: [(FilePath, SandboxAccessType)] -> Maybe FilePath -> M ProcessConfiguration -> M ProcessConfiguration
15+inNixSandbox extraSandboxPaths xdgCacheHome procConfigM = do
16+ procConfig <- procConfigM
17+ path <- liftIO $ getEnv "PATH"
18+ let env = fromMaybe identity (environmentModification procConfig) [("PATH", path)]
19+ netrcFile <- do
20+ userNixConfig <- view #userNixConfig
21+ pure $ case getNetRcFileSetting userNixConfig of
22+ (Just (NetRcFile netrcFile)) -> [(netrcFile, ReadOnly)]
23+ Nothing -> []
24+ dir <- case workingDir procConfig of
25+ Nothing -> liftIO getCurrentDirectory
26+ Just d -> pure d
27+ args <- argsForNixSandbox xdgCacheHome dir (extraSandboxPaths ++ netrcFile) env
28+ let oldCommand = executable procConfig : arguments procConfig
29+ pure
30+ $ procConfig
31+ { executable = "bwrap",
32+ arguments = args <> oldCommand
33+ }
34+
35+data SandboxAccessType = TryReadOnly | ReadOnly | ReadWrite | LockFile
36+ deriving (Eq)
37+
38+argsForNixSandbox :: (MonadIO m) => Maybe FilePath -> FilePath -> [(FilePath, SandboxAccessType)] -> [(String, String)] -> m [String]
39+argsForNixSandbox xdgCacheHome workingDir extraPaths env = do
40+ let extraPathOptions = flip concatMap extraPaths $ \case
41+ (path, TryReadOnly) -> ["--ro-bind-try", path, path]
42+ (path, ReadOnly) -> ["--ro-bind", path, path]
43+ (path, ReadWrite) -> ["--bind", path, path]
44+ (path, LockFile) -> ["--bind", path, path, "--lock-file", path]
45+ -- We use `-try` versions often since many of these dirs don't always exist.
46+ let baseArgs =
47+ ["--proc", "/proc"]
48+ <> ["--dev", "/dev"]
49+ <> ["--tmpfs", "/tmp"]
50+ <> ["--die-with-parent"]
51+ <> ["--unshare-pid"]
52+ <> ["--clearenv"]
53+ <> ["--ro-bind-try", "/bin/sh", "/bin/sh"]
54+ -- necessary for running tests on some non nixos systems
55+ let nonNixosSharedLibArgs =
56+ ["--ro-bind-try", "/lib", "/lib"]
57+ <> ["--ro-bind-try", "/lib64", "/lib64"]
58+ let nixArgs =
59+ ["--ro-bind-try", "/nix/", "/nix/"]
60+ <> ["--bind-try", "/nix/var/log/", "/nix/var/log/"]
61+ <> ["--ro-bind-try", "/etc/nix/machines", "/etc/nix/machines"]
62+ <> ["--ro-bind-try", "/etc/static/nix/machines", "/etc/static/nix/machines"]
63+ <> ["--ro-bind-try", "/etc/nix/nix.conf", "/etc/nix/nix.conf"]
64+ <> ["--ro-bind-try", "/etc/static/nix/nix.conf", "/etc/static/nix/nix.conf"]
65+ let networkingArgs =
66+ ["--ro-bind-try", "/etc/localtime", "/etc/localtime"]
67+ <> ["--ro-bind-try", "/etc/ssl", "/etc/ssl"]
68+ <> ["--ro-bind-try", "/etc/static/ssl", "/etc/static/ssl"]
69+ <> ["--ro-bind-try", "/etc/zoneinfo", "/etc/zoneinfo"]
70+ <> ["--ro-bind-try", "/etc/static/zoneinfo", "/etc/static/zoneinfo"]
71+ <> ["--ro-bind-try", "/etc/resolv.conf", "/etc/resolv.conf"]
72+ <> ["--ro-bind-try", "/etc/nsswitch.conf", "/etc/nsswitch.conf"]
73+ <> ["--ro-bind-try", "/etc/hostname", "/etc/hostname"]
74+ <> ["--ro-bind-try", "/etc/static/hostname", "/etc/static/hostname"]
75+ let systemProgramArgs = ["--ro-bind-try", "/run/current-system/sw", "/run/current-system/sw"]
76+ let sandboxHome = "/home/nix-runner"
77+ for_ xdgCacheHome $ \cache ->
78+ liftIO $ createDirectoryIfMissing True (cache </> "nix/gitv3")
79+ let homeArgs =
80+ ["--tmpfs", sandboxHome, "--setenv", "HOME", sandboxHome]
81+ <> ["--setenv", "XDG_CACHE_HOME", sandboxHome </> ".cache"]
82+ <> maybe
83+ ["--tmpfs", sandboxHome </> ".cache", "--tmpfs", sandboxHome </> ".cache/nix/gitv3"]
84+ (\dir -> ["--bind", dir, sandboxHome </> ".cache"])
85+ xdgCacheHome
86+ nixConfigForTests <- do
87+ hostHome <- liftIO $ getEnv "HOME"
88+ let nixConfigPath home = home </> ".config/nix/nix.conf"
89+ pure ["--ro-bind-try", nixConfigPath hostHome, nixConfigPath sandboxHome]
90+ let workingDirArgs = ["--bind", workingDir, workingDir]
91+ let setEnvs = concatMap (\(name, value) -> ["--setenv", name, value]) env
92+ pure
93+ $ baseArgs
94+ <> nonNixosSharedLibArgs
95+ <> nixArgs
96+ <> networkingArgs
97+ <> systemProgramArgs
98+ <> homeArgs
99+ <> nixConfigForTests
100+ <> workingDirArgs
101+ <> extraPathOptions
102+ <> setEnvs
backend/src/Garnix/StripeLib.hsnew file+298
@@ -0,0 +1,298 @@
1+{-# LANGUAGE DuplicateRecordFields #-}
2+{-# LANGUAGE ExtendedDefaultRules #-}
3+{-# OPTIONS_GHC -Wno-type-defaults #-}
4+
5+module Garnix.StripeLib (module Garnix.StripeLib, module Garnix.StripeLib.Types) where
6+
7+import Control.Concurrent
8+import Control.Exception.Safe qualified as SafeException
9+import Data.Aeson
10+import Data.Aeson.Encode.Pretty (encodePretty)
11+import Data.Map.Strict (Map, insert, member)
12+import Data.Row (Rec, (.+), (.==), type (.+), type (.==))
13+import Garnix.Duration
14+import Garnix.Monad
15+import Garnix.Prelude hiding (get, insert, putStr)
16+import Garnix.StripeLib.Types
17+import Garnix.Types
18+import Network.Wreq (FormParam (..))
19+import Network.Wreq qualified as Wreq
20+
21+testImplementation ::
22+ IO
23+ ( Mock (GhRepoOwner, Name, Email) CustomerDto,
24+ Mock (CustomerId, PriceId, Text, Text) SubscriptionDto,
25+ Mock (CustomerId, InvoiceId, Text, UnitAmount, Int64) (),
26+ Mock CustomerId SubscriptionListDto,
27+ Mock SubscriptionId (),
28+ Mock PriceId PriceDto
29+ )
30+testImplementation = do
31+ customers :: MVar (Int, Map Email Int) <- newMVar (0, mempty)
32+ createCustomer <- newMock $ \(repoOwner, name, email) ->
33+ liftIO $ modifyMVar customers $ \(currentId, customers) -> do
34+ when (email `member` customers) $ do
35+ error $ "already a customer: " <> getEmail email
36+ let id = CustomerId $ "test-customer-id-" <> show currentId
37+ pure
38+ ( (currentId + 1, insert email currentId customers),
39+ #object
40+ .== ConstString @"customer"
41+ .+ #id
42+ .== id
43+ .+ #name
44+ .== name
45+ .+ #email
46+ .== email
47+ .+ #metadata
48+ .== insert "github_account" (getGhLogin $ getGhRepoOwner repoOwner) mempty
49+ )
50+ now <- getCurrentTime
51+ subscriptions :: MVar (Int, [SubscriptionId]) <- newMVar (0, mempty)
52+ createSubscription <- newMock $ \(customerId, _priceId, description, product) -> do
53+ liftIO $ modifyMVar subscriptions $ \(currentId, subscriptions) -> do
54+ let subscriptionId = SubscriptionId $ "test-subscription-id-" <> show currentId
55+ let subscription =
56+ #object
57+ .== (ConstString @"subscription")
58+ .+ #id
59+ .== subscriptionId
60+ .+ #customer
61+ .== customerId
62+ .+ #latest_invoice
63+ .== ( #object
64+ .== (ConstString @"invoice")
65+ .+ #id
66+ .== InvoiceId "test-invoice-id"
67+ .+ #payment_intent
68+ .== ( #object
69+ .== (ConstString @"payment_intent")
70+ .+ #id
71+ .== "test-payment-intent-id"
72+ .+ #status
73+ .== "requires_payment_method"
74+ .+ #client_secret
75+ .== ClientSecret ("client_secret(" <> getCustomerId customerId <> ")")
76+ )
77+ )
78+ .+ #status .== SubscriptionStatusIncomplete
79+ .+ #pending_setup_intent
80+ .== Null
81+ .+ #description
82+ .== Just description
83+ .+ #metadata .== insert "product" product mempty
84+ .+ #current_period_start .== UnixTimeStamp now
85+ .+ #current_period_end .== UnixTimeStamp (addTime (fromDays @Int 30) now)
86+ pure ((currentId + 1, subscriptionId : subscriptions), subscription)
87+ createInvoice <- newMock $ \(_customerId, _invoiceId, _description, _unitAmount, _quantity) -> pure ()
88+ listSubscriptions <- newMock $ \_customer -> do
89+ (_, subscriptionIds) <- liftIO $ readMVar subscriptions
90+ pure
91+ $ #object
92+ .== (ConstString @"list")
93+ .+ #data
94+ .== map
95+ ( \subscriptionId ->
96+ #object .== (ConstString @"subscription")
97+ .+ #id .== subscriptionId
98+ )
99+ subscriptionIds
100+ cancelSubscription <- newMock $ \subscriptionIdToCancel -> do
101+ liftIO $ modifyMVar_ subscriptions $ pure . second (filter (== subscriptionIdToCancel))
102+ getPrice <- newMock $ \priceId -> do
103+ pure
104+ ( #id
105+ .== priceId
106+ .+ #currency
107+ .== "usd"
108+ .+ #unit_amount
109+ .== 4200
110+ )
111+ pure (createCustomer, createSubscription, createInvoice, listSubscriptions, cancelSubscription, getPrice)
112+
113+createCustomer :: GhRepoOwner -> Name -> Email -> M CustomerDto
114+createCustomer = curry3 $ mockable #createCustomerMock $ \(repoOwner, name, email) -> do
115+ let body :: [FormParam] =
116+ [ "name" := getName name,
117+ "email" := getEmail email,
118+ "metadata[github_account]" := getGhLogin (getGhRepoOwner repoOwner)
119+ ]
120+ post "/customers" body
121+
122+getCustomers :: M (StripeListPage CustomerDto)
123+getCustomers = get "/customers"
124+
125+createSubscription :: CustomerId -> PriceId -> Text -> Text -> M SubscriptionDto
126+createSubscription = curry4 $ mockable #createSubscriptionMock $ \(customerId, priceId, product, description) -> do
127+ let body :: [FormParam] =
128+ [ "customer" := customerId,
129+ "items[0][price]" := getPriceId priceId,
130+ "payment_behavior" := "default_incomplete",
131+ "payment_settings[save_default_payment_method]" := "on_subscription",
132+ "expand[]" := "latest_invoice.payment_intent",
133+ "expand[]" := "pending_setup_intent",
134+ "description" := description,
135+ "metadata[product]" := product
136+ ]
137+ post "/subscriptions" body
138+
139+listSubscriptions :: CustomerId -> M SubscriptionListDto
140+listSubscriptions = mockable #listSubscriptionsMock $ \customer -> do
141+ getWith (Wreq.param "customer" .~ [getCustomerId customer]) "/subscriptions"
142+
143+cancelSubscription :: SubscriptionId -> M ()
144+cancelSubscription = mockable #cancelSubscriptionMock $ \subscription -> do
145+ delete $ "/subscriptions/" <> getSubscriptionId subscription
146+
147+createInvoiceItem :: CustomerId -> InvoiceId -> Text -> UnitAmount -> Int64 -> M ()
148+createInvoiceItem = curry5 $ mockable #createInvoiceItemMock $ \(customerId, invoiceId, description, unitAmount, quantity) -> do
149+ let body :: [FormParam] =
150+ [ "customer" := customerId,
151+ "currency" := "usd",
152+ "description" := description,
153+ "invoice" := invoiceId,
154+ "unit_amount_decimal" := toDecimalInCents unitAmount,
155+ "quantity" := quantity
156+ ]
157+ post "/invoiceitems" body
158+
159+getEvents :: M (StripeListPage (EventDto Text ()))
160+getEvents = get "/events"
161+
162+getEvent :: Text -> M (EventDto Text ())
163+getEvent id = get $ "/events/" <> id
164+
165+getPrice :: PriceId -> M PriceDto
166+getPrice = mockable #getPriceMock $ \id -> get $ "/prices/" <> getPriceId id
167+
168+taxCalculation :: Int64 -> String -> AddressDto -> M TaxCalculationDto
169+taxCalculation unitAmount currency address = do
170+ let body :: [FormParam] =
171+ [ "currency" := currency,
172+ "line_items[0][amount]" := unitAmount,
173+ "line_items[0][reference]" := "L1",
174+ "line_items[0][tax_behavior]" := "inclusive",
175+ "customer_details[address_source]" := "billing",
176+ "customer_details[address][line1]" := address ^. #line1,
177+ "customer_details[address][line2]" := address ^. #line2,
178+ "customer_details[address][city]" := address ^. #city,
179+ "customer_details[address][state]" := address ^. #state,
180+ "customer_details[address][postal_code]" := address ^. #postal_code,
181+ "customer_details[address][country]" := address ^. #country
182+ ]
183+ post "/tax/calculations" body
184+
185+-- * webhooks
186+
187+data SubscriptionCreatedOrUpdatedEvent = SubscriptionCreatedOrUpdatedEvent
188+ { eventType :: CustomerSubscriptionEventType,
189+ customerId :: CustomerId,
190+ priceId :: PriceId,
191+ status :: SubscriptionStatus,
192+ periodStart :: UTCTime,
193+ periodEnd :: UTCTime
194+ }
195+ deriving stock (Show, Eq, Generic)
196+
197+data InvoiceCreatedEvent = InvoiceCreatedEvent
198+ { customerId :: CustomerId,
199+ invoiceId :: InvoiceId,
200+ reason :: InvoiceBillingReason,
201+ periodStart :: UTCTime,
202+ periodEnd :: UTCTime
203+ }
204+ deriving stock (Show, Eq, Generic)
205+
206+data WebhookEvent
207+ = SubscriptionCreatedOrUpdated SubscriptionCreatedOrUpdatedEvent
208+ | InvoiceCreated InvoiceCreatedEvent
209+ deriving stock (Show, Eq)
210+
211+fromWebhookRequest :: Value -> M (Maybe WebhookEvent)
212+fromWebhookRequest v = case fromJSON v :: Result (Rec ("object" .== ConstString "event" .+ "type" .== Text)) of
213+ Data.Aeson.Error e -> throw $ DecodeError (cs $ encodePretty v) (cs e)
214+ Data.Aeson.Success event -> do
215+ case event ^. #type of
216+ "customer.subscription.created" -> parseSubscriptionCreatedOrUpdated
217+ "customer.subscription.updated" -> parseSubscriptionCreatedOrUpdated
218+ "invoice.created" -> parseInvoiceCreated
219+ _ -> pure Nothing
220+ where
221+ parseSubscriptionCreatedOrUpdated =
222+ case fromJSON v :: Result CustomerSubscriptionCreatedOrUpdatedEventDto of
223+ Data.Aeson.Success event ->
224+ case event ^. #data . #object . #items . #data . to (map (^. #price . #id)) of
225+ [priceId] ->
226+ pure
227+ $ Just
228+ $ SubscriptionCreatedOrUpdated
229+ $ SubscriptionCreatedOrUpdatedEvent
230+ { eventType = event ^. #type,
231+ customerId = event ^. #data . #object . #customer,
232+ priceId,
233+ status = event ^. #data . #object . #status,
234+ periodStart = getUnixTimeStamp $ event ^. #data . #object . #current_period_start,
235+ periodEnd = getUnixTimeStamp $ event ^. #data . #object . #current_period_end
236+ }
237+ priceIds -> throw $ DecodeError {original = cs $ encode priceIds, message = "expected: exactly one price_id"}
238+ Data.Aeson.Error e -> throw $ DecodeError (cs $ encodePretty v) (cs e)
239+
240+ parseInvoiceCreated =
241+ case fromJSON v :: Result InvoiceCreatedEventDto of
242+ Data.Aeson.Success event ->
243+ pure
244+ $ Just
245+ $ InvoiceCreated
246+ $ InvoiceCreatedEvent
247+ { customerId = event ^. #data . #object . #customer,
248+ invoiceId = event ^. #data . #object . #id,
249+ reason = event ^. #data . #object . #billing_reason,
250+ periodStart = getUnixTimeStamp $ event ^. #data . #object . #period_start,
251+ periodEnd = getUnixTimeStamp $ event ^. #data . #object . #period_end
252+ }
253+ Data.Aeson.Error e -> throw $ DecodeError (cs $ encodePretty v) (cs e)
254+
255+-- * stripe http helpers
256+
257+type StripeListPage a =
258+ Rec
259+ ( "object"
260+ .== ConstString "list"
261+ .+ "has_more"
262+ .== Bool
263+ .+ "data"
264+ .== [a]
265+ )
266+
267+withStripeAuth :: (Wreq.Options -> IO a) -> M a
268+withStripeAuth action = do
269+ secretKey <- view $ #stripe . #secretKey . to cs
270+ withWreqOptions $ \options -> do
271+ action
272+ $ options
273+ & Wreq.auth
274+ ?~ Wreq.basicAuth secretKey ""
275+
276+get :: (FromJSON a) => Text -> M a
277+get = getWith identity
278+
279+getWith :: (FromJSON a) => (Wreq.Options -> Wreq.Options) -> Text -> M a
280+getWith setOpts path = do
281+ res <-
282+ withStripeAuth (\opts -> Wreq.getWith (setOpts opts) ("https://api.stripe.com/v1" <> cs path))
283+ `SafeException.catchAny` (throw . OtherError . show)
284+ aesonDecode ("stripe GET /v1" <> path) parseJSON $ cs $ res ^. Wreq.responseBody
285+
286+post :: (FromJSON b) => Text -> [FormParam] -> M b
287+post path body = do
288+ res <-
289+ withStripeAuth (\opts -> Wreq.postWith opts ("https://api.stripe.com/v1" <> cs path) body)
290+ `SafeException.catchAny` (throw . OtherError . show)
291+ aesonDecode ("stripe POST /v1" <> path) parseJSON $ cs $ res ^. Wreq.responseBody
292+
293+delete :: (FromJSON b) => Text -> M b
294+delete path = do
295+ res <-
296+ withStripeAuth (\opts -> Wreq.deleteWith opts ("https://api.stripe.com/v1" <> cs path))
297+ `SafeException.catchAny` (throw . OtherError . show)
298+ aesonDecode ("stripe POST /v1" <> path) parseJSON $ cs $ res ^. Wreq.responseBody
backend/src/Garnix/StripeLib/Types.hsnew file+238
@@ -0,0 +1,238 @@
1+module Garnix.StripeLib.Types where
2+
3+import Data.Aeson (Options (constructorTagModifier), Value (..), camelTo2, defaultOptions, genericParseJSON, withText)
4+import Data.Map.Strict (Map)
5+import Data.Row (Rec, type (.+), type (.==))
6+import Data.Time.Clock.POSIX (posixSecondsToUTCTime)
7+import GHC.TypeLits (KnownSymbol, symbolVal)
8+import Garnix.MonetaryCost
9+import Garnix.Prelude
10+import Garnix.Types
11+import Prelude qualified (Show (..))
12+
13+newtype UnitAmount = UnitAmount {toDecimalInCents :: Text}
14+ deriving stock (Generic, Show, Eq)
15+
16+unitAmountFromCost :: MonetaryCost -> UnitAmount
17+unitAmountFromCost = UnitAmount . show . getCostInCents
18+
19+newtype Name = Name {getName :: Text}
20+ deriving stock (Generic, Show, Eq)
21+ deriving newtype (FromJSON, ToJSON)
22+
23+newtype PriceId = PriceId {getPriceId :: Text}
24+ deriving stock (Generic, Show, Eq)
25+ deriving newtype (FromJSON, ToJSON, FromHttpApiData)
26+
27+type CustomerDto =
28+ Rec
29+ ( "object" .== ConstString "customer"
30+ .+ "id" .== CustomerId
31+ .+ "name" .== Name
32+ .+ "email" .== Email
33+ .+ "metadata" .== Map Text Text
34+ )
35+
36+type SubscriptionDto =
37+ Rec
38+ ( "object" .== ConstString "subscription"
39+ .+ "id" .== SubscriptionId
40+ .+ "customer" .== CustomerId
41+ .+ "latest_invoice" .== InvoiceDto
42+ .+ "pending_setup_intent" .== Value
43+ .+ "status" .== SubscriptionStatus
44+ .+ "description" .== Maybe Text
45+ .+ "metadata" .== Map Text Text
46+ .+ "current_period_start" .== UnixTimeStamp
47+ .+ "current_period_end" .== UnixTimeStamp
48+ )
49+
50+type SubscriptionListDto =
51+ Rec
52+ ( "object" .== ConstString "list"
53+ .+ "data"
54+ .== [ Rec
55+ ( "object" .== ConstString "subscription"
56+ .+ "id" .== SubscriptionId
57+ )
58+ ]
59+ )
60+
61+data SubscriptionStatus
62+ = SubscriptionStatusIncomplete
63+ | SubscriptionStatusIncompleteExpired
64+ | SubscriptionStatusTrialing
65+ | SubscriptionStatusActive
66+ | SubscriptionStatusPastDue
67+ | SubscriptionStatusCanceled
68+ | SubscriptionStatusUnpaid
69+ | SubscriptionStatusPaused
70+ deriving (Generic, Show, Eq)
71+
72+instance FromJSON SubscriptionStatus where
73+ parseJSON =
74+ genericParseJSON
75+ $ defaultOptions
76+ { constructorTagModifier = \constructor ->
77+ camelTo2 '_'
78+ $ fromMaybe (error $ "unexpected constructor prefix: " <> cs constructor)
79+ $ stripPrefix ("SubscriptionStatus" :: String) constructor
80+ }
81+
82+type InvoiceDto =
83+ Rec
84+ ( "object" .== ConstString "invoice"
85+ .+ "id" .== InvoiceId
86+ .+ "payment_intent" .== PaymentIntentDto
87+ )
88+
89+type PaymentIntentDto =
90+ Rec
91+ ( "object" .== ConstString "payment_intent"
92+ .+ "id" .== Text
93+ .+ "status" .== Text
94+ .+ "client_secret" .== ClientSecret
95+ )
96+
97+newtype ClientSecret = ClientSecret Text
98+ deriving stock (Generic, Show, Eq)
99+ deriving newtype (FromJSON, ToJSON)
100+
101+type PriceDto =
102+ Rec
103+ ( "id" .== PriceId
104+ .+ "currency" .== Text
105+ .+ "unit_amount" .== Int64
106+ )
107+
108+type EventDto typ inner =
109+ Rec
110+ ( "object" .== ConstString "event"
111+ .+ "api_version" .== Text
112+ .+ "created" .== UnixTimeStamp
113+ .+ "type" .== typ
114+ .+ "request"
115+ .== Rec
116+ ( "id" .== Maybe Text
117+ .+ "idempotency_key" .== Maybe Text
118+ )
119+ .+ "data"
120+ .== Rec
121+ ( "object" .== inner
122+ )
123+ )
124+
125+type CustomerSubscriptionCreatedOrUpdatedEventDto =
126+ EventDto
127+ CustomerSubscriptionEventType
128+ ( Rec
129+ ( "customer" .== CustomerId
130+ .+ "items"
131+ .== Rec
132+ ( "data"
133+ .== [Rec ("price" .== Rec ("id" .== PriceId))]
134+ )
135+ .+ "status"
136+ .== SubscriptionStatus
137+ .+ "current_period_start" .== UnixTimeStamp
138+ .+ "current_period_end" .== UnixTimeStamp
139+ )
140+ )
141+
142+type InvoiceCreatedEventDto =
143+ EventDto
144+ (ConstString "invoice.created")
145+ ( Rec
146+ ( "id" .== InvoiceId
147+ .+ "customer" .== CustomerId
148+ .+ "billing_reason" .== InvoiceBillingReason
149+ .+ "period_start" .== UnixTimeStamp
150+ .+ "period_end" .== UnixTimeStamp
151+ )
152+ )
153+
154+data InvoiceBillingReason
155+ = -- | A subscription advanced into a new period.
156+ SubscriptionCycle
157+ | -- | A new subscription was created.
158+ SubscriptionCreate
159+ | -- | Unhandled billing reason -- see `billing_reason` on stripe invoice object.
160+ OtherInvoiceBillingReason Text
161+ deriving stock (Show, Eq)
162+
163+instance FromJSON InvoiceBillingReason where
164+ parseJSON = withText "InvoiceBillingReason" $ \case
165+ "subscription_create" -> pure SubscriptionCreate
166+ "subscription_cycle" -> pure SubscriptionCycle
167+ text -> pure $ OtherInvoiceBillingReason text
168+
169+data CustomerSubscriptionEventType = Created | Updated
170+ deriving stock (Show, Eq)
171+
172+instance FromJSON CustomerSubscriptionEventType where
173+ parseJSON = withText "CustomerSubscriptionEventType" $ \case
174+ "customer.subscription.created" -> pure Created
175+ "customer.subscription.updated" -> pure Updated
176+ text -> fail $ "unknown event type: " <> cs text
177+
178+type AddressDto =
179+ Rec
180+ ( "line1" .== Text
181+ .+ "line2" .== Maybe Text
182+ .+ "city" .== Text
183+ .+ "state" .== Text
184+ .+ "postal_code" .== Text
185+ .+ "country" .== Text
186+ )
187+
188+type TaxCalculationDto =
189+ Rec
190+ ( "object" .== ConstString "tax.calculation"
191+ .+ "id" .== Text
192+ .+ "currency" .== Text
193+ .+ "amount_total" .== Int64
194+ .+ "tax_breakdown"
195+ .== [ Rec
196+ ( "amount" .== Int64
197+ .+ "tax_rate_details"
198+ .== Rec
199+ ( "tax_type" .== Maybe Text
200+ )
201+ )
202+ ]
203+ .+ "customer_details"
204+ .== Rec
205+ ( "address" .== AddressDto
206+ )
207+ )
208+
209+-- * JSON helpers
210+
211+data ConstString (value :: Symbol) = ConstString
212+ deriving stock (Eq)
213+
214+instance (KnownSymbol symbol) => Show (ConstString symbol) where
215+ show ConstString = symbolVal $ Proxy @symbol
216+
217+instance (KnownSymbol symbol) => FromJSON (ConstString symbol) where
218+ parseJSON =
219+ let constString = symbolVal (Proxy @symbol)
220+ in withText ("ConstString " <> cs (show constString)) $ \text ->
221+ if text == cs constString
222+ then pure ConstString
223+ else fail $ "expected const string: " <> cs (show (symbolVal (Proxy @symbol))) <> ", got: " <> cs (show text)
224+
225+instance (KnownSymbol symbol) => ToJSON (ConstString symbol) where
226+ toJSON ConstString = String $ cs $ symbolVal $ Proxy @symbol
227+
228+-- Like UTCTime, but the json serialization is the number of seconds since epoch.
229+newtype UnixTimeStamp = UnixTimeStamp {getUnixTimeStamp :: UTCTime}
230+ deriving newtype (Eq)
231+
232+instance Show UnixTimeStamp where
233+ show (UnixTimeStamp utcTime) = "<" <> Prelude.show utcTime <> ">"
234+
235+instance FromJSON UnixTimeStamp where
236+ parseJSON value = do
237+ n :: Int64 <- parseJSON value
238+ pure $ UnixTimeStamp $ posixSecondsToUTCTime $ fromIntegral n
backend/src/Garnix/Types.hsnew file+1808
@@ -0,0 +1,1808 @@
1+{-# LANGUAGE DuplicateRecordFields #-}
2+{-# LANGUAGE FunctionalDependencies #-}
3+{-# LANGUAGE NamedFieldPuns #-}
4+{-# LANGUAGE TemplateHaskell #-}
5+{-# LANGUAGE UndecidableInstances #-}
6+{-# OPTIONS_GHC -fno-warn-orphans #-}
7+
8+module Garnix.Types
9+ ( module Garnix.Types,
10+ module Garnix.Types.ExternalLenses,
11+ module Garnix.Types.Keys,
12+ )
13+where
14+
15+import Control.Lens
16+import Control.Lens.Regex.Text qualified as RE
17+import Control.Monad (mzero)
18+import Data.Aeson (defaultOptions, genericParseJSON, withObject)
19+import Data.Aeson qualified as Aeson
20+import Data.Aeson.Encode.Pretty (encodePretty)
21+import Data.Aeson.Encoding qualified as Aeson
22+import Data.Aeson.Key qualified as AesonKey
23+import Data.Aeson.KeyMap (toMapText)
24+import Data.Aeson.Types qualified as JSON
25+import Data.ByteString.Lazy qualified as BSL
26+import Data.Generics.Product (HasField' (..))
27+import Data.List.Extra (enumerate)
28+import Data.Map qualified as Map
29+import Data.Map.Strict qualified as StrictMap
30+import Data.Pool (Pool)
31+import Data.Text qualified as T
32+import Data.Text.Encoding (decodeUtf8)
33+import Database.PostgreSQL.Typed.Types (PGStringType)
34+import Garnix.Duration
35+import Garnix.Hosting.ServerPool.Types
36+import Garnix.MonetaryCost
37+import Garnix.Nix.Types qualified as Nix
38+import Garnix.Prelude
39+import Garnix.Types.ExternalLenses
40+import Garnix.Types.Keys
41+import GitHub.App.Auth (InstallationAuth)
42+import Network.HTTP.Types (Header, statusMessage)
43+import Network.Wreq qualified as Wreq
44+import Prettyprinter qualified as Pretty
45+import Servant qualified
46+import System.Log.FastLogger as FastLogger
47+import Prelude qualified
48+
49+-- * System
50+
51+data System
52+ = X8664Linux
53+ | X8664Darwin
54+ | AArch64Linux
55+ | AArch64Darwin
56+ | AArmV6LLinux
57+ | AArmV7LLinux
58+ | I686Linux
59+ | MIPSelLinux
60+ | OtherSystem Text
61+ deriving stock (Eq, Show, Generic, Ord)
62+
63+supportedSystems :: [System]
64+supportedSystems = [X8664Linux, AArch64Darwin, AArch64Linux, X8664Darwin, I686Linux]
65+
66+systemTextIso :: Iso' System Text
67+systemTextIso = iso from' to'
68+ where
69+ from' x = case x of
70+ X8664Linux -> "x86_64-linux"
71+ X8664Darwin -> "x86_64-darwin"
72+ AArch64Linux -> "aarch64-linux"
73+ AArch64Darwin -> "aarch64-darwin"
74+ AArmV6LLinux -> "armv6l-linux"
75+ AArmV7LLinux -> "armv7l-linux"
76+ I686Linux -> "i686-linux"
77+ MIPSelLinux -> "mipsel-linux"
78+ OtherSystem s -> s
79+ to' x = case x of
80+ "x86_64-linux" -> X8664Linux
81+ "x86_64-darwin" -> X8664Darwin
82+ "aarch64-linux" -> AArch64Linux
83+ "aarch64-darwin" -> AArch64Darwin
84+ "armv6l-linux" -> AArmV6LLinux
85+ "armv7l-linux" -> AArmV7LLinux
86+ "i686-linux" -> I686Linux
87+ "mipsel-linux" -> MIPSelLinux
88+ _ -> OtherSystem x
89+
90+instance Pretty System where
91+ pretty x = pretty $ x ^. systemTextIso
92+
93+instance ToJSON System where
94+ toJSON x = Aeson.String $ x ^. systemTextIso
95+
96+instance FromJSON System where
97+ parseJSON = Aeson.withText "system" $ \t -> pure $ t ^. from systemTextIso
98+
99+instance ToJSONKey System where
100+ toJSONKey = contramap (^. systemTextIso) Aeson.toJSONKey
101+
102+instance FromJSONKey System where
103+ fromJSONKey = fmap (^. from systemTextIso) Aeson.fromJSONKey
104+
105+-- | This is a 'Maybe System', but we use a special datatype because it
106+-- is not encoded as NULL in the DB (since it's part of the primary key), but
107+-- as 'noSystem'.
108+--
109+-- It is 'noSystem' for 'buildStarting'.
110+data MaybeSystem = IsSystem System | NoSystem
111+ deriving stock (Eq, Show, Ord, Generic)
112+
113+instance Pretty MaybeSystem where
114+ pretty = \case
115+ IsSystem s -> pretty s
116+ NoSystem -> "none"
117+
118+maybeSystemIso :: Iso' MaybeSystem (Maybe System)
119+maybeSystemIso = iso from' to'
120+ where
121+ from' x = case x of
122+ NoSystem -> Nothing
123+ IsSystem s -> Just s
124+ to' x = case x of
125+ Nothing -> NoSystem
126+ Just s -> IsSystem s
127+
128+instance ToJSON MaybeSystem where
129+ toJSON x = Aeson.toJSON $ x ^. maybeSystemIso
130+
131+instance FromJSON MaybeSystem where
132+ parseJSON x = (^. from maybeSystemIso) <$> Aeson.parseJSON x
133+
134+instance Servant.FromHttpApiData MaybeSystem where
135+ parseUrlPiece piece = case piece of
136+ "noSystem" -> pure NoSystem
137+ _ -> pure $ IsSystem $ piece ^. from systemTextIso
138+
139+instance PGType "system" where
140+ type PGVal "system" = MaybeSystem
141+
142+instance PGParameter "system" MaybeSystem where
143+ pgEncode _ msys = case msys of
144+ NoSystem -> "noSystem"
145+ IsSystem sys -> sys ^. systemTextIso . to cs
146+
147+instance PGColumn "system" MaybeSystem where
148+ pgDecode _ msys = case msys of
149+ "noSystem" -> NoSystem
150+ sys -> IsSystem $ sys ^. to cs . from systemTextIso
151+
152+instance PGType "citext" where
153+ type PGVal "citext" = Text
154+
155+instance PGStringType "citext"
156+
157+-- * Packages
158+
159+data SystemPackage = SystemPackage
160+ { systemPackageSystem :: System,
161+ systemPackagePackage :: Package
162+ }
163+ deriving stock (Eq, Show, Generic)
164+
165+newtype Package = Package {_packageName :: Text}
166+ deriving stock (Eq, Show, Generic)
167+
168+instance ToJSON Package where
169+ toEncoding = ourToEncoding
170+ toJSON = ourToJSON
171+
172+instance FromJSON Package where parseJSON = ourParseJSON
173+
174+newtype PackageName = PackageName {getPackageName :: Text}
175+ deriving stock (Eq, Show, Ord, Generic)
176+ deriving newtype
177+ ( ToJSON,
178+ FromJSON,
179+ ToJSONKey,
180+ FromJSONKey,
181+ Servant.FromHttpApiData,
182+ Servant.ToHttpApiData,
183+ PGParameter "character varying",
184+ PGParameter "text",
185+ PGColumn "character varying",
186+ PGColumn "text",
187+ Pretty,
188+ IsString
189+ )
190+
191+instance ConvertibleStrings Text PackageName where
192+ convertString = PackageName
193+
194+instance ConvertibleStrings PackageName Text where
195+ convertString = getPackageName
196+
197+instance Inject Text PackageName where inj = cs
198+
199+instance Project Text PackageName where prj = cs
200+
201+instance Isomorphic Text PackageName
202+
203+data PackageType
204+ = TypePackage
205+ | TypeCheck
206+ | TypeHomeConfiguration
207+ | TypeDarwinConfiguration
208+ | TypeNixosConfiguration
209+ | TypeDevShell
210+ | TypeDefaultDevShell
211+ | TypeDefaultPackage
212+ | TypeApp
213+ | TypeOverall
214+ deriving stock (Eq, Show, Enum, Bounded, Ord, Generic)
215+
216+instance PGType "package_type" where
217+ type PGVal "package_type" = PackageType
218+
219+asPackageType :: Prism' Text PackageType
220+asPackageType = prism there back
221+ where
222+ there i = case lookup i asPackageTypePairs of
223+ Just x -> x
224+ Nothing -> error $ "missing pattern: " <> show i
225+ back i = case lookup i (swap <$> asPackageTypePairs) of
226+ Just x -> Right x
227+ Nothing -> Left i
228+
229+asPackageTypePairs :: [(PackageType, Text)]
230+asPackageTypePairs = annotate <$> enumerate @PackageType
231+ where
232+ annotate :: PackageType -> (PackageType, Text)
233+ annotate = \case
234+ TypePackage -> (TypePackage, "package")
235+ TypeCheck -> (TypeCheck, "check")
236+ TypeHomeConfiguration -> (TypeHomeConfiguration, "homeConfiguration")
237+ TypeDarwinConfiguration -> (TypeDarwinConfiguration, "darwinConfiguration")
238+ TypeNixosConfiguration -> (TypeNixosConfiguration, "nixosConfiguration")
239+ TypeDevShell -> (TypeDevShell, "devShell")
240+ TypeDefaultDevShell -> (TypeDefaultDevShell, "defaultDevShell")
241+ TypeDefaultPackage -> (TypeDefaultPackage, "defaultPackage")
242+ TypeApp -> (TypeApp, "app")
243+ TypeOverall -> (TypeOverall, "overall")
244+
245+instance PGParameter "package_type" PackageType where
246+ pgEncode _ status = cs $ review asPackageType status
247+
248+instance PGColumn "package_type" PackageType where
249+ pgDecode _ status = case cs status ^? asPackageType of
250+ Just p -> p
251+ Nothing ->
252+ error
253+ $ "Expected one of: "
254+ <> show (snd <$> asPackageTypePairs)
255+ <> ", got: "
256+ <> cs status
257+
258+instance Servant.FromHttpApiData PackageType where
259+ parseUrlPiece piece = case piece ^? asPackageType of
260+ Just p -> Right p
261+ Nothing ->
262+ Left
263+ $ "Expected one of: "
264+ <> show (snd <$> asPackageTypePairs)
265+ <> ", got: "
266+ <> cs piece
267+
268+instance Pretty PackageType where
269+ pretty = pretty . review asPackageType
270+
271+instance ToJSON PackageType where
272+ toJSON = Aeson.toJSON . review asPackageType
273+
274+instance FromJSON PackageType where
275+ parseJSON = Aeson.withText "PackageType" $ \pt -> case pt ^? asPackageType of
276+ Just p -> pure p
277+ Nothing ->
278+ fail
279+ . cs
280+ $ "Expected one of: "
281+ <> show (snd <$> asPackageTypePairs)
282+ <> ", got: "
283+ <> cs pt
284+
285+-- | A fake package for reporting basic results such as whether the flake.nix
286+-- can be parsed.
287+buildStarting :: PackageName
288+buildStarting = PackageName "Build starting"
289+
290+newtype Packages a = Packages {getPackages :: Map.Map System (Map.Map PackageName a)}
291+ deriving stock (Eq, Show, Generic, Functor, Foldable, Traversable)
292+ deriving newtype (Semigroup, Monoid, ToJSON, FromJSON)
293+
294+data GhRun = GhRun
295+ { _ghRunName :: Text,
296+ _ghRunHeadSha :: CommitHash,
297+ _ghRunStatus :: Text,
298+ _ghRunOutput :: Maybe RunOutput,
299+ _ghRunDetailsUrl :: Maybe Text,
300+ _ghRunConclusion :: Maybe Text
301+ }
302+ deriving stock (Eq, Ord, Show, Generic)
303+
304+instance ToJSON GhRun where
305+ toEncoding = ourToEncoding
306+ toJSON = ourToJSON
307+
308+instance FromJSON GhRun where parseJSON = ourParseJSON
309+
310+data RunOutput = RunOutput
311+ { _runOutputTitle :: Text,
312+ _runOutputSummary :: Text,
313+ _runOutputText :: Text
314+ }
315+ deriving stock (Eq, Ord, Show, Generic)
316+
317+instance ToJSON RunOutput where
318+ toEncoding = ourToEncoding
319+ toJSON = ourToJSON
320+
321+instance FromJSON RunOutput where parseJSON = ourParseJSON
322+
323+-- * Repo
324+
325+data Repo = Repo
326+ { _repoReqUser :: UserId,
327+ _repoRepoUser :: GhRepoOwner,
328+ _repoRepoName :: GhRepoName,
329+ _repoEnabledAt :: UTCTime
330+ }
331+ deriving stock (Eq, Show, Generic)
332+
333+instance ToJSON Repo where
334+ toEncoding = ourToEncoding
335+ toJSON = ourToJSON
336+
337+instance FromJSON Repo where parseJSON = ourParseJSON
338+
339+data UserOverviewRepo = UserOverviewRepo
340+ { _userOverviewRepoRepoUser :: GhRepoOwner,
341+ _userOverviewRepoRepoName :: GhRepoName,
342+ _userOverviewRepoEnabled :: Bool
343+ }
344+ deriving stock (Eq, Show, Generic)
345+
346+instance ToJSON UserOverviewRepo where
347+ toEncoding = ourToEncoding
348+ toJSON = ourToJSON
349+
350+instance FromJSON UserOverviewRepo where parseJSON = ourParseJSON
351+
352+-- * Build
353+
354+newtype BuildId = BuildId {getBuildId :: HashId}
355+ deriving stock (Eq, Show, Generic)
356+ deriving newtype
357+ ( ToJSON,
358+ FromJSON,
359+ FromHttpApiData,
360+ ToHttpApiData,
361+ Pretty,
362+ PGColumn "bigint",
363+ PGParameter "bigint"
364+ )
365+
366+newtype RepoPublicity = RepoIsPublic {isRepoPublic :: Bool}
367+ deriving stock (Eq, Show, Generic)
368+ deriving newtype
369+ ( ToJSON,
370+ FromJSON,
371+ FromHttpApiData,
372+ ToHttpApiData,
373+ Pretty,
374+ PGColumn "boolean",
375+ PGParameter "boolean"
376+ )
377+
378+newtype BuildStreamId = BuildStreamId {getBuildStreamId :: HashId}
379+ deriving stock (Eq, Show, Generic)
380+ deriving newtype
381+ ( ToJSON,
382+ FromJSON,
383+ FromHttpApiData,
384+ ToHttpApiData,
385+ Pretty,
386+ PGColumn "bigint",
387+ PGParameter "bigint"
388+ )
389+
390+newtype BuildOutputsPgColumn = BuildOutputsPgColumn {buildOutputs :: Nix.BuildOutputs}
391+ deriving stock (Eq, Show, Generic)
392+
393+instance PGColumn "json" BuildOutputsPgColumn where
394+ pgDecode PGTypeProxy = fromMaybe (error "Failed to deserialize PGColumn BuildOutputsPgColumn") . Aeson.decodeStrict'
395+
396+instance PGParameter "json" BuildOutputsPgColumn where
397+ pgEncode PGTypeProxy = cs . Aeson.encode
398+
399+instance FromJSON BuildOutputsPgColumn where
400+ parseJSON = withObject "build outputs pg column" $ \o -> do
401+ outputs <- forM o $ \v -> do
402+ text :: Text <- parseJSON v
403+ case Nix.parseStorePath text of
404+ Left err -> JSON.parseFail $ cs err
405+ Right path -> pure path
406+ pure $ BuildOutputsPgColumn $ Nix.BuildOutputs $ toMapText outputs
407+
408+instance ToJSON BuildOutputsPgColumn where
409+ toJSON (BuildOutputsPgColumn (Nix.BuildOutputs map)) = toJSON $ Nix.getStorePath <$> map
410+
411+outputsForBuild :: Build -> Maybe Nix.BuildOutputs
412+outputsForBuild build = buildOutputs <$> _buildOutputPaths build
413+
414+data Build = Build
415+ { _buildId :: BuildId,
416+ _buildRepoUser :: GhRepoOwner,
417+ _buildRepoName :: GhRepoName,
418+ _buildPrFromFork :: Maybe PrFromFork,
419+ _buildBranch :: Maybe Branch,
420+ _buildRepoIsPublic :: RepoPublicity,
421+ _buildGitCommit :: CommitHash,
422+ _buildPackage :: PackageName,
423+ _buildPackageType :: PackageType,
424+ _buildSystem :: MaybeSystem,
425+ _buildReqUser :: GhLogin,
426+ _buildStatus :: Maybe Status,
427+ _buildStartTime :: UTCTime,
428+ _buildEndTime :: Maybe UTCTime,
429+ _buildDrvPath :: Maybe FilePath,
430+ _buildOutputPaths :: Maybe BuildOutputsPgColumn,
431+ _buildGithubRunId :: Maybe GhRunId,
432+ _buildPersistenceName :: Maybe Text,
433+ _buildWantsIncrementalism :: Bool,
434+ _buildEvalHost :: Maybe Text,
435+ _buildUploadedToCache :: Maybe Bool,
436+ _buildAlreadyBuilt :: Maybe Bool
437+ }
438+ deriving stock (Eq, Show, Generic)
439+
440+instance ToJSON Build where
441+ toEncoding = ourToEncoding
442+ toJSON = ourToJSON
443+
444+instance FromJSON Build where parseJSON = ourParseJSON
445+
446+instance Pretty Build where
447+ pretty s =
448+ "build:"
449+ <+> Pretty.line
450+ <+> Pretty.nest
451+ 2
452+ ( vsep
453+ [ "id:" <> pretty (_buildId s),
454+ "repoUser:" <+> pretty (_buildRepoUser s),
455+ "repoName:" <+> pretty (_buildRepoName s),
456+ "prFromFork:" <+> pretty (_buildPrFromFork s),
457+ "repoIsPublic:" <+> pretty (_buildRepoIsPublic s),
458+ "gitCommit:" <+> pretty (_buildGitCommit s),
459+ "package:" <+> pretty (_buildPackage s),
460+ "packageType:" <+> pretty (_buildPackageType s),
461+ "system:" <+> pretty (_buildSystem s),
462+ "reqUser:" <+> pretty (_buildReqUser s),
463+ "status:" <+> pretty (_buildStatus s),
464+ "startTime:" <+> pretty (show $ _buildStartTime s),
465+ "endTime:" <+> pretty (show $ _buildEndTime s),
466+ "drvPath:" <+> pretty (_buildDrvPath s),
467+ "githubRunId:" <+> pretty (_buildGithubRunId s)
468+ ]
469+ )
470+
471+buildComment :: Build -> Text
472+buildComment build =
473+ show (pretty (_buildId build))
474+ <> "_"
475+ <> show (pretty (_buildRepoUser build))
476+ <> "/"
477+ <> show (pretty (_buildRepoName build))
478+ <> "/"
479+ <> show (pretty (_buildBranch build))
480+
481+newtype LockedFile = LockedFile {getLockedFile :: FilePath}
482+
483+-- | Logs as nix gives us (prior to any filtering or processing)
484+newtype RawLogs = RawLogs {getRawLogs :: Text}
485+ deriving stock (Eq, Show, Generic)
486+ deriving newtype (IsString, Pretty, ConvertibleStrings String)
487+
488+data BuildResponse = BuildResponse
489+ { _buildResponseId :: BuildId,
490+ _buildResponseRepoUser :: GhRepoOwner,
491+ _buildResponseRepoName :: GhRepoName,
492+ _buildResponseGitCommit :: CommitHash,
493+ _buildResponsePackage :: PackageName,
494+ _buildResponsePackageType :: PackageType,
495+ _buildResponseSystem :: MaybeSystem,
496+ _buildResponseReqUser :: GhLogin,
497+ _buildResponseBranch :: Maybe Branch,
498+ _buildResponseStatus :: Maybe Status,
499+ _buildResponseStartTime :: UTCTime,
500+ _buildResponseEndTime :: Maybe UTCTime,
501+ _buildResponseGithubRunId :: Maybe GhRunId,
502+ _buildResponseOriginalBuild :: Maybe OriginalBuild,
503+ _buildResponseRelatedBuilds :: [OriginalBuild] -- deprecated
504+ }
505+ deriving (Eq, Show, Generic)
506+
507+instance ToJSON BuildResponse where
508+ toEncoding = ourToEncoding
509+ toJSON = ourToJSON
510+
511+instance FromJSON BuildResponse where parseJSON = ourParseJSON
512+
513+data OriginalBuild = OriginalBuild
514+ { _originalBuildId :: BuildId,
515+ _originalBuildGitCommit :: CommitHash,
516+ _originalBuildStatus :: Maybe Status
517+ }
518+ deriving (Eq, Show, Generic)
519+
520+instance ToJSON OriginalBuild where
521+ toEncoding = ourToEncoding
522+ toJSON = ourToJSON
523+
524+instance FromJSON OriginalBuild where parseJSON = ourParseJSON
525+
526+data BuildUpdate = BuildUpdate
527+ { _buildUpdateStatus :: Maybe Status
528+ }
529+ deriving (Eq, Show, Generic)
530+
531+instance ToJSON BuildUpdate where
532+ toEncoding = ourToEncoding
533+ toJSON = ourToJSON
534+
535+instance FromJSON BuildUpdate where parseJSON = ourParseJSON
536+
537+-- * Project
538+
539+data Status = Success | Failure | Timeout | Cancelled
540+ deriving stock (Eq, Ord, Show, Generic, Bounded, Enum)
541+ deriving anyclass (ToJSON, FromJSON)
542+
543+instance Pretty Status where
544+ pretty = \case
545+ Success -> "success"
546+ Failure -> "failure"
547+ Timeout -> "timeout"
548+ Cancelled -> "cancelled"
549+
550+instance PGType "build_status" where
551+ type PGVal "build_status" = Status
552+
553+instance PGParameter "build_status" Status where
554+ pgEncode _ status = case status of
555+ Success -> "success"
556+ Failure -> "failure"
557+ Timeout -> "timeout"
558+ Cancelled -> "cancelled"
559+
560+instance PGColumn "build_status" Status where
561+ pgDecode _ status = case status of
562+ "success" -> Success
563+ "failure" -> Failure
564+ "timeout" -> Timeout
565+ "cancelled" -> Cancelled
566+ e -> error $ "Impossible: expected 'success', 'failure', 'timeout' or 'cancelled', got: " <> cs e
567+
568+newtype Logs = Logs {getLogs :: Text}
569+ deriving stock (Eq, Show, Generic)
570+ deriving newtype (ToJSON, FromJSON)
571+
572+data SingleRunResult = SingleRunResult
573+ { _singleRunResultStatus :: Status,
574+ _singleRunResultLogs :: Maybe Logs
575+ }
576+ deriving stock (Eq, Show, Generic)
577+
578+instance ToJSON SingleRunResult where
579+ toEncoding = ourToEncoding
580+ toJSON = ourToJSON
581+
582+instance FromJSON SingleRunResult where parseJSON = ourParseJSON
583+
584+data RunResult = RunResult
585+ { _runResultStatus :: Status,
586+ _runResultPackages :: Map.Map System (Map.Map PackageName (Maybe SingleRunResult))
587+ }
588+ deriving stock (Eq, Show, Generic)
589+
590+instance ToJSON RunResult where
591+ toEncoding = ourToEncoding
592+ toJSON = ourToJSON
593+
594+instance FromJSON RunResult where parseJSON = ourParseJSON
595+
596+-- * Commit
597+
598+data CommitStatus = Evaluating | Evaluated
599+ deriving stock (Eq, Show, Enum, Bounded, Ord, Generic)
600+
601+instance PGType "commit_status" where
602+ type PGVal "commit_status" = CommitStatus
603+
604+asCommitStatus :: Prism' Text CommitStatus
605+asCommitStatus = prism there back
606+ where
607+ there i = case lookup i asCommitStatusPairs of
608+ Just x -> x
609+ Nothing -> error $ "missing pattern: " <> show i
610+ back i = case lookup i (swap <$> asCommitStatusPairs) of
611+ Just x -> Right x
612+ Nothing -> Left i
613+
614+asCommitStatusPairs :: [(CommitStatus, Text)]
615+asCommitStatusPairs =
616+ [ (Evaluating, "evaluating"),
617+ (Evaluated, "evaluated")
618+ ]
619+
620+instance PGParameter "commit_status" CommitStatus where
621+ pgEncode _ status = cs $ review asCommitStatus status
622+
623+instance PGColumn "commit_status" CommitStatus where
624+ pgDecode _ status = case cs status ^? asCommitStatus of
625+ Just p -> p
626+ Nothing ->
627+ error
628+ $ "Expected one of: "
629+ <> show (snd <$> asCommitStatusPairs)
630+ <> ", got: "
631+ <> cs status
632+
633+instance Pretty CommitStatus where
634+ pretty = pretty . review asCommitStatus
635+
636+data CheckStatus = CheckPending | CheckFail | CheckSuccess
637+ deriving stock (Eq, Show, Enum, Bounded, Ord, Generic)
638+
639+instance PGType "check_status" where
640+ type PGVal "check_status" = CheckStatus
641+
642+asCheckStatus :: Prism' Text CheckStatus
643+asCheckStatus = prism there back
644+ where
645+ there i = case lookup i asCheckStatusPairs of
646+ Just x -> x
647+ Nothing -> error $ "missing pattern: " <> show i
648+ back i = case lookup i (swap <$> asCheckStatusPairs) of
649+ Just x -> Right x
650+ Nothing -> Left i
651+
652+asCheckStatusPairs :: [(CheckStatus, Text)]
653+asCheckStatusPairs =
654+ [ (CheckPending, "pending"),
655+ (CheckFail, "fail"),
656+ (CheckSuccess, "success")
657+ ]
658+
659+instance PGParameter "check_status" CheckStatus where
660+ pgEncode _ status = cs $ review asCheckStatus status
661+
662+instance PGColumn "check_status" CheckStatus where
663+ pgDecode _ status = case cs status ^? asCheckStatus of
664+ Just p -> p
665+ Nothing ->
666+ error
667+ $ "Expected one of: "
668+ <> show (snd <$> asCheckStatusPairs)
669+ <> ", got: "
670+ <> cs status
671+
672+instance Pretty CheckStatus where
673+ pretty = pretty . review asCheckStatus
674+
675+data Commit = Commit
676+ { _commitRepoOwner :: GhRepoOwner,
677+ _commitRepoName :: GhRepoName,
678+ _commitHash :: CommitHash,
679+ _commitStatus :: CommitStatus,
680+ _commitMetaCheck :: CheckStatus
681+ }
682+ deriving stock (Eq, Show, Generic)
683+
684+data FullCommitState
685+ = CommitEvaluating
686+ | CommitEvaluated Commit [Build] [Run]
687+ deriving stock (Eq, Show, Generic)
688+
689+data Run = Run
690+ { _runId :: RunId,
691+ _runName :: Text,
692+ _runRepoUser :: GhRepoOwner,
693+ _runRepoName :: GhRepoName,
694+ _runGitCommit :: CommitHash,
695+ _runBranch :: Maybe Branch,
696+ _runStatus :: Maybe Status,
697+ _runReqUser :: GhLogin,
698+ _runStartTime :: UTCTime,
699+ _runEndTime :: Maybe UTCTime
700+ }
701+ deriving stock (Eq, Show, Generic)
702+
703+newtype RunId = RunId {getRunId :: HashId}
704+ deriving stock (Eq, Show, Generic)
705+ deriving newtype
706+ ( PGColumn "bigint",
707+ PGParameter "bigint",
708+ FromHttpApiData
709+ )
710+
711+-- * User
712+
713+--
714+newtype RequestingGhLogin = RequestingGhLogin {getRequestingGhLogin :: GhLogin}
715+ deriving stock (Eq, Show, Generic)
716+ deriving newtype
717+ ( ToJSON,
718+ FromJSON,
719+ PGParameter "character varying",
720+ PGParameter "text",
721+ PGColumn "character varying",
722+ PGColumn "text",
723+ IsString
724+ )
725+
726+newtype GhRepoOwner = GhRepoOwner {getGhRepoOwner :: GhLogin}
727+ deriving stock (Eq, Show, Generic)
728+ deriving newtype
729+ ( ToJSON,
730+ FromJSON,
731+ ToJSONKey,
732+ Ord,
733+ Servant.FromHttpApiData,
734+ Servant.ToHttpApiData,
735+ PGParameter "character varying",
736+ PGParameter "text",
737+ PGColumn "character varying",
738+ PGColumn "text",
739+ Pretty,
740+ IsString
741+ )
742+
743+newtype GhLogin = GhLogin {getGhLogin :: Text}
744+ deriving stock (Eq, Show, Generic)
745+ deriving newtype
746+ ( ToJSON,
747+ FromJSON,
748+ ToJSONKey,
749+ Ord,
750+ Servant.FromHttpApiData,
751+ Servant.ToHttpApiData,
752+ PGParameter "character varying",
753+ PGParameter "text",
754+ PGColumn "character varying",
755+ PGColumn "text",
756+ Pretty,
757+ IsString
758+ )
759+
760+newtype GhRepoName = GhRepoName {getGhRepoName :: Text}
761+ deriving stock (Eq, Show, Generic)
762+ deriving newtype
763+ ( ToJSON,
764+ FromJSON,
765+ Ord,
766+ Servant.FromHttpApiData,
767+ Servant.ToHttpApiData,
768+ PGParameter "character varying",
769+ PGParameter "text",
770+ PGColumn "character varying",
771+ PGColumn "text",
772+ Pretty,
773+ IsString
774+ )
775+
776+newtype InternalCacheToken = InternalCacheToken {getInternalCacheToken :: Text}
777+ deriving stock (Eq, Show, Generic)
778+
779+generateInternalCacheToken :: (MonadIO m) => m InternalCacheToken
780+generateInternalCacheToken = InternalCacheToken <$> randomBase64 64
781+
782+newtype NetRcFile = NetRcFile {getNetRcFile :: FilePath}
783+ deriving stock (Show)
784+
785+newtype NixConfig = NixConfig {getNixConfig :: StrictMap.Map String String}
786+ deriving stock (Show)
787+
788+instance Semigroup NixConfig where
789+ NixConfig left <> NixConfig right = NixConfig $ left <> right
790+
791+newtype PrFromFork = PrFromFork {getPrFromForkFullName :: Text}
792+ deriving stock (Eq, Show, Generic)
793+ deriving newtype
794+ ( ToJSON,
795+ FromJSON,
796+ Ord,
797+ Servant.FromHttpApiData,
798+ Servant.ToHttpApiData,
799+ PGParameter "character varying",
800+ PGParameter "text",
801+ PGColumn "character varying",
802+ PGColumn "text",
803+ Pretty,
804+ IsString
805+ )
806+
807+newtype Email = Email {getEmail :: Text}
808+ deriving stock (Eq, Ord, Show, Generic)
809+ deriving newtype
810+ ( ToJSON,
811+ FromJSON,
812+ PGParameter "character varying",
813+ PGParameter "text",
814+ PGColumn "character varying",
815+ PGColumn "text",
816+ Pretty,
817+ IsString
818+ )
819+
820+newtype CommitHash = CommitHash {getCommitHash :: Text}
821+ deriving stock (Eq, Show, Ord, Generic)
822+ deriving newtype
823+ ( ToJSON,
824+ FromJSON,
825+ Servant.FromHttpApiData,
826+ Servant.ToHttpApiData,
827+ PGParameter "character varying",
828+ PGParameter "text",
829+ PGColumn "character varying",
830+ PGColumn "text",
831+ Pretty,
832+ IsString
833+ )
834+
835+instance Inject Text CommitHash where
836+ inj = CommitHash
837+
838+instance Project Text CommitHash where
839+ prj = getCommitHash
840+
841+instance Isomorphic Text CommitHash
842+
843+newtype Branch = Branch {getBranch :: Text}
844+ deriving stock (Eq, Show, Ord, Generic)
845+ deriving newtype
846+ ( ToJSON,
847+ FromJSON,
848+ Servant.FromHttpApiData,
849+ Servant.ToHttpApiData,
850+ PGParameter "character varying",
851+ PGParameter "text",
852+ PGColumn "character varying",
853+ PGColumn "text",
854+ Pretty,
855+ IsString
856+ )
857+
858+instance ConvertibleStrings Text Branch where
859+ convertString = Branch
860+
861+instance ConvertibleStrings Branch Text where
862+ convertString = getBranch
863+
864+instance Inject Text Branch where inj = cs
865+
866+instance Project Text Branch where prj = cs
867+
868+instance Isomorphic Text Branch
869+
870+newtype GhPullRequestId = GhPullRequestId {getGhPullRequestId :: Int64}
871+ deriving stock (Eq, Show)
872+ deriving newtype
873+ ( Num,
874+ ToJSON,
875+ FromJSON,
876+ PGParameter "bigint",
877+ PGColumn "bigint"
878+ )
879+
880+instance (Inject a b, Functor f) => Inject (f a) (f b) where
881+ inj = fmap inj
882+
883+instance (Project a b, Functor f) => Project (f a) (f b) where
884+ prj = fmap prj
885+
886+instance (Inject a b, Project a b, Functor f) => Isomorphic (f a) (f b)
887+
888+newtype GhToken = GhToken {getGhToken :: Text}
889+ deriving stock (Eq, Show, Generic)
890+ deriving newtype (ToJSON, FromJSON, Servant.FromHttpApiData, Servant.ToHttpApiData)
891+
892+obfuscateGithubToken :: Text -> Text
893+obfuscateGithubToken =
894+ [RE.regex|gh[pousr]_\w{15,255}|] . RE.match .~ "XXXXXXXXXXXXXXXX"
895+
896+data CommitSummary = CommitSummary
897+ { _commitSummaryRepoOwner :: GhRepoOwner,
898+ _commitSummaryRepoName :: GhRepoName,
899+ _commitSummaryRepoIsPublic :: RepoPublicity,
900+ _commitSummaryGitCommit :: CommitHash,
901+ _commitSummaryBranch :: Maybe Branch,
902+ _commitSummaryReqUser :: GhLogin,
903+ _commitSummaryStartTime :: UTCTime,
904+ _commitSummarySucceeded :: Int64,
905+ _commitSummaryFailed :: Int64,
906+ _commitSummaryPending :: Int64,
907+ _commitSummaryCancelled :: Int64
908+ }
909+ deriving (Eq, Show, Generic)
910+
911+instance ToJSON CommitSummary where
912+ toEncoding = ourToEncoding
913+ toJSON = ourToJSON
914+
915+-- * Errors
916+
917+data ErrorWithContext = ErrorWithContext
918+ { callstack :: CallStack,
919+ spans :: [(Text, Text)],
920+ severity :: Severity,
921+ err :: Error
922+ }
923+ deriving stock (Show, Generic)
924+
925+errLens :: Lens' ErrorWithContext Error
926+errLens = lens err (\error err -> error {err})
927+
928+instance Eq ErrorWithContext where
929+ e1 == e2 = err e1 == err e2
930+
931+instance ToJSON ErrorWithContext where
932+ toJSON = Aeson.toJSON . err
933+
934+showDebug :: ErrorWithContext -> Text
935+showDebug e =
936+ message <> "\n\n" <> cs (prettyCallStack $ callstack e)
937+ where
938+ message = case err e of
939+ OtherError message -> "OtherError: " <> message
940+ inner -> showPretty inner
941+
942+data SubdomainKind
943+ = RepoOwnerSubdomain
944+ | RepoNameSubdomain
945+ | BranchSubdomain
946+ | PackageNameSubdomain
947+ | PersistenceNameSubdomain
948+ deriving stock (Eq, Show, Generic)
949+
950+instance Pretty SubdomainKind where
951+ pretty =
952+ \case
953+ RepoOwnerSubdomain -> "repo owner"
954+ RepoNameSubdomain -> "repo name"
955+ BranchSubdomain -> "branch"
956+ PackageNameSubdomain -> "package name"
957+ PersistenceNameSubdomain -> "persistence name"
958+
959+data Error
960+ = UncaughtRuntimeException {message :: Text}
961+ | RunProcessError
962+ { command :: Text,
963+ arguments :: [Text],
964+ stdErr :: Text,
965+ stdOut :: Text,
966+ exitCode :: Int
967+ }
968+ | DecodeError {original :: Text, message :: Text}
969+ | NoSuchBuild {buildId :: BuildId}
970+ | NoSuchBuildRunId {ghRunId :: GhRunId}
971+ | NoSuchRun {runId :: RunId}
972+ | NoSuchCommit CommitHash
973+ | NoSuchRepo {_owner :: GhRepoOwner, _name :: GhRepoName}
974+ | NoSuchUser GhLogin
975+ | ErrorGettingBuildPlan {message :: Text}
976+ | ErrorGettingAttributesToBuild {message :: Text}
977+ | IsDeniedAccess
978+ | DuplicateBuild
979+ | UserAlreadyExists GhLogin
980+ | DbError {statement :: Text, message :: Text}
981+ | ProvisioningError {message :: Text}
982+ | ActivationError {serverInfo :: ServerInfo, stdErr :: Text}
983+ | GithubDidntGiveUsAToken
984+ | RedirectFound {location :: Text}
985+ | BadRequest {message :: Text}
986+ | Unauthorized
987+ | UnauthorizedWithMessage {message :: Text}
988+ | Forbidden
989+ | ForbiddenWithMessage {message :: Text}
990+ | NotFound
991+ | OtherError Text
992+ | DecodeConfigError {message :: Text}
993+ | SshTimeout {command :: Text}
994+ | InvalidEmail
995+ | InvalidAccessToken
996+ | EntitlementError {message :: Text}
997+ | DevModeOnly
998+ | DeploymentWantsNixosConfigurationsThatDontExist [PackageName]
999+ | NameIsNotValidSubdomain SubdomainKind Text
1000+ | TransactionAlreadyStarted
1001+ | BuildAlreadyStopped {buildId :: BuildId}
1002+ | InvalidBuildUpdate {buildUpdateBody :: BuildUpdate}
1003+ | FailedToParseDrvFile {drvFile :: FilePath, message :: Text}
1004+ | CachedError {inner :: ErrorWithContext}
1005+ | GarnixAppUnauthorized GhRepoOwner GhRepoName
1006+ | GithubRequestTimeout
1007+ | FailedToParseCreateReportResult Aeson.Value
1008+ | ModuleErrorFlakeExists
1009+ | ActionExecutionTimeout
1010+ | ActionPreconditionNixStore Text
1011+ | ActionPreconditionFileExists {sandboxTypeStr :: Text}
1012+ | ActionSandboxTypeNotAllowed Text
1013+ | ActionKeyDecryptionFailure
1014+ | ActionEvaluationFailure Text
1015+ deriving stock (Eq, Show, Generic)
1016+
1017+instance Pretty Error where
1018+ pretty x = case x of
1019+ UncaughtRuntimeException {message} -> "runtime exception:" <+> pretty message
1020+ NoSuchUser user ->
1021+ "No user with github login"
1022+ <+> pretty (getGhLogin user)
1023+ <+> "could be found"
1024+ NoSuchBuild {} ->
1025+ "No build matching that description could be found."
1026+ <+> "Either it doesn't exist, or you don't have access to it."
1027+ NoSuchBuildRunId {..} ->
1028+ "No build with github run id"
1029+ <+> pretty ghRunId
1030+ <+> "found."
1031+ <+> "Either it doesn't exist, or you don't have access to it."
1032+ NoSuchRun {} ->
1033+ "No run matching that description could be found."
1034+ <+> "Either it doesn't exist, or you don't have access to it."
1035+ NoSuchCommit commit ->
1036+ "Commit"
1037+ <+> pretty commit
1038+ <+> "not found."
1039+ <+> "Either it doesn't exist, or you don't have access to it."
1040+ NoSuchRepo {..} ->
1041+ "The repo https://github.com/"
1042+ <> pretty (getGhLogin (getGhRepoOwner _owner))
1043+ <> "/"
1044+ <> pretty (getGhRepoName _name)
1045+ <> " doesn't exist, has not enabled garnix, or you don't have access to it."
1046+ IsDeniedAccess ->
1047+ "This request has been denied. Anomalous activity detected. Please contact contact@garnix.io for more information."
1048+ UserAlreadyExists user ->
1049+ "User with github login"
1050+ <+> pretty (getGhLogin user)
1051+ <+> "already exists"
1052+ GithubDidntGiveUsAToken -> "Github didn't give us a user token"
1053+ DecodeError {..} ->
1054+ "Error decoding ("
1055+ <> pretty message
1056+ <> "):"
1057+ <+> pretty (limitMessage original)
1058+ where
1059+ limit = 10000
1060+ limitMessage text =
1061+ if T.length text > limit
1062+ then T.take limit text <> "[...snip]"
1063+ else text
1064+ DuplicateBuild -> "Build already exists"
1065+ RedirectFound {location} -> "Redirect error to " <> pretty location
1066+ BadRequest {message} -> "Bad Request: " <> pretty message
1067+ Unauthorized -> "You need to login to continue"
1068+ UnauthorizedWithMessage {message} -> "Unauthorized: " <> pretty message
1069+ Forbidden -> "You are not authorized for this resource"
1070+ ForbiddenWithMessage {message} -> "Forbidden: " <> pretty message
1071+ NotFound -> "Resource not found"
1072+ DbError {} -> "Something went wrong"
1073+ ErrorGettingBuildPlan {..} -> "Couldn't get build plan. Error was: " <> pretty message
1074+ ErrorGettingAttributesToBuild {..} -> "Couldn't get attributes to build. Error was: " <> pretty message
1075+ ProvisioningError {..} ->
1076+ "Error provisioning server:" <+> pretty message
1077+ ActivationError {serverInfo, stdErr} ->
1078+ pretty
1079+ $ T.unlines
1080+ [ "Failed to activate server",
1081+ "You may be able to debug this by sshing into "
1082+ <> cs (_serverInfoIpv4Addr serverInfo)
1083+ <> " or "
1084+ <> cs (_serverInfoIpv6Addr serverInfo),
1085+ "Stderr:",
1086+ stdErr
1087+ ]
1088+ SshTimeout {command} ->
1089+ pretty $ "ssh command timed out: " <> command
1090+ OtherError message -> pretty message
1091+ DecodeConfigError {..} ->
1092+ "Error decoding garnix.yaml:"
1093+ <+> pretty message
1094+ RunProcessError {..} ->
1095+ pretty
1096+ $ obfuscateGithubToken
1097+ $ show
1098+ ( "Command"
1099+ <+> Pretty.squotes (pretty (command <> " " <> T.unwords arguments))
1100+ <+> "failed with exit code"
1101+ <+> pretty (show exitCode)
1102+ <> "."
1103+ <> Pretty.softline
1104+ <> "Standard err was:"
1105+ <> Pretty.line
1106+ <> pretty stdErr
1107+ )
1108+ InvalidEmail -> "Invalid email address"
1109+ InvalidAccessToken -> "Invalid access token"
1110+ EntitlementError {message} ->
1111+ pretty message
1112+ DevModeOnly -> "dev mode only"
1113+ DeploymentWantsNixosConfigurationsThatDontExist packages -> "Deployment wants package(s) that have not been built: " <> pretty packages
1114+ NameIsNotValidSubdomain kind name -> pretty kind <+> "name" <+> Pretty.squotes (pretty name) <+> "is not a valid subdomain name."
1115+ TransactionAlreadyStarted -> "Internal transaction error."
1116+ BuildAlreadyStopped {buildId} -> "Build with id" <+> pretty buildId <+> "has already been stopped."
1117+ InvalidBuildUpdate {buildUpdateBody} -> "Invalid build update:" <+> pretty (decodeUtf8 $ BSL.toStrict (encodePretty buildUpdateBody))
1118+ FailedToParseDrvFile {drvFile, message} -> "Failed to parse " <+> pretty drvFile <+> ": " <+> pretty message
1119+ CachedError inner -> "(cached error)" <+> pretty (err inner)
1120+ GarnixAppUnauthorized owner repo -> Pretty.squotes (pretty owner) <+> "uninstalled or did not give enough permissions to the garnix app for" <+> Pretty.squotes (pretty repo)
1121+ GithubRequestTimeout -> "Request timeout when talking with Github."
1122+ FailedToParseCreateReportResult value -> "Failed to parse create report response: " <> Pretty.squotes (pretty $ show value)
1123+ ModuleErrorFlakeExists -> "This repo already has a flake.nix file. You should be able to use garnix without using modules."
1124+ ActionExecutionTimeout -> "The action took too long to complete and it was cancelled."
1125+ ActionPreconditionNixStore p -> "The action's 'program' needs to be a (path) from a derivation. Program is: " <> Pretty.squotes (pretty p)
1126+ ActionPreconditionFileExists p -> "The action's 'program' is wrong. Please make sure the path exists and that you're not trying to pass arguments via the 'program' field. Program is: " <> Pretty.squotes (pretty p)
1127+ ActionSandboxTypeNotAllowed typ -> "You are not allowed to run actions with the '" <> pretty typ <> "'. If you want access, get in touch with us."
1128+ ActionEvaluationFailure t -> "Evaluation for the action failed: " <> Pretty.line <> pretty t
1129+ ActionKeyDecryptionFailure -> "Could not decrypt private key."
1130+
1131+instance ToJSON Error where
1132+ toJSON x =
1133+ Aeson.object
1134+ $ [ "status" Aeson..= ("error" :: Text),
1135+ "message" Aeson..= msg
1136+ ]
1137+ ++ additionalFields
1138+ where
1139+ msg = showPretty x
1140+ additionalFields = case x of
1141+ NoSuchUser user -> ["garnixUser" Aeson..= user]
1142+ _ -> []
1143+
1144+servantizeError :: ErrorWithContext -> Servant.ServerError
1145+servantizeError e =
1146+ let details = toErrorDetails e
1147+ in Servant.ServerError
1148+ (statusCode details)
1149+ (cs $ statusMessage $ toEnum $ statusCode details)
1150+ (cs $ userMessage details)
1151+ (headers details)
1152+
1153+-- Unsafe constructor, use `errorDetails` instead!
1154+data ErrorDetails = UnsafeErrorDetails
1155+ { statusCode :: Int,
1156+ headers :: [Network.HTTP.Types.Header],
1157+ -- | An error message that is safe to display to the user
1158+ userMessage :: Text
1159+ }
1160+
1161+errorDetails :: Int -> Text -> ErrorDetails
1162+errorDetails statusCode userMessage = UnsafeErrorDetails statusCode [] (obfuscateGithubToken userMessage)
1163+
1164+fromStatusCode :: Int -> ErrorDetails
1165+fromStatusCode statusCode = errorDetails statusCode $ cs $ statusMessage $ toEnum statusCode
1166+
1167+toErrorDetails :: ErrorWithContext -> ErrorDetails
1168+toErrorDetails e = case err e of
1169+ UncaughtRuntimeException {} -> errorDetails 500 "Something went wrong"
1170+ RedirectFound {location} -> (fromStatusCode 302) {headers = [("location", cs location)]}
1171+ BadRequest {message} -> errorDetails 400 $ "Bad Request: " <> message
1172+ Unauthorized -> fromStatusCode 401
1173+ UnauthorizedWithMessage {message} -> errorDetails 401 $ "Unauthorized: " <> message
1174+ Forbidden -> fromStatusCode 403
1175+ ForbiddenWithMessage {message} -> errorDetails 403 $ "Forbidden: " <> message
1176+ NotFound -> fromStatusCode 404
1177+ e'@DecodeConfigError {} -> errorDetails 400 $ cs $ Aeson.encode e'
1178+ e'@DuplicateBuild -> errorDetails 409 $ cs $ Aeson.encode e'
1179+ e'@NoSuchBuild {} -> errorDetails 404 $ cs $ Aeson.encode e'
1180+ e'@NoSuchBuildRunId {} -> errorDetails 404 $ cs $ Aeson.encode e'
1181+ e'@NoSuchRun {} -> errorDetails 404 $ cs $ Aeson.encode e'
1182+ e'@NoSuchCommit {} -> errorDetails 404 $ cs $ Aeson.encode e'
1183+ e'@NoSuchUser {} -> errorDetails 404 $ cs $ Aeson.encode e'
1184+ e'@IsDeniedAccess -> errorDetails 403 $ cs $ Aeson.encode e'
1185+ e'@UserAlreadyExists {} -> errorDetails 409 $ cs $ Aeson.encode e'
1186+ e'@GithubDidntGiveUsAToken {} -> errorDetails 500 $ cs $ Aeson.encode e'
1187+ e'@InvalidEmail -> errorDetails 400 $ cs $ Aeson.encode e'
1188+ e'@InvalidAccessToken -> errorDetails 401 $ cs $ Aeson.encode e'
1189+ e'@RunProcessError {} ->
1190+ errorDetails 400
1191+ $ command e'
1192+ <> " "
1193+ <> T.unwords (arguments e')
1194+ <> " failed with exit code "
1195+ <> show (exitCode e')
1196+ <> "\nStderr:\n"
1197+ <> stdErr e'
1198+ DecodeError _ _ ->
1199+ errorDetails 500 "Decode error"
1200+ DbError _ _ ->
1201+ fromStatusCode 500
1202+ e@(ErrorGettingBuildPlan _) ->
1203+ errorDetails 400 $ showPretty e
1204+ e@(ErrorGettingAttributesToBuild _) ->
1205+ errorDetails 400 $ showPretty e
1206+ ProvisioningError _errMsg ->
1207+ errorDetails 500 "Error provisioning server. This could be a temporary error."
1208+ ActivationError serverInfo stdErr ->
1209+ errorDetails 400 $ "Error activating server at " <> _serverInfoIpv4Addr serverInfo <> ": " <> stdErr
1210+ SshTimeout {command} ->
1211+ errorDetails 500 $ "Timeout: ssh command " <> command
1212+ OtherError errMsg ->
1213+ errorDetails 500 errMsg
1214+ EntitlementError message ->
1215+ errorDetails 403 message
1216+ DevModeOnly -> fromStatusCode 404
1217+ DeploymentWantsNixosConfigurationsThatDontExist packages ->
1218+ errorDetails 400
1219+ $ "NixOS configuration(s) not found: "
1220+ <> T.intercalate ", " (map getPackageName packages)
1221+ NameIsNotValidSubdomain kind name ->
1222+ errorDetails 400 $ show kind <> " name '" <> name <> "' is not a valid subdomain name."
1223+ TransactionAlreadyStarted ->
1224+ errorDetails 500 "Internal transaction error"
1225+ BuildAlreadyStopped {} ->
1226+ errorDetails 400 "Build has already been stopped."
1227+ InvalidBuildUpdate {} ->
1228+ errorDetails 500 "Invalid build update."
1229+ NoSuchRepo {..} ->
1230+ errorDetails 404
1231+ $ "The repo https://github.com/"
1232+ <> getGhLogin (getGhRepoOwner _owner)
1233+ <> "/"
1234+ <> getGhRepoName _name
1235+ <> " doesn't exist, has not enabled garnix, or you don't have access to it."
1236+ FailedToParseDrvFile {drvFile, message} -> errorDetails 500 $ "Failed to parse drv file " <> cs drvFile <> ": " <> message
1237+ CachedError inner ->
1238+ let details = toErrorDetails inner
1239+ in details
1240+ { userMessage = "(cached error) " <> userMessage details
1241+ }
1242+ GarnixAppUnauthorized (GhRepoOwner (GhLogin owner)) (GhRepoName repo) ->
1243+ errorDetails 403 $ "The Garnix application does not have enough rights or was removed from '" <> show owner <> "/" <> show repo <> "'"
1244+ GithubRequestTimeout -> errorDetails 503 "Request timeout when talking with Github."
1245+ FailedToParseCreateReportResult _ -> errorDetails 500 "Internal error when creating report."
1246+ ModuleErrorFlakeExists -> errorDetails 400 $ show $ pretty $ err e
1247+ ActionExecutionTimeout -> errorDetails 400 $ show $ pretty $ err e
1248+ ActionPreconditionNixStore _ -> errorDetails 400 $ show $ pretty $ err e
1249+ ActionPreconditionFileExists _ -> errorDetails 400 $ show $ pretty $ err e
1250+ ActionSandboxTypeNotAllowed _ -> errorDetails 403 $ show $ pretty $ err e
1251+ ActionEvaluationFailure _ -> errorDetails 400 $ show $ pretty $ err e
1252+ ActionKeyDecryptionFailure -> errorDetails 400 $ show $ pretty $ err e
1253+
1254+data SubscriptionType
1255+ = FreeSubscription
1256+ | Admin
1257+ deriving stock (Eq, Show, Generic)
1258+
1259+instance ToJSON SubscriptionType where
1260+ toJSON = \case
1261+ FreeSubscription -> "free"
1262+ Admin -> "admin"
1263+
1264+instance FromJSON SubscriptionType where
1265+ parseJSON = Aeson.withText "subscription_type" $ \case
1266+ "free" -> pure FreeSubscription
1267+ "admin" -> pure Admin
1268+ _ -> mzero
1269+
1270+instance PGType "subscription_type" where
1271+ type PGVal "subscription_type" = SubscriptionType
1272+
1273+instance PGParameter "subscription_type" SubscriptionType where
1274+ pgEncode _ status = case status of
1275+ FreeSubscription -> "free"
1276+ Admin -> "admin"
1277+
1278+instance PGColumn "subscription_type" SubscriptionType where
1279+ pgDecode _ status = case status of
1280+ "free" -> FreeSubscription
1281+ "admin" -> Admin
1282+ e -> error $ "Impossible: unknown subscription type " <> cs e
1283+
1284+newtype UserId = UserId {getUserId :: Int32}
1285+ deriving stock (Eq, Show, Generic)
1286+ deriving newtype (ToJSON, Ord, FromJSON, PGColumn "integer", PGParameter "integer")
1287+
1288+data User = User
1289+ { _userId :: UserId,
1290+ _userGithubLogin :: GhLogin,
1291+ _userEmail :: Email,
1292+ _userSubscriptionType :: SubscriptionType,
1293+ _userCreatedAt :: UTCTime
1294+ }
1295+ deriving stock (Eq, Show, Generic)
1296+
1297+instance ToJSON User where
1298+ toEncoding = ourToEncoding
1299+ toJSON = ourToJSON
1300+
1301+instance FromJSON User where parseJSON = ourParseJSON
1302+
1303+data AuthJwtPayload = WebSession User GhToken | ApiSession User
1304+ deriving stock (Eq, Show, Generic)
1305+ deriving anyclass (ToJWT, FromJWT)
1306+
1307+instance {-# OVERLAPS #-} HasField' "user" AuthJwtPayload User where
1308+ field' :: Lens' AuthJwtPayload User
1309+ field' =
1310+ lens
1311+ ( \case
1312+ WebSession user _ -> user
1313+ ApiSession user -> user
1314+ )
1315+ ( \payload user -> case payload of
1316+ WebSession _user tok -> WebSession user tok
1317+ ApiSession _user -> ApiSession user
1318+ )
1319+
1320+instance ToJSON AuthJwtPayload where
1321+ toJSON (WebSession user ghToken) =
1322+ JSON.Object
1323+ $ ("id" Aeson..= _userId user)
1324+ <> ("github_login" Aeson..= _userGithubLogin user)
1325+ <> ("email" Aeson..= _userEmail user)
1326+ <> ("subscription_type" Aeson..= _userSubscriptionType user)
1327+ <> ("created_at" Aeson..= _userCreatedAt user)
1328+ <> ("github_token" Aeson..= ghToken)
1329+ toJSON (ApiSession user) =
1330+ JSON.Object
1331+ $ ("id" Aeson..= _userId user)
1332+ <> ("github_login" Aeson..= _userGithubLogin user)
1333+ <> ("email" Aeson..= _userEmail user)
1334+ <> ("subscription_type" Aeson..= _userSubscriptionType user)
1335+ <> ("created_at" Aeson..= _userCreatedAt user)
1336+
1337+instance FromJSON AuthJwtPayload where
1338+ parseJSON = withObject "AuthJwtPayload" $ \obj -> do
1339+ user <- parseJSON (Aeson.Object obj)
1340+ token <- obj Aeson..:? "github_token"
1341+ pure $ maybe (ApiSession user) (WebSession user) token
1342+
1343+newtype CustomerId = CustomerId {getCustomerId :: Text}
1344+ deriving stock (Generic, Show, Eq)
1345+ deriving newtype (ToJSON, FromJSON, Wreq.FormValue)
1346+
1347+newtype InvoiceId = InvoiceId {getInvoiceId :: Text}
1348+ deriving stock (Generic, Show, Eq)
1349+ deriving newtype (ToJSON, FromJSON, Wreq.FormValue)
1350+
1351+newtype SubscriptionId = SubscriptionId {getSubscriptionId :: Text}
1352+ deriving stock (Generic, Show, Eq, Ord)
1353+ deriving newtype (ToJSON, FromJSON, Wreq.FormValue)
1354+
1355+data InstallationStatus
1356+ = NoActiveInstallation
1357+ | InstallationRenewing UTCTime
1358+ | InstallationCancelling UTCTime
1359+ deriving stock (Eq, Show, Generic)
1360+
1361+instance ToJSON InstallationStatus where
1362+ toEncoding = ourToEncoding
1363+ toJSON = ourToJSON
1364+
1365+data CreatingUser a = CreatingUser
1366+ { _creatingUserExists :: Bool,
1367+ _creatingUserGithubLogin :: GhLogin,
1368+ _creatingUserEmail :: Email,
1369+ _creatingUserGithubToken :: a
1370+ }
1371+ deriving stock (Eq, Show, Generic, Functor)
1372+ deriving anyclass (ToJWT, FromJWT)
1373+
1374+instance (ToJSON a) => ToJSON (CreatingUser a) where
1375+ toEncoding = ourToEncoding
1376+ toJSON = ourToJSON
1377+
1378+instance (FromJSON a) => FromJSON (CreatingUser a) where parseJSON = ourParseJSON
1379+
1380+data CreateUser = CreateUser
1381+ { _createUserEmail :: Email,
1382+ _createUserSubscriptionType :: SubscriptionType,
1383+ _createUserAgreeToEmails :: Bool
1384+ }
1385+ deriving stock (Eq, Show, Generic)
1386+ deriving anyclass (ToJWT, FromJWT)
1387+
1388+instance ToJSON CreateUser where
1389+ toEncoding = ourToEncoding
1390+ toJSON = ourToJSON
1391+
1392+instance FromJSON CreateUser where parseJSON = ourParseJSON
1393+
1394+data LoginLinks = LoginLinks
1395+ {_loginLinksGithub :: T.Text}
1396+ deriving stock (Eq, Show, Generic)
1397+
1398+instance ToJSON LoginLinks where
1399+ toEncoding = ourToEncoding
1400+ toJSON = ourToJSON
1401+
1402+instance FromJSON LoginLinks where parseJSON = ourParseJSON
1403+
1404+data SignupLinks = SignupLinks
1405+ { _signupLinksGithub :: T.Text
1406+ }
1407+ deriving stock (Eq, Show, Generic)
1408+
1409+instance ToJSON SignupLinks where
1410+ toEncoding = ourToEncoding
1411+ toJSON = ourToJSON
1412+
1413+instance FromJSON SignupLinks where parseJSON = ourParseJSON
1414+
1415+data UserOverview = UserOverview
1416+ { _userOverviewRepos :: [UserOverviewRepo]
1417+ }
1418+ deriving stock (Eq, Show, Generic)
1419+
1420+instance ToJSON UserOverview where
1421+ toEncoding = ourToEncoding
1422+ toJSON = ourToJSON
1423+
1424+instance FromJSON UserOverview where parseJSON = ourParseJSON
1425+
1426+newtype OAuthCode = OAuthCode {getOAuthCode :: T.Text}
1427+ deriving stock (Eq, Show, Generic)
1428+ deriving newtype (ToJSON, FromJSON, Servant.FromHttpApiData, Servant.ToHttpApiData)
1429+
1430+-- | Yes this really must be at least Int64
1431+newtype GhRunId = GhRunId {getGhRunId :: Int64}
1432+ deriving stock (Eq, Generic)
1433+ deriving newtype
1434+ ( ToJSON,
1435+ FromJSON,
1436+ Num,
1437+ Ord,
1438+ Show,
1439+ PGParameter "bigint",
1440+ PGColumn "bigint",
1441+ Pretty
1442+ )
1443+
1444+data DoesFileExist = FileExists | FileDoesntExist
1445+ deriving (Eq, Show)
1446+
1447+newtype PullRequestResult = PullRequestResult
1448+ { _pullRequestResultUrl :: Text
1449+ }
1450+ deriving stock (Generic)
1451+
1452+instance ToJSON PullRequestResult where
1453+ toEncoding = ourToEncoding
1454+ toJSON = ourToJSON
1455+
1456+-- * Hosting Types
1457+
1458+newtype ServerId = ServerId {getServerId :: HashId}
1459+ deriving stock (Eq, Show, Generic)
1460+ deriving newtype
1461+ ( ToJSON,
1462+ FromJSON,
1463+ FromHttpApiData,
1464+ ToHttpApiData,
1465+ PGColumn "bigint",
1466+ PGParameter "bigint"
1467+ )
1468+
1469+instance Pretty ServerId where
1470+ pretty = pretty . getHashId . getServerId
1471+
1472+newtype PreprovisionedServerId = PreprovisionedServerId {getPreprovisionedServerId :: Int64}
1473+ deriving stock (Eq, Show, Generic)
1474+ deriving newtype
1475+ ( ToJSON,
1476+ FromJSON,
1477+ FromHttpApiData,
1478+ ToHttpApiData,
1479+ PGColumn "bigint",
1480+ PGParameter "bigint"
1481+ )
1482+
1483+instance Pretty PreprovisionedServerId where
1484+ pretty = pretty . getPreprovisionedServerId
1485+
1486+newtype HetznerServerId = HetznerServerId {getHetznerServerId :: Int32}
1487+ deriving stock (Eq, Show, Ord, Generic)
1488+ deriving newtype
1489+ ( ToJSON,
1490+ FromJSON,
1491+ FromHttpApiData,
1492+ ToHttpApiData,
1493+ PGColumn "integer",
1494+ PGParameter "integer",
1495+ Enum
1496+ )
1497+
1498+instance Pretty HetznerServerId where
1499+ pretty = pretty . show . getHetznerServerId
1500+
1501+data ServerInfo = ServerInfo
1502+ { _serverInfoId :: ServerId,
1503+ _serverInfoHetznerServerId :: HetznerServerId,
1504+ _serverInfoIpv4Addr :: Text,
1505+ _serverInfoIpv6Addr :: Text,
1506+ _serverInfoCreatedAt :: UTCTime,
1507+ _serverInfoEndedAt :: Maybe UTCTime,
1508+ _serverInfoConfigurationBuildId :: BuildId,
1509+ _serverInfoPullRequest :: Maybe GhPullRequestId,
1510+ _serverInfoReadyAt :: Maybe UTCTime,
1511+ _serverInfoBuildPersistenceName :: Maybe Text,
1512+ _serverInfoTier :: ServerTier,
1513+ _serverInfoIsPrimary :: Bool
1514+ }
1515+ deriving stock (Eq, Show, Generic)
1516+
1517+instance Pretty ServerInfo where
1518+ pretty s =
1519+ "server:"
1520+ <+> Pretty.line
1521+ <+> Pretty.nest
1522+ 2
1523+ ( vsep
1524+ [ "id:" <+> pretty (_serverInfoId s),
1525+ "hetzner id:" <+> pretty (_serverInfoHetznerServerId s),
1526+ "ipv4:" <+> pretty (_serverInfoIpv4Addr s),
1527+ "ipv6:" <+> pretty (_serverInfoIpv6Addr s),
1528+ "created at:" <+> pretty (show $ _serverInfoCreatedAt s),
1529+ "initialized at:" <+> pretty (show $ _serverInfoReadyAt s),
1530+ "ended at:" <+> pretty (show $ _serverInfoEndedAt s)
1531+ ]
1532+ )
1533+
1534+data DeploymentType
1535+ = BranchDeployment Branch
1536+ | GhPrDeployment GhPullRequestId
1537+ deriving stock (Eq, Show, Generic)
1538+
1539+instance ToJSON DeploymentType where
1540+ toJSON = ourToJSON
1541+
1542+fromDeploymentType :: (Branch -> a) -> (GhPullRequestId -> a) -> DeploymentType -> a
1543+fromDeploymentType a b = \case
1544+ BranchDeployment branch -> a branch
1545+ GhPrDeployment prId -> b prId
1546+
1547+ghPrDeployment :: DeploymentType -> Maybe GhPullRequestId
1548+ghPrDeployment = fromDeploymentType (const Nothing) Just
1549+
1550+data PreprovisionedServer = PreprovisionedServer
1551+ { _preprovisionedServerId :: PreprovisionedServerId,
1552+ _preprovisionedServerHetznerServerId :: HetznerServerId,
1553+ _preprovisionedServerIpv4Addr :: Text,
1554+ _preprovisionedServerIpv6Addr :: Text,
1555+ _preprovisionedServerCreatedAt :: UTCTime,
1556+ _preprovisionedServerReadyAt :: Maybe UTCTime
1557+ }
1558+ deriving stock (Eq, Show, Generic)
1559+
1560+-- * DeployPlan
1561+
1562+data ServerToSpinUp = ServerToSpinUp
1563+ { serverTier :: ServerTier,
1564+ build :: Build,
1565+ domainIsPrimary :: Bool
1566+ }
1567+ deriving stock (Show, Eq, Generic)
1568+
1569+data DeployPlan = DeployPlan
1570+ { toSpinDown :: [ServerInfo],
1571+ toSpinUp :: [ServerToSpinUp],
1572+ toRedeploy :: [(ServerInfo, Build)]
1573+ }
1574+ deriving stock (Show, Eq, Generic)
1575+
1576+-- * FrontendConfig
1577+
1578+data FrontendConfig = FrontendConfig
1579+ { _frontendConfigGithubAppName :: Text
1580+ }
1581+ deriving stock (Eq, Show, Generic)
1582+
1583+instance ToJSON FrontendConfig where
1584+ toEncoding = ourToEncoding
1585+ toJSON = ourToJSON
1586+
1587+-- * Log
1588+
1589+data LogItem = LogItem
1590+ { severity :: Severity,
1591+ context :: [(Text, Text)],
1592+ msg :: Text
1593+ }
1594+ deriving stock (Show, Eq, Generic)
1595+
1596+instance FastLogger.ToLogStr LogItem where
1597+ toLogStr logItem =
1598+ toLogStr
1599+ $ Aeson.encodingToLazyByteString
1600+ $ Aeson.pairs
1601+ ( "logLevel"
1602+ Aeson..= show (logItem ^. #severity)
1603+ <> foldMap (\(a, b) -> AesonKey.fromText a Aeson..= b) (context logItem)
1604+ <> ("message" Aeson..= msg logItem)
1605+ )
1606+
1607+data Severity
1608+ = -- | Critical conditions, alert the entire dev team. An action is likely required.
1609+ -- | Examples: system down, database down, invariant does not hold, etc.
1610+ Critical
1611+ | -- | Error conditions, notify the alerts channel. Action may or may not be required.
1612+ -- | Examples: Any sort of unexpected error case that does not fit into `Critical`.
1613+ Error
1614+ | -- | Expected errors, user errors, etc.
1615+ -- | Examples: eval errors, build errors, build timeouts, etc.
1616+ Warning
1617+ | -- | Key events that we may want to follow for a specific build.
1618+ -- | Examples: pr/branch webhook received, build started, etc.
1619+ Notice
1620+ | -- | Information that is useful to log for users/customer support.
1621+ -- | Examples: Individual package build start/completion, detailed deployment events, debugging info, etc.
1622+ Informational
1623+ deriving (Eq, Enum, Bounded, Read, Show, Ord)
1624+
1625+newtype PrHostList = PrHostList
1626+ { prHostList :: [Host]
1627+ }
1628+ deriving stock (Eq, Show, Generic)
1629+
1630+hostToDomainName :: Host -> Text
1631+hostToDomainName host =
1632+ getPackageName (_hostPackageName host)
1633+ <> "."
1634+ <> maybe (getBranch (_hostBranch host)) (("pull-" <>) . show . getGhPullRequestId) (_hostPullRequest host)
1635+ <> "."
1636+ <> getGhRepoName (_hostRepoName host)
1637+ <> "."
1638+ <> getGhLogin (getGhRepoOwner (_hostRepoOwner host))
1639+
1640+data Host = Host
1641+ { _hostRepoOwner :: GhRepoOwner,
1642+ _hostRepoName :: GhRepoName,
1643+ _hostBranch :: Branch,
1644+ _hostPackageName :: PackageName,
1645+ _hostPullRequest :: Maybe GhPullRequestId,
1646+ _hostIpV4Addr :: Text,
1647+ _hostIpV6Addr :: Text,
1648+ _hostDrvPath :: Maybe FilePath,
1649+ _hostPersistenceName :: Maybe Text,
1650+ _hostServerId :: ServerId,
1651+ _hostHetznerId :: HetznerServerId,
1652+ _hostIsPrimary :: Bool
1653+ }
1654+ deriving stock (Eq, Show, Generic)
1655+
1656+newtype Candidate a = Candidate {promoteCandidate :: a}
1657+
1658+-- * opensearch
1659+
1660+data OpenSearchId
1661+ = FromRun RunId
1662+ | FromBuild BuildId
1663+ deriving stock (Eq, Show, Generic)
1664+
1665+data OpenSearchMessage = OpenSearchMessage
1666+ { _openSearchMessageTimestamp :: UTCTime,
1667+ _openSearchMessagePackage :: Maybe PackageName,
1668+ _openSearchMessagePhase :: Maybe Text,
1669+ _openSearchMessageLogMessage :: Text
1670+ }
1671+ deriving stock (Eq, Show, Generic)
1672+
1673+instance ToJSON OpenSearchMessage where
1674+ toEncoding = ourToEncoding
1675+ toJSON = ourToJSON
1676+
1677+-- * combined data types
1678+
1679+data CommitInfo = CommitInfo
1680+ { _commitInfoReqUser :: GhLogin,
1681+ _commitInfoRepoPublicity :: RepoPublicity,
1682+ _commitInfoRepoInfo :: RepoInfo,
1683+ _commitInfoBranch :: Maybe Branch,
1684+ _commitInfoPrFromFork :: Maybe PrFromFork,
1685+ _commitInfoCommit :: CommitHash
1686+ }
1687+ deriving stock (Show)
1688+
1689+data RepoInfo = RepoInfo
1690+ { _repoInfoInstallationAuth :: InstallationAuth,
1691+ _repoInfoGhToken :: GhToken,
1692+ _repoInfoGhRepoOwner :: GhRepoOwner,
1693+ _repoInfoGhRepoName :: GhRepoName
1694+ }
1695+
1696+instance Show RepoInfo where
1697+ show (RepoInfo _iAuth _ghToken repoOwner repoName) =
1698+ "RepoInfo <iAuth> <ghToken>" <> unwords [Prelude.show repoOwner, Prelude.show repoName]
1699+
1700+data PackageInfo = PackageInfo
1701+ { _packageInfoPackageType :: PackageType,
1702+ _packageInfoMaybeSystem :: MaybeSystem,
1703+ _packageInfoPackageName :: PackageName
1704+ }
1705+
1706+data BuildKind = Webhook | ModulePreview
1707+
1708+data RepoConfig = RepoConfig
1709+ { _repoConfigSkipPrivateInputsCheckForCollaborators :: Bool,
1710+ _repoConfigMaxEvalMemory :: Memory
1711+ }
1712+ deriving stock (Show)
1713+
1714+defaultRepoConfig :: RepoConfig
1715+defaultRepoConfig = RepoConfig False (fromGigabytes 8)
1716+
1717+newtype Memory = Memory Int64
1718+ deriving stock (Show, Eq, Generic, Ord)
1719+ deriving newtype
1720+ ( PGColumn "bigint",
1721+ PGParameter "bigint"
1722+ )
1723+
1724+data ProductPlan = ProductPlan
1725+ { _productPlanDisplayName :: Text,
1726+ _productPlanDescription :: Maybe Text,
1727+ _productPlanBaseCiTime :: Duration,
1728+ _productPlanMaximumPrDeploymentTime :: Duration,
1729+ _productPlanIncludedBranchDeploymentHosts :: Int64,
1730+ _productPlanMaximumPackagesPerFlake :: Int32,
1731+ _productPlanPackageEvaluationTimeout :: Int16,
1732+ _productPlanPackageBuildTimeout :: Int16,
1733+ _productPlanExtraUsage :: ExtraUsageLimits,
1734+ _productPlanIsPaid :: Bool
1735+ }
1736+ deriving stock (Eq, Show, Generic)
1737+
1738+instance ToJSON ProductPlan where
1739+ toEncoding = ourToEncoding
1740+ toJSON = ourToJSON
1741+
1742+data ExtraUsageLimits = ExtraUsageLimits
1743+ { ciTime :: Duration,
1744+ prDeployTime :: Duration,
1745+ hostingSpend :: MonetaryCost
1746+ }
1747+ deriving (Eq, Show, Generic, ToJSON)
1748+
1749+instance FromJSON ExtraUsageLimits where
1750+ parseJSON = withObject "ExtraUsageLimits" $ \v -> do
1751+ limits <- genericParseJSON defaultOptions $ Aeson.Object v
1752+ when (limits ^. #ciTime < emptyDuration) $ fail "CI time cannot be negative"
1753+ when (limits ^. #prDeployTime < emptyDuration) $ fail "PR deploy time cannot be negative"
1754+ when (limits ^. #hostingSpend < usd 0) $ fail "Hosting spend cannot be negative"
1755+ pure limits
1756+
1757+emptyUsageLimits :: ExtraUsageLimits
1758+emptyUsageLimits =
1759+ ExtraUsageLimits
1760+ { ciTime = emptyDuration,
1761+ prDeployTime = emptyDuration,
1762+ hostingSpend = usd 0
1763+ }
1764+
1765+toBytes :: Memory -> Int64
1766+toBytes (Memory bytes) = bytes
1767+
1768+fromGigabytes :: Int64 -> Memory
1769+fromGigabytes gbs = Memory $ gbs * 1024 * 1024 * 1024
1770+
1771+data DatabaseConnection
1772+ = ConnectionPool (Pool PGConnection)
1773+ | Transaction PGConnection
1774+
1775+makeFields ''Repo
1776+makeFields ''Build
1777+makeFields ''OpenSearchMessage
1778+makeFields ''Package
1779+makeFields ''GhRun
1780+makeFields ''RunOutput
1781+makeFields ''User
1782+makeFields ''CreatingUser
1783+makeFields ''CreateUser
1784+makeFields ''ServerInfo
1785+makeFields ''PreprovisionedServer
1786+makeFields ''Host
1787+makeFields ''CommitInfo
1788+makeFields ''RepoInfo
1789+makeFields ''PackageInfo
1790+makeFields ''RepoConfig
1791+makeFields ''CommitSummary
1792+makeFields ''BuildUpdate
1793+makeFields ''Commit
1794+makeFields ''Run
1795+makeFields ''ProductPlan
1796+
1797+makePrisms ''User
1798+makePrisms ''Repo
1799+makePrisms ''Build
1800+makePrisms ''ServerInfo
1801+makePrisms ''Host
1802+makePrisms ''CommitInfo
1803+makePrisms ''PackageInfo
1804+makePrisms ''RepoInfo
1805+makePrisms ''CommitStatus
1806+makePrisms ''CheckStatus
1807+makePrisms ''Commit
1808+makePrisms ''FullCommitState
backend/src/Garnix/Types/ExternalLenses.hsnew file+18
@@ -0,0 +1,18 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+
3+module Garnix.Types.ExternalLenses where
4+
5+import Control.Lens
6+import Garnix.Types.MakeLensHelpers
7+import GitHub.Data.Webhooks.Events
8+import GitHub.Data.Webhooks.Payload
9+
10+makeLensesWith (externalTypes "evCheckSuite") ''CheckSuiteEvent
11+makeLensesWith (externalTypes "evPullReq") ''PullRequestEvent
12+makeLensesWith (externalTypes "whChecksInstallation") ''HookChecksInstallation
13+makeLensesWith (externalTypes "whCheckSuite") ''HookCheckSuite
14+makeLensesWith (externalTypes "whCheckSuiteApp") ''HookCheckSuiteApp
15+makeLensesWith (externalTypes "whPullReq") ''HookPullRequest
16+makeLensesWith (externalTypes "whPullReqTarget") ''PullRequestTarget
17+makeLensesWith (externalTypes "whRepo") ''HookRepository
18+makeLensesWith (externalTypes "whUser") ''HookUser
backend/src/Garnix/Types/Keys.hsnew file+94
@@ -0,0 +1,94 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+
3+module Garnix.Types.Keys
4+ ( PublicKey (..),
5+ _PublicKey,
6+ PrivateKey,
7+ makePrivateKey,
8+ ExportKeysOpts (..),
9+ RepoSecretsEncryptionKeyPath (..),
10+ RepoSecretsEncryptionPubKey (..),
11+ unsafeDecryptPrivateKey,
12+ exportKeys,
13+ )
14+where
15+
16+import Data.ByteString qualified as SBS
17+import Development.Shake qualified as Shake
18+import Garnix.Prelude
19+import Servant qualified
20+import System.Exit
21+import System.Process qualified as Proc
22+
23+newtype PublicKey = PublicKey {getPublicKey :: Text}
24+ deriving stock (Eq, Show)
25+ deriving newtype
26+ ( Servant.MimeRender Servant.PlainText,
27+ PGColumn "character varying",
28+ PGColumn "text",
29+ PGParameter "character varying",
30+ PGParameter "text"
31+ )
32+
33+-- * Private keys
34+
35+-- Don't export the constructor or allow deserialization (besides DB).
36+-- That way only in this module can there be dangerous uses.
37+newtype PrivateKey = PrivateKey SBS.ByteString
38+ deriving newtype
39+ ( PGColumn "bytea",
40+ PGParameter "bytea"
41+ )
42+
43+newtype RepoSecretsEncryptionKeyPath = RepoSecretsEncryptionKeyPath FilePath
44+
45+newtype RepoSecretsEncryptionPubKey = RepoSecretsEncryptionPubKey Text
46+
47+makePrivateKey :: Text -> RepoSecretsEncryptionPubKey -> IO (Either Text PrivateKey)
48+makePrivateKey unencrypted (RepoSecretsEncryptionPubKey i) = do
49+ result <-
50+ Shake.cmd
51+ ("age" :: String)
52+ ["--recipient" :: String, cs i]
53+ (Shake.Stdin $ cs unencrypted)
54+ case result of
55+ (Shake.Exit ExitSuccess, Shake.Stdout (stdout :: SBS.ByteString)) -> pure . Right . PrivateKey $ stdout
56+ _ -> pure $ Left "Encrypting keys failed"
57+
58+unsafeDecryptPrivateKey :: PrivateKey -> RepoSecretsEncryptionKeyPath -> IO (Either Text Text)
59+unsafeDecryptPrivateKey (PrivateKey key) (RepoSecretsEncryptionKeyPath i) = do
60+ result <-
61+ Shake.cmd
62+ ("age" :: String)
63+ ["--decrypt" :: String, "-i", i]
64+ (Shake.StdinBS $ SBS.fromStrict key)
65+ case result of
66+ (Shake.Exit ExitSuccess, Shake.Stdout (stdout :: String)) -> pure . Right $ cs stdout
67+ _ -> pure $ Left "Decrypting keys failed"
68+
69+data ExportKeysOpts = ExportKeysOpts
70+ { privateKey :: PrivateKey,
71+ ipAddr :: Text,
72+ targetPath :: FilePath,
73+ sshArgs :: [Text]
74+ }
75+
76+exportKeys :: ExportKeysOpts -> RepoSecretsEncryptionKeyPath -> IO (Either Text ())
77+exportKeys opts id = do
78+ eprivKey <- unsafeDecryptPrivateKey (privateKey opts) id
79+ case eprivKey of
80+ Left e -> return $ Left e
81+ Right privKey -> do
82+ -- Use stdin so we don't have to store the key in the filesystem. We don't
83+ -- capture or log stderr in case they contain the key.
84+ (exitCode, _, _) <-
85+ Proc.readProcessWithExitCode
86+ "ssh"
87+ ( (cs <$> sshArgs opts) <> ["root@" <> cs (ipAddr opts), "cat >" <> targetPath opts]
88+ )
89+ (cs privKey)
90+ case exitCode of
91+ ExitSuccess -> pure $ Right ()
92+ ExitFailure _ -> pure $ Left "Exporting keys failed"
93+
94+makePrisms ''PublicKey
backend/src/Garnix/Types/MakeLensHelpers.hsnew file+25

This diff is too big to show here.

backend/src/Garnix/UserLogs.hsnew file+213

This diff is too big to show here.

backend/src/Garnix/YamlConfig.hsnew file+464

This diff is too big to show here.

backend/ssh-key-for-testsnew file+38

This diff is too big to show here.

backend/ssh-key-for-tests.pubnew file+1

This diff is too big to show here.

backend/test/spec/Garnix/API/AccountSpec.hsnew file+1140

This diff is too big to show here.

backend/test/spec/Garnix/API/AuthSpec.hsnew file+146

This diff is too big to show here.

backend/test/spec/Garnix/API/BadgesSpec.hsnew file+187

This diff is too big to show here.

backend/test/spec/Garnix/API/BuildSpec.hsnew file+389

This diff is too big to show here.

backend/test/spec/Garnix/API/Cache/PermissionsSpec.hsnew file+97

This diff is too big to show here.

backend/test/spec/Garnix/API/CommitSpec.hsnew file+461

This diff is too big to show here.

backend/test/spec/Garnix/API/ConfigSchemaSpec.hsnew file+28

This diff is too big to show here.

backend/test/spec/Garnix/API/DevSpec.hsnew file+61

This diff is too big to show here.

backend/test/spec/Garnix/API/GhWebhooksSpec.hsnew file+19

This diff is too big to show here.

backend/test/spec/Garnix/API/HealthSpec.hsnew file+16

This diff is too big to show here.

backend/test/spec/Garnix/API/HostsSpec.hsnew file+522

This diff is too big to show here.

backend/test/spec/Garnix/API/KeysSpec.hsnew file+56

This diff is too big to show here.

backend/test/spec/Garnix/API/ModulesSpec.hsnew file+278

This diff is too big to show here.

backend/test/spec/Garnix/API/RunsSpec.hsnew file+234

This diff is too big to show here.

backend/test/spec/Garnix/AccessTokenSpec.hsnew file+79

This diff is too big to show here.

backend/test/spec/Garnix/ActionSpec.hsnew file+525

This diff is too big to show here.

backend/test/spec/Garnix/AsyncSpec.hsnew file+91

This diff is too big to show here.

backend/test/spec/Garnix/AttributeSpec.hsnew file+85

This diff is too big to show here.

backend/test/spec/Garnix/Build/CheckoutSpec.hsnew file+42

This diff is too big to show here.

backend/test/spec/Garnix/Build/FodCheckSpec.hsnew file+538

This diff is too big to show here.

backend/test/spec/Garnix/Build/ModuleSpec.hsnew file+424

This diff is too big to show here.

backend/test/spec/Garnix/BuildSpec.hsnew file+1344

This diff is too big to show here.

backend/test/spec/Garnix/DB/FeatureFlagsSpec.hsnew file+73

This diff is too big to show here.

backend/test/spec/Garnix/DBSpec.hsnew file+216

This diff is too big to show here.

backend/test/spec/Garnix/DeploySpec.hsnew file+1583

This diff is too big to show here.

backend/test/spec/Garnix/EntitlementsSpec.hsnew file+96

This diff is too big to show here.

backend/test/spec/Garnix/ExpiringCacheSpec.hsnew file+93

This diff is too big to show here.

backend/test/spec/Garnix/FlakeInputAuthorizationSpec.hsnew file+139

This diff is too big to show here.

backend/test/spec/Garnix/GetAttributesSpec.hsnew file+214

This diff is too big to show here.

backend/test/spec/Garnix/GithubInterfaceSpec.hsnew file+54

This diff is too big to show here.

backend/test/spec/Garnix/HetznerInterfaceSpec.hsnew file+24

This diff is too big to show here.

backend/test/spec/Garnix/Hosting/BillingSpec.hsnew file+282

This diff is too big to show here.

backend/test/spec/Garnix/Hosting/DeploySpec.hsnew file+39

This diff is too big to show here.

backend/test/spec/Garnix/Hosting/ServerPoolSpec.hsnew file+152

This diff is too big to show here.

backend/test/spec/Garnix/IncrementalSpec.hsnew file+184

This diff is too big to show here.

backend/test/spec/Garnix/Modules/SchemaSpec.hsnew file+529

This diff is too big to show here.

backend/test/spec/Garnix/ModulesSpec.hsnew file+145

This diff is too big to show here.

backend/test/spec/Garnix/Monad/BubblingSpec.hsnew file+34

This diff is too big to show here.

backend/test/spec/Garnix/Monad/ConcurrencySpec.hsnew file+51

This diff is too big to show here.

backend/test/spec/Garnix/Monad/ForkTSpec.hsnew file+165

This diff is too big to show here.

backend/test/spec/Garnix/Monad/MemoizationSpec.hsnew file+57

This diff is too big to show here.

backend/test/spec/Garnix/Monad/NoThrowSpec.hsnew file+104

This diff is too big to show here.

backend/test/spec/Garnix/Monad/PoolSpec.hsnew file+134

This diff is too big to show here.

backend/test/spec/Garnix/Monad/SubProcessSpec.hsnew file+112

This diff is too big to show here.

backend/test/spec/Garnix/MonadSpec.hsnew file+145

This diff is too big to show here.

backend/test/spec/Garnix/Nix/StorePathSpec.hsnew file+117

This diff is too big to show here.

backend/test/spec/Garnix/PreludeSpec.hsnew file+35

This diff is too big to show here.

backend/test/spec/Garnix/ProcessLogsSpec.hsnew file+282

This diff is too big to show here.

backend/test/spec/Garnix/Reporters/GithubReporterSpec.hsnew file+106

This diff is too big to show here.

backend/test/spec/Garnix/Reporters/OpenSearchReporterSpec.hsnew file+133

This diff is too big to show here.

backend/test/spec/Garnix/Reporters/UtilsSpec.hsnew file+108

This diff is too big to show here.

backend/test/spec/Garnix/S3CacheSpec.hsnew file+833

This diff is too big to show here.

backend/test/spec/Garnix/SandboxSpec.hsnew file+267

This diff is too big to show here.

backend/test/spec/Garnix/StripeLibSpec.hsnew file+138

This diff is too big to show here.

backend/test/spec/Garnix/TestHelpers.hsnew file+601

This diff is too big to show here.

backend/test/spec/Garnix/TestHelpers/Common.hsnew file+23

This diff is too big to show here.

backend/test/spec/Garnix/TestHelpers/Deprecated.hsnew file+145

This diff is too big to show here.

backend/test/spec/Garnix/TestHelpers/GithubInterface.hsnew file+179

This diff is too big to show here.

backend/test/spec/Garnix/TestHelpers/GithubInterface/Deprecated.hsnew file+52

This diff is too big to show here.

backend/test/spec/Garnix/TestHelpers/GithubInterface/Internal.hsnew file+243

This diff is too big to show here.

backend/test/spec/Garnix/TestHelpers/HetznerMock.hsnew file+244

This diff is too big to show here.

backend/test/spec/Garnix/TestHelpers/Monad.hsnew file+422

This diff is too big to show here.

backend/test/spec/Garnix/TestHelpers/NixosVmScripts.hsnew file+41

This diff is too big to show here.

backend/test/spec/Garnix/TestHelpers/Reporter.hsnew file+55

This diff is too big to show here.

backend/test/spec/Garnix/TestHelpers/ServerPool.hsnew file+86

This diff is too big to show here.

backend/test/spec/Garnix/TestHelpers/WithServer.hsnew file+117

This diff is too big to show here.

backend/test/spec/Garnix/TestInstances.hsnew file+208

This diff is too big to show here.

backend/test/spec/Garnix/TypesSpec.hsnew file+101

This diff is too big to show here.

backend/test/spec/Garnix/UserLogsSpec.hsnew file+140

This diff is too big to show here.

backend/test/spec/Garnix/YamlConfigSpec.hsnew file+414

This diff is too big to show here.

backend/test/spec/Integration/FlakesSpec.hsnew file+340

This diff is too big to show here.

backend/test/spec/Integration/bad-flake-nix/flake.nixnew file+1

This diff is too big to show here.

backend/test/spec/Integration/bad-flake-nix/spec.yamlnew file+7

This diff is too big to show here.

backend/test/spec/Integration/bad-yaml-config/garnix.yamlnew file+5

This diff is too big to show here.

backend/test/spec/Integration/bad-yaml-config/spec.yamlnew file+7

This diff is too big to show here.

backend/test/spec/Integration/basic/flake.locknew file+25

This diff is too big to show here.

backend/test/spec/Integration/basic/flake.nixnew file+44

This diff is too big to show here.

backend/test/spec/Integration/basic/spec.yamlnew file+11

This diff is too big to show here.

backend/test/spec/Integration/darwin-configuration/flake.locknew file+61

This diff is too big to show here.

backend/test/spec/Integration/darwin-configuration/flake.nixnew file+42

This diff is too big to show here.

backend/test/spec/Integration/darwin-configuration/spec.yamlnew file+12

This diff is too big to show here.

backend/test/spec/Integration/failed-dependency/flake.locknew file+25

This diff is too big to show here.

backend/test/spec/Integration/failed-dependency/flake.nixnew file+61

This diff is too big to show here.

backend/test/spec/Integration/failed-dependency/spec.yamlnew file+13

This diff is too big to show here.

backend/test/spec/Integration/home-configuration/flake.locknew file+46

This diff is too big to show here.

backend/test/spec/Integration/home-configuration/flake.nixnew file+35

This diff is too big to show here.

backend/test/spec/Integration/home-configuration/spec.yamlnew file+12

This diff is too big to show here.

backend/test/spec/Integration/nixos-configuration/flake.locknew file+27

This diff is too big to show here.

backend/test/spec/Integration/nixos-configuration/flake.nixnew file+33

This diff is too big to show here.

backend/test/spec/Integration/nixos-configuration/spec.yamlnew file+12

This diff is too big to show here.

backend/test/spec/Integration/private-github-dependency-different-org-failing/flake.locknew file+42

This diff is too big to show here.

backend/test/spec/Integration/private-github-dependency-different-org-failing/flake.nixnew file+8

This diff is too big to show here.

backend/test/spec/Integration/private-github-dependency-different-org-failing/spec.yamlnew file+9

This diff is too big to show here.

backend/test/spec/Integration/private-github-dependency-external-pr/flake.locknew file+42

This diff is too big to show here.

backend/test/spec/Integration/private-github-dependency-external-pr/flake.nixnew file+8

This diff is too big to show here.

backend/test/spec/Integration/private-github-dependency-external-pr/spec.yamlnew file+9

This diff is too big to show here.

backend/test/spec/Integration/private-github-dependency-succeeding/flake.locknew file+42

This diff is too big to show here.

backend/test/spec/Integration/private-github-dependency-succeeding/flake.nixnew file+35

This diff is too big to show here.

backend/test/spec/Integration/private-github-dependency-succeeding/garnix.yamlnew file

No changes to the content.

backend/test/spec/Integration/private-github-dependency-succeeding/spec.yamlnew file+8

This diff is too big to show here.

backend/test/spec/Integration/private-github-dependency-users-without-access-failing/flake.locknew file+42

This diff is too big to show here.

backend/test/spec/Integration/private-github-dependency-users-without-access-failing/flake.nixnew file+8

This diff is too big to show here.

backend/test/spec/Integration/private-github-dependency-users-without-access-failing/spec.yamlnew file+12

This diff is too big to show here.

backend/test/spec/Integration/private-github-dependency-users-without-access-skip-check-succeeding/flake.locknew file+42

This diff is too big to show here.

backend/test/spec/Integration/private-github-dependency-users-without-access-skip-check-succeeding/flake.nixnew file+8

This diff is too big to show here.

backend/test/spec/Integration/private-github-dependency-users-without-access-skip-check-succeeding/garnix.yamlnew file+1

This diff is too big to show here.

backend/test/spec/Integration/private-github-dependency-users-without-access-skip-check-succeeding/spec.yamlnew file+11

This diff is too big to show here.

backend/test/spec/Integration/public-github-dependency-external-pr/flake.locknew file+42

This diff is too big to show here.

backend/test/spec/Integration/public-github-dependency-external-pr/flake.nixnew file+36

This diff is too big to show here.

backend/test/spec/Integration/public-github-dependency-external-pr/spec.yamlnew file+9

This diff is too big to show here.

backend/test/spec/Integration/public-github-dependency/flake.locknew file+42

This diff is too big to show here.

backend/test/spec/Integration/public-github-dependency/flake.nixnew file+34

This diff is too big to show here.

backend/test/spec/Integration/public-github-dependency/spec.yamlnew file+7

This diff is too big to show here.

backend/test/spec/Integration/public-tarball-dependency/flake.locknew file+38

This diff is too big to show here.

backend/test/spec/Integration/public-tarball-dependency/flake.nixnew file+33

This diff is too big to show here.

backend/test/spec/Integration/public-tarball-dependency/spec.yamlnew file+7

This diff is too big to show here.

backend/test/spec/Integration/yaml-config/flake.locknew file+27

This diff is too big to show here.

backend/test/spec/Integration/yaml-config/flake.nixnew file+20

This diff is too big to show here.

backend/test/spec/Integration/yaml-config/garnix.yamlnew file+5

This diff is too big to show here.

backend/test/spec/Integration/yaml-config/spec.yamlnew file+11

This diff is too big to show here.

backend/test/spec/Spec.hsnew file+1

This diff is too big to show here.

backend/test/spec/SpecHook.hsnew file+29

This diff is too big to show here.

backend/test/spec/data/create-server-response.jsonnew file+235

This diff is too big to show here.

backend/test/spec/data/emptyDir/.gitignorenew file

No changes to the content.

backend/test/spec/data/flake.locknew file+364

This diff is too big to show here.

backend/test/spec/data/get-servers-response.jsonnew file+400

This diff is too big to show here.

backend/test/spec/data/repo-secrets.keynew file+3

This diff is too big to show here.

cabal.projectnew file+1

This diff is too big to show here.

dns/.gitignorenew file+1

This diff is too big to show here.

dns/Cargo.locknew file+2289

This diff is too big to show here.

dns/Cargo.tomlnew file+18

This diff is too big to show here.

dns/default.nixnew file+32

This diff is too big to show here.

dns/nixos-module.nixnew file+59

This diff is too big to show here.

dns/src/dns.rsnew file+111

This diff is too big to show here.

dns/src/main.rsnew file+387

This diff is too big to show here.

dns/src/test_helpers.rsnew file+78

This diff is too big to show here.

examples/default.nixnew file+13

This diff is too big to show here.

examples/example-multi-server-deployment.nixnew file+91

This diff is too big to show here.

flake.locknew file+251

This diff is too big to show here.

flake.nixnew file+169

This diff is too big to show here.

frontend/.eslintrc.jsonnew file+50

This diff is too big to show here.

frontend/.gitignorenew file+22

This diff is too big to show here.

frontend/README.mdnew file+36

This diff is too big to show here.

frontend/age-wasm/default.nixnew file+49

This diff is too big to show here.

frontend/age-wasm/go.modnew file+10

This diff is too big to show here.

frontend/age-wasm/go.sumnew file+6

This diff is too big to show here.

frontend/age-wasm/index.tsnew file+21

This diff is too big to show here.

frontend/age-wasm/integration-test.mjsnew file+58

This diff is too big to show here.

frontend/age-wasm/main.gonew file+61

This diff is too big to show here.

frontend/age-wasm/tsconfig.jsonnew file+8

This diff is too big to show here.

frontend/default.nixnew file+93

This diff is too big to show here.

frontend/jest-wasm-mock.jsnew file+1

This diff is too big to show here.

frontend/jest.config.tsnew file+16

This diff is too big to show here.

frontend/knip.jsonnew file+12

This diff is too big to show here.

frontend/next-env.d.tsnew file+5

This diff is too big to show here.

frontend/next.config.jsnew file+48

This diff is too big to show here.

frontend/package-lock.jsonnew file+15025

This diff is too big to show here.

frontend/package.jsonnew file+60

This diff is too big to show here.

frontend/src/app/account/access_tokens/index.tsxnew file+386

This diff is too big to show here.

frontend/src/app/account/access_tokens/styles.module.cssnew file+61

This diff is too big to show here.

frontend/src/app/account/gh/[slug]/page.tsxnew file+370

This diff is too big to show here.

frontend/src/app/account/gh/[slug]/styles.module.cssnew file+198

This diff is too big to show here.

frontend/src/app/account/manage_plans/page.tsxnew file+340

This diff is too big to show here.

frontend/src/app/account/manage_plans/styles.module.cssnew file+50

This diff is too big to show here.

frontend/src/app/account/page.tsxnew file+24

This diff is too big to show here.

frontend/src/app/account/styles.module.cssnew file+16

This diff is too big to show here.

frontend/src/app/account/usage/index.tsxnew file+104

This diff is too big to show here.

frontend/src/app/account/usage/styles.module.cssnew file+28

This diff is too big to show here.

frontend/src/app/build/[slug]/page.tsxnew file+194

This diff is too big to show here.

frontend/src/app/build/[slug]/styles.module.cssnew file+77

This diff is too big to show here.

frontend/src/app/build/layout.tsxnew file+7

This diff is too big to show here.

frontend/src/app/commit/[slug]/page.tsxnew file+72

This diff is too big to show here.

frontend/src/app/commit/[slug]/styles.module.cssnew file+53

This diff is too big to show here.

frontend/src/app/commit/layout.tsxnew file+7

This diff is too big to show here.

frontend/src/app/favicon.iconew file

Binary file not shown.

frontend/src/app/garnix-admin/createGithubApp.tsxnew file+121

This diff is too big to show here.

frontend/src/app/garnix-admin/page.tsxnew file+78

This diff is too big to show here.

frontend/src/app/garnix-admin/styles.module.cssnew file+15

This diff is too big to show here.

frontend/src/app/globals.cssnew file+67

This diff is too big to show here.

frontend/src/app/layout.tsxnew file+65

This diff is too big to show here.

frontend/src/app/login/cb/page.tsxnew file+78

This diff is too big to show here.

frontend/src/app/login/cb/styles.module.cssnew file+12

This diff is too big to show here.

frontend/src/app/login/page.tsxnew file+32

This diff is too big to show here.

frontend/src/app/login/styles.module.cssnew file+7

This diff is too big to show here.

frontend/src/app/logout/page.tsxnew file+17

This diff is too big to show here.

frontend/src/app/modules/configure/markdown/index.tsxnew file+23

This diff is too big to show here.

frontend/src/app/modules/configure/markdown/styles.module.cssnew file+16

This diff is too big to show here.

frontend/src/app/modules/configure/moduleInputs/list/index.tsxnew file+58

This diff is too big to show here.

frontend/src/app/modules/configure/moduleInputs/list/styles.module.cssnew file+25

This diff is too big to show here.

frontend/src/app/modules/configure/moduleInputs/listOfPackages/addPackageModal/all-nix-pkgs.tsnew file+22211

This diff is too big to show here.

frontend/src/app/modules/configure/moduleInputs/listOfPackages/addPackageModal/index.tsxnew file+121

This diff is too big to show here.

frontend/src/app/modules/configure/moduleInputs/listOfPackages/addPackageModal/styles.module.cssnew file+36

This diff is too big to show here.

frontend/src/app/modules/configure/moduleInputs/listOfPackages/index.tsxnew file+41

This diff is too big to show here.

frontend/src/app/modules/configure/moduleInputs/listOfPackages/styles.module.cssnew file+26

This diff is too big to show here.

frontend/src/app/modules/configure/moduleInputs/moduleSelector/index.tsxnew file+46

This diff is too big to show here.

frontend/src/app/modules/configure/moduleInputs/moduleSelector/styles.module.cssnew file+19

This diff is too big to show here.

frontend/src/app/modules/configure/moduleInputs/nixPath.tsxnew file+42

This diff is too big to show here.

frontend/src/app/modules/configure/moduleInputs/repoPicker/index.tsxnew file+69

This diff is too big to show here.

frontend/src/app/modules/configure/moduleInputs/repoPicker/styles.module.cssnew file+42

This diff is too big to show here.

frontend/src/app/modules/configure/moduleInputs/secretInput/index.tsxnew file+124

This diff is too big to show here.

frontend/src/app/modules/configure/moduleInputs/secretInput/styles.module.cssnew file+26

This diff is too big to show here.

frontend/src/app/modules/configure/openPrModal/index.tsxnew file+58

This diff is too big to show here.

frontend/src/app/modules/configure/openPrModal/styles.module.cssnew file+42

This diff is too big to show here.

frontend/src/app/modules/configure/page.test.tsxnew file+1083

This diff is too big to show here.

frontend/src/app/modules/configure/page.tsxnew file+675

This diff is too big to show here.

frontend/src/app/modules/configure/previewModal/index.tsxnew file+49

This diff is too big to show here.

frontend/src/app/modules/configure/previewModal/styles.module.cssnew file+36

This diff is too big to show here.

frontend/src/app/modules/configure/resetModal/index.tsxnew file+46

This diff is too big to show here.

frontend/src/app/modules/configure/resetModal/styles.module.cssnew file+36

This diff is too big to show here.

frontend/src/app/modules/configure/styles.module.cssnew file+56

This diff is too big to show here.

frontend/src/app/og.pngnew file

Binary file not shown.

frontend/src/app/page.tsxnew file+10

This diff is too big to show here.

frontend/src/app/repo/[owner]/[repo]/page.tsxnew file+15

This diff is too big to show here.

frontend/src/app/run/[slug]/page.tsxnew file+38

This diff is too big to show here.

frontend/src/app/run/layout.tsxnew file+7

This diff is too big to show here.

frontend/src/app/servers/filters/index.tsxnew file+71

This diff is too big to show here.

frontend/src/app/servers/filters/styles.module.cssnew file+13

This diff is too big to show here.

frontend/src/app/servers/page.tsxnew file+228

This diff is too big to show here.

frontend/src/app/servers/styles.module.cssnew file+56

This diff is too big to show here.

frontend/src/app/signup/fill/page.tsxnew file+74

This diff is too big to show here.

frontend/src/app/signup/fill/styles.module.cssnew file+12

This diff is too big to show here.

frontend/src/app/signup/page.tsxnew file+70

This diff is too big to show here.

frontend/src/app/signup/start/page.tsxnew file+126

This diff is too big to show here.

frontend/src/app/signup/start/styles.module.cssnew file+85

This diff is too big to show here.

frontend/src/app/signup/styles.module.cssnew file+50

This diff is too big to show here.

frontend/src/app/signup/tos/page.tsxnew file+28

This diff is too big to show here.

frontend/src/app/signup/tos/styles.module.cssnew file+20

This diff is too big to show here.

frontend/src/components/authModalHeader/index.tsxnew file+78

This diff is too big to show here.

frontend/src/components/authModalHeader/styles.module.cssnew file+78

This diff is too big to show here.

frontend/src/components/body/index.tsxnew file+8

This diff is too big to show here.

frontend/src/components/build/index.tsxnew file+123

This diff is too big to show here.

frontend/src/components/build/styles.module.cssnew file+62

This diff is too big to show here.

frontend/src/components/buildLog/index.tsxnew file+103

This diff is too big to show here.

frontend/src/components/buildLog/styles.module.cssnew file+72

This diff is too big to show here.

frontend/src/components/button/index.tsxnew file+87

This diff is too big to show here.

frontend/src/components/button/styles.module.cssnew file+58

This diff is too big to show here.

frontend/src/components/commitList/index.tsxnew file+81

This diff is too big to show here.

frontend/src/components/commitList/styles.module.cssnew file+67

This diff is too big to show here.

frontend/src/components/expander/index.tsxnew file+22

This diff is too big to show here.

frontend/src/components/expander/styles.module.cssnew file+4

This diff is too big to show here.

frontend/src/components/formSubmitResult/index.tsxnew file+15

This diff is too big to show here.

frontend/src/components/formSubmitResult/styles.module.cssnew file+16

This diff is too big to show here.

frontend/src/components/icons/account.tsxnew file+20

This diff is too big to show here.

frontend/src/components/icons/arrow-left.svgnew file+3

This diff is too big to show here.

frontend/src/components/icons/arrow-right.svgnew file+3

This diff is too big to show here.

frontend/src/components/icons/billing.tsxnew file+20

This diff is too big to show here.

frontend/src/components/icons/blocks.svgnew file+1264

This diff is too big to show here.

frontend/src/components/icons/blog.svgnew file+5

This diff is too big to show here.

frontend/src/components/icons/branch.svgnew file+16

This diff is too big to show here.

frontend/src/components/icons/builds.tsxnew file+20

This diff is too big to show here.

frontend/src/components/icons/clock.svgnew file+17

This diff is too big to show here.

frontend/src/components/icons/commit.svgnew file+14

This diff is too big to show here.

frontend/src/components/icons/cross.svgnew file+11

This diff is too big to show here.

frontend/src/components/icons/cube.svgnew file+17

This diff is too big to show here.

frontend/src/components/icons/dash.svgnew file+4

This diff is too big to show here.

frontend/src/components/icons/discord.tsxnew file+32

This diff is too big to show here.

frontend/src/components/icons/docs.svgnew file+5

This diff is too big to show here.

frontend/src/components/icons/documentation.tsxnew file+20

This diff is too big to show here.

frontend/src/components/icons/download.tsxnew file+9

This diff is too big to show here.

frontend/src/components/icons/email.tsxnew file+32

This diff is too big to show here.

frontend/src/components/icons/failure.svgnew file+13

This diff is too big to show here.

frontend/src/components/icons/folder.svgnew file+16

This diff is too big to show here.

frontend/src/components/icons/github.tsxnew file+24

This diff is too big to show here.

frontend/src/components/icons/globe.svgnew file+1268

This diff is too big to show here.

frontend/src/components/icons/hamburgerMenu.svgnew file+6

This diff is too big to show here.

frontend/src/components/icons/lock.svgnew file+4

This diff is too big to show here.

frontend/src/components/icons/login.svgnew file+5

This diff is too big to show here.

frontend/src/components/icons/logo.svgnew file+31

This diff is too big to show here.

frontend/src/components/icons/logoWhite.svgnew file+4

This diff is too big to show here.

frontend/src/components/icons/logout.tsxnew file+20

This diff is too big to show here.

frontend/src/components/icons/matrix.tsxnew file+21

This diff is too big to show here.

frontend/src/components/icons/menu.svgnew file+5

This diff is too big to show here.

frontend/src/components/icons/modules.tsxnew file+32

This diff is too big to show here.

frontend/src/components/icons/pending.svgnew file+19

This diff is too big to show here.

frontend/src/components/icons/repo.svgnew file+25

This diff is too big to show here.

frontend/src/components/icons/servers.tsxnew file+20

This diff is too big to show here.

frontend/src/components/icons/settings.tsxnew file+20

This diff is too big to show here.

frontend/src/components/icons/share.svgnew file+10

This diff is too big to show here.

frontend/src/components/icons/status.svgnew file+18

This diff is too big to show here.

frontend/src/components/icons/stopwatch.svgnew file+18

This diff is too big to show here.

frontend/src/components/icons/success.svgnew file+11

This diff is too big to show here.

frontend/src/components/icons/terminal.svgnew file+10

This diff is too big to show here.

frontend/src/components/icons/timestamp.svgnew file+17

This diff is too big to show here.

frontend/src/components/icons/trash.tsxnew file+11

This diff is too big to show here.

frontend/src/components/icons/video.svgnew file+5

This diff is too big to show here.

frontend/src/components/icons/x.svgnew file+5

This diff is too big to show here.

frontend/src/components/images/globe.svgnew file+786

This diff is too big to show here.

frontend/src/components/input/index.tsxnew file+85

This diff is too big to show here.

frontend/src/components/input/styles.module.cssnew file+18

This diff is too big to show here.

frontend/src/components/link/index.tsxnew file+51

This diff is too big to show here.

frontend/src/components/link/styles.module.cssnew file+55

This diff is too big to show here.

frontend/src/components/loading/index.tsxnew file+99

This diff is too big to show here.

frontend/src/components/loading/styles.module.cssnew file+14

This diff is too big to show here.

frontend/src/components/loginAnimation/animation.jsonnew file+1

This diff is too big to show here.

frontend/src/components/loginAnimation/index.tsxnew file+34

This diff is too big to show here.

frontend/src/components/loginAnimation/styles.module.cssnew file+6

This diff is too big to show here.

frontend/src/components/loginPrompt/index.tsxnew file+34

This diff is too big to show here.

frontend/src/components/loginPrompt/styles.module.cssnew file+11

This diff is too big to show here.

frontend/src/components/mdxDocument/header/index.tsxnew file+51

This diff is too big to show here.

frontend/src/components/mdxDocument/header/styles.module.cssnew file+42

This diff is too big to show here.

frontend/src/components/mdxDocument/index.tsxnew file+183

This diff is too big to show here.

frontend/src/components/mdxDocument/styles.module.cssnew file+141

This diff is too big to show here.

frontend/src/components/modal/index.tsxnew file+55

This diff is too big to show here.

frontend/src/components/modal/styles.module.cssnew file+61

This diff is too big to show here.

frontend/src/components/navigation/index.tsxnew file+243

This diff is too big to show here.

frontend/src/components/navigation/styles.module.cssnew file+203

This diff is too big to show here.

frontend/src/components/portal/index.tsxnew file+17

This diff is too big to show here.

frontend/src/components/run/index.tsxnew file+84

This diff is too big to show here.

frontend/src/components/run/styles.module.cssnew file+77

This diff is too big to show here.

frontend/src/components/sampleCode.tsxnew file+39

This diff is too big to show here.

frontend/src/components/select/index.tsxnew file+37

This diff is too big to show here.

frontend/src/components/select/styles.module.cssnew file+24

This diff is too big to show here.

frontend/src/components/statusIcon/index.tsxnew file+50

This diff is too big to show here.

frontend/src/components/statusIcon/styles.module.cssnew file+30

This diff is too big to show here.

frontend/src/components/table/index.tsxnew file+15

This diff is too big to show here.

frontend/src/components/table/styles.module.cssnew file+27

This diff is too big to show here.

frontend/src/components/terminal/index.tsxnew file+34

This diff is too big to show here.

frontend/src/components/terminal/styles.module.cssnew file+50

This diff is too big to show here.

frontend/src/components/text/index.tsxnew file+87

This diff is too big to show here.

frontend/src/components/text/styles.module.cssnew file+64

This diff is too big to show here.

frontend/src/components/toggleSwitch/index.tsxnew file+24

This diff is too big to show here.

frontend/src/components/toggleSwitch/styles.module.cssnew file+42

This diff is too big to show here.

frontend/src/components/tooltip/index.tsxnew file+35

This diff is too big to show here.

frontend/src/components/tooltip/styles.module.cssnew file+22

This diff is too big to show here.

frontend/src/components/withSidebar/index.tsxnew file+25

This diff is too big to show here.

frontend/src/components/withSidebar/styles.module.cssnew file+65

This diff is too big to show here.

frontend/src/hooks/useDebounced.tsnew file+15

This diff is too big to show here.

frontend/src/hooks/useForm.tsnew file+98

This diff is too big to show here.

frontend/src/hooks/useLoading.tsnew file+55

This diff is too big to show here.

frontend/src/hooks/useLogStream.tsnew file+80

This diff is too big to show here.

frontend/src/hooks/useLoginLinkForCurrentPage.tsnew file+15

This diff is too big to show here.

frontend/src/middleware.tsnew file+39

This diff is too big to show here.

frontend/src/services/account.tsnew file+288

This diff is too big to show here.

frontend/src/services/apiErrors.tsnew file+7

This diff is too big to show here.

frontend/src/services/auth.tsnew file+114

This diff is too big to show here.

frontend/src/services/build.tsnew file+72

This diff is too big to show here.

frontend/src/services/commit.tsnew file+72

This diff is too big to show here.

frontend/src/services/config.tsnew file+14

This diff is too big to show here.

frontend/src/services/index.tsnew file+70

This diff is too big to show here.

frontend/src/services/keys.tsnew file+8

This diff is too big to show here.

frontend/src/services/logs.tsnew file+32

This diff is too big to show here.

frontend/src/services/modules/index.test.tsnew file+264

This diff is too big to show here.

frontend/src/services/modules/index.tsnew file+526

This diff is too big to show here.

frontend/src/services/modules/nixValue.tsnew file+51

This diff is too big to show here.

frontend/src/services/run.tsnew file+31

This diff is too big to show here.

frontend/src/services/servers.tsnew file+43

This diff is too big to show here.

frontend/src/store/configContext.tsxnew file+40

This diff is too big to show here.

frontend/src/store/providers.tsxnew file+13

This diff is too big to show here.

frontend/src/store/userContext.tsxnew file+87

This diff is too big to show here.

frontend/src/utils/analytics.tsnew file+8

This diff is too big to show here.

frontend/src/utils/ansi/colors.tsnew file+263

This diff is too big to show here.

frontend/src/utils/ansi/index.test.tsnew file+340

This diff is too big to show here.

frontend/src/utils/ansi/index.tsnew file+124

This diff is too big to show here.

frontend/src/utils/appPage.tsxnew file+36

This diff is too big to show here.

frontend/src/utils/colors.cssnew file+20

This diff is too big to show here.

frontend/src/utils/currency.test.tsnew file+54

This diff is too big to show here.

frontend/src/utils/currency.tsnew file+18

This diff is too big to show here.

frontend/src/utils/custom.d.tsnew file+9

This diff is too big to show here.

frontend/src/utils/deferred.tsnew file+15

This diff is too big to show here.

frontend/src/utils/dev.tsxnew file+10

This diff is too big to show here.

frontend/src/utils/duration.test.tsnew file+110

This diff is too big to show here.

frontend/src/utils/duration.tsnew file+74

This diff is too big to show here.

frontend/src/utils/featureFlags.tsnew file+30

This diff is too big to show here.

frontend/src/utils/fonts/BerlinTypeWeb-Regular.woffnew file

Binary file not shown.

frontend/src/utils/fonts/MatterSQMono-Light.woffnew file

Binary file not shown.

frontend/src/utils/fonts/MatterSQMono-LightItalic.woffnew file

Binary file not shown.

frontend/src/utils/fonts/MatterSQMono-Regular.woffnew file

Binary file not shown.

frontend/src/utils/fonts/MatterSQMono-RegularItalic.woffnew file

Binary file not shown.

frontend/src/utils/fonts/index.tsnew file+34

This diff is too big to show here.

frontend/src/utils/format.tsnew file+52

This diff is too big to show here.

frontend/src/utils/index.test.tsnew file+55

This diff is too big to show here.

frontend/src/utils/index.tsnew file+52

This diff is too big to show here.

frontend/src/utils/sizes.cssnew file+6

This diff is too big to show here.

frontend/src/utils/withPropCheck.tsnew file+8

This diff is too big to show here.

frontend/src/utils/zod.tsnew file+7

This diff is too big to show here.

frontend/tsconfig.jsonnew file+28

This diff is too big to show here.

garnix.yamlnew file+18

This diff is too big to show here.

hosting-gateway/default.nixnew file+51

This diff is too big to show here.

hosting-gateway/heartbeatmiddleware/.traefik.ymlnew file+5

This diff is too big to show here.

hosting-gateway/heartbeatmiddleware/go.modnew file+3

This diff is too big to show here.

hosting-gateway/heartbeatmiddleware/hostset.gonew file+32

This diff is too big to show here.

hosting-gateway/heartbeatmiddleware/plugin.gonew file+70

This diff is too big to show here.

hosting-gateway/nixos-module.nixnew file+184

This diff is too big to show here.

hosting-gateway/on-demand-resolver/.gitignorenew file+2

This diff is too big to show here.

hosting-gateway/on-demand-resolver/package-lock.jsonnew file+59

This diff is too big to show here.

hosting-gateway/on-demand-resolver/package.jsonnew file+16

This diff is too big to show here.

hosting-gateway/on-demand-resolver/src/index.tsnew file+49

This diff is too big to show here.

hosting-gateway/on-demand-resolver/src/lib.test.tsnew file+199

This diff is too big to show here.

hosting-gateway/on-demand-resolver/src/lib.tsnew file+58

This diff is too big to show here.

hosting-gateway/on-demand-resolver/tsconfig.jsonnew file+16

This diff is too big to show here.

justfilenew file+48

This diff is too big to show here.

nix/data/emptyDir/.gitignorenew file

No changes to the content.

nix/data/grafana-node-exporter-full.jsonnew file+14119

This diff is too big to show here.

nix/data/ssh-key-for-local-dev-secretsnew file+7

This diff is too big to show here.

nix/data/ssh-key-for-local-dev-secrets.pubnew file+1

This diff is too big to show here.

nix/modules/action-runner.nixnew file+261

This diff is too big to show here.

nix/modules/common.nixnew file+79

This diff is too big to show here.

nix/modules/custom-gc.nixnew file+174

This diff is too big to show here.

nix/modules/database.nixnew file+414

This diff is too big to show here.

nix/modules/default.nixnew file+15

This diff is too big to show here.

nix/modules/dev-mode.nixnew file+94

This diff is too big to show here.

nix/modules/fluent-bit.nixnew file+356

This diff is too big to show here.

nix/modules/linux-common.nixnew file+452

This diff is too big to show here.

nix/modules/monitoring-client.nixnew file+157

This diff is too big to show here.

nix/modules/monitoring-server.nixnew file+178

This diff is too big to show here.

nix/modules/monitoring.nixnew file+76

This diff is too big to show here.

nix/packages/HDBC.nixnew file+33

This diff is too big to show here.

nix/packages/db.nixnew file+96

This diff is too big to show here.

nix/packages/garnix.nixnew file+453

This diff is too big to show here.

nix/packages/generic-random.nixnew file+11

This diff is too big to show here.

nix/packages/generics-eot.nixnew file+42

This diff is too big to show here.

nix/packages/github-app.nixnew file+60

This diff is too big to show here.

nix/packages/github-webhooks.nixnew file+43

This diff is too big to show here.

nix/packages/iso-deriving.nixnew file+20

This diff is too big to show here.

nix/packages/migrate.nixnew file+14

This diff is too big to show here.

nix/packages/oauth2-simple.nixnew file+46

This diff is too big to show here.

nix/packages/opensearch-dashboards/default.nixnew file+49

This diff is too big to show here.

nix/packages/opensearch-dashboards/disable-nodejs-version-check.patchnew file+10

This diff is too big to show here.

nix/packages/servant-github-webhook.nixnew file+62

This diff is too big to show here.

nix/packages/withSecrets.nixnew file+57

This diff is too big to show here.

nix/tests/data/foo.garnix.me.cert.pemnew file+20

This diff is too big to show here.

nix/tests/data/foo.garnix.me.key.pemnew file+27

This diff is too big to show here.

nix/tests/data/id_ed25519_garnix_server_testingnew file+7

This diff is too big to show here.

nix/tests/data/id_ed25519_garnix_server_testing.pubnew file+1

This diff is too big to show here.

nix/tests/database.nixnew file+57

This diff is too big to show here.

nix/tests/default.nixnew file+197

This diff is too big to show here.

nix/tests/fluent-bit.nixnew file+88

This diff is too big to show here.

nix/tests/garnixServer.nixnew file+67

This diff is too big to show here.

nix/tests/hosting-gateway.nixnew file+223

This diff is too big to show here.

nix/tests/monitoring-node.nixnew file+64

This diff is too big to show here.

nix/tests/monitoring-server.nixnew file+18

This diff is too big to show here.

nix/tests/opensearch.nixnew file+81

This diff is too big to show here.

nix/treefmt.nixnew file+40

This diff is too big to show here.

opensearch/default.nixnew file+3

This diff is too big to show here.

opensearch/nixos-module.nixnew file+255

This diff is too big to show here.

secrets/dev.yamlnew file+45

This diff is too big to show here.

sql/README.mdnew file+30

This diff is too big to show here.

sql/deploy/init.sqlnew file+527

This diff is too big to show here.

sql/local-fixtures.sqlnew file+154

This diff is too big to show here.

sql/queries/build_time_percentiles.sqlnew file+32

This diff is too big to show here.

sql/queries/monthly_build_times.sqlnew file+15

This diff is too big to show here.

sql/sqitch.confnew file+2

This diff is too big to show here.

sql/sqitch.plannew file+4

This diff is too big to show here.

watchdog/cabal.projectnew file+1

This diff is too big to show here.

watchdog/data/ci/datenew file+1

This diff is too big to show here.

watchdog/data/ci/flake.locknew file+60

This diff is too big to show here.

watchdog/data/ci/flake.nixnew file+14

This diff is too big to show here.

watchdog/data/ci/garnix.yamlnew file+3

This diff is too big to show here.

watchdog/data/pr-deployment/datenew file+1

This diff is too big to show here.

watchdog/data/pr-deployment/flake.locknew file+25

This diff is too big to show here.

watchdog/data/pr-deployment/flake.nixnew file+33

This diff is too big to show here.

watchdog/data/pr-deployment/garnix.yamlnew file+4

This diff is too big to show here.

watchdog/default.nixnew file+60

This diff is too big to show here.

watchdog/nixos-module.nixnew file+55

This diff is too big to show here.

watchdog/package.yamlnew file+73

This diff is too big to show here.

watchdog/src/Garnix/Watchdog/Checks.hsnew file+284

This diff is too big to show here.

watchdog/src/Garnix/Watchdog/Github.hsnew file+48

This diff is too big to show here.

watchdog/src/Garnix/Watchdog/Utils.hsnew file+39

This diff is too big to show here.

watchdog/src/Main.hsnew file+70

This diff is too big to show here.

watchdog/watchdog.cabalnew file+78

This diff is too big to show here.