analytics/HOSTING.md
Chandra Tungathurthi 5eb8929929
Support for docker based self-hosting (#64)
* first commit with test and compile job

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding 'prepare' stage

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated ci script to include "test" compile phase

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding environment variables for connecting to postgresql

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated ci config for postgres

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* using non-alpine version of elixir

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* re-using the 'compile' artifacts and added explict env variables for testing

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removing redundant deps fetching from common code

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* formatting using mix.format -- beware no-code changes!

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* added release config

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding consistent env variable for Database

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* more cleaning up of environment variables

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding releases config for enabling releases

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* cleaning up env configs

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Cleaned up config and prepared config for releases

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated CI script with new config for test

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added Dockerfile for creating production docker image

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding "docker" build job yay!

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* using non-slim version of debian and installing webpack

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding overlays for migrations on releases

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* restricting the docker built to master branch only

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* typo fix

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding "Hosting.md" to explain hosting instructions

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removed the default comments

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added documentation related to env variables

* updated documentation and fixed typo

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated documentation

* Bumping up elixir version as `overlays` are only supported in latest version

read release notes: https://github.com/elixir-lang/elixir/releases/tag/v1.10.0

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding tarball assembly during release

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated HOSTING.md

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added support for db migration

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* minor corrections

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* initializing admin user

Admin user has been added in the "migration" phase. A default user is automatically created in the process. One can provide the related env variables, else a new one will be automatically created for you.

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Initial base domain update - phase#1

These changes are only meant for correct operating it under self-hosting. There are many other cosmetic changes, that require updates to email, site and other places where the original website and author is used.

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Using dedicated config variable `base_domain` instead

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding base_domain to releases config

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removing the dedicated config "base_domain", relying on endpoint host

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Removed the usage of "Mix" in code!

It is bad practice to use "mix" module inside the code as in actual release this module is unavailable. Replacing this with a config environment variable

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added support for SMTP via Bamboo Smtp Adapter

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Capturing SMTP errors via Sentry

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Minor updates

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding junit formatter -- useful for generating test reports

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding documentation for default user

* Resolve "Gitlab Adoption: Add supported services in "Security & Compliance""

* bumping up the debian version to fix issues

fixing some vulnerabilities identified by the scanning tools

* More updates for self-hosting

Changes in most of the places to suit self-hosting. Although, there are some which have been left-off.

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* quick-dirty-fix!

* bumping up the db connect timeout

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* bumping up the db connect timeout

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* bumping up the db connect timeout

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* bumping up timeout - skipping MRs :-/

* removing restrictions on watching for changes

this stuff isn't working

* Update HOSTING.md

* renamed the module name

* reverting formatting-whitespace changes

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* reverting the name to release

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding docker-compose.yml and related instructions

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* using `plausible_url` instead of assuming `https`

this is because, it is much to test in local dev machines and in most cases there's already a layer above which is capable for `https` termination and http -> https upgrade

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* WIP: merging changes from upstream

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* wip: more changes

* Pushing in changes from upstream

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* changes to ci for testing

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* cleaning up and finishing clickhouse integration

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updating readme with hosting details
2020-05-26 16:09:34 +03:00

6.3 KiB

Plausible Analytics

Self-hosting is possible based on the docker images and are automatically pushed into Gitlab hosted docker registry for all commits on master branch.
All master-* tags are considered to be stable and are persisted. Any other tag in the registry is considered to be for development purposes and/or unstable and are auto-deleted after a week.

Building Docker image

Besides the GitlabCI, one can build docker image from Dockerfile.

Up and Running

The repo supplies with a Docker Compose file, this serves as a sample for running Plausible with Docker. In this sample, the db migration is done by default on startup, so you need to clean the data up every time you run:

First run

$ docker-compose up

subsequent runs--

$ docker-compose down
$ docker volume rm plausible_db-data -f
$ docker-compose up

Non-docker building

It is possible to create a release artifact by running a release.

MIX_ENV=prod mix release plausible

the release will create the pre-packed artifact at _build/prod/rel/plausible/bin/plausible, the release will also create a tarball at _build/prod/ for convenience.

Note, that you have to feed in the related environment variables (see below Environment Variables)

Database Migration

On the initial setup, a migration step is necessary to create database and table schemas needed for initial bootup. Normally, this done by mix aliases like ecto.setup defined in the mix.exs. As this not available in "released" artifact, plausible_migration.ex facilitates this process. The overlay scripts take care of these.

After the release, these are available under _build/prod/rel/plausible --

_build/prod/rel/plausible/createdb.sh
_build/prod/rel/plausible/migrate.sh
_build/prod/rel/plausible/rollback.sh
_build/prod/rel/plausible/seed.sh

the same is available in the docker images as follows --

docker run plausible:master-12add db createdb
docker run plausible:master-12add db migrate
docker run plausible:master-12add db rollback
docker run plausible:master-12add db seed

Environment Variables

Plausible relies on the several services for operating, the expected environment variables are explaiend below.

Server

Following are the variables that can be used to configure the availability of the server.

  • HOST (String)
    • The hosting address of the server. For running on local system, this can be set to localhost. In production systems, this can be your ingress host.
  • PORT (Number)
    • The port on which the server is available.
  • SECRET_KEY_BASE (String)
  • ENVIRONMENT (String)
    • The current running environment. defaults to prod
  • APP_VERSION (String)
    • The version of the app running. defaults to current docker tag

Default User Generation

For self-hosting, a default user is generated during the [Database Migration](#Database Migration) to access Plausible. To be noted that, a default user is a user whose trial period expires in 100 Years ;). It is highly recommended that you configure these parameters.

  • ADMIN_USER_NAME
    • The default ("admin") username. if not provided, one will be generated for you
  • ADMIN_USER_EMAIL
    • The default ("admin") user email. if not provided, one will be generated for you
  • ADMIN_USER_PWD
    • The default ("admin") user password. if not provided, one will be generated for you

Mailer/SMTP Setup

  • MAILER_ADAPTER (String)
    • The adapter used for sending out e-mails. Available: Bamboo.PostmarkAdapter / Bamboo.SMTPAdapter
  • MAILER_EMAIL (String)
    • The email id to use for as from address of all communications from Plausible.

In case of Bamboo.SMTPAdapter you need to supply the following variables:

  • SMTP_HOST_ADDR (String)
    • The host address of your smtp server.
  • SMTP_HOST_PORT (Number)
    • The port of your smtp server.
  • SMTP_USER_NAME (String)
    • The username/email for smtp auth.
  • SMTP_USER_PWD (String)
    • The password for smtp auth.
  • SMTP_HOST_SSL_ENABLED (Boolean String)
    • If ssl is enabled for connecting to Smtp, defaults to false
  • SMTP_RETRIES (Number)
    • Number of retries to make until mailer gives up. defaults to 2
  • SMTP_MX_LOOKUPS_ENABLED (Boolean String)
    • If MX lookups should be done before sending out emails. defaults to false

Database

Plausible uses postgresql as database for storing all the user-data. Use the following the variables to configure it.

  • DATABASE_URL (String)
    • The repo Url as dictated here
  • DATABASE_POOL_SIZE (Number)
    • A default pool size for connecting to the database, defaults to 10, a higher number is recommended for a production system.
  • DATABASE_TLS_ENABLED (Boolean String)
    • A flag that says whether to connect to the database via TLS, read here

For performance reasons, all the analytics events are stored in clickhouse:

  • CLICKHOUSE_DATABASE_HOST (String)
  • CLICKHOUSE_DATABASE_NAME (String)
  • CLICKHOUSE_DATABASE_USER (String)
  • CLICKHOUSE_DATABASE_PASSWORD (String)
  • CLICKHOUSE_DATABASE_POOLSIZE (Number)
    • A default pool size for connecting to the database, defaults to 10, a higher number is recommended for a production system.

External Services

  • Google Client
    • GOOGLE_CLIENT_ID
    • GOOGLE_CLIENT_SECRET
  • Sentry
    • SENTRY_DSN
  • Paddle
    • PADDLE_VENDOR_AUTH_CODE
  • PostMark, only in case of Bamboo.PostmarkAdapter mail adapter.
    • POSTMARK_API_KEY

Apart from these, there are also the following integrations

  • Twitter
    • TWITTER_CONSUMER_KEY
    • TWITTER_CONSUMER_SECRET
    • TWITTER_ACCESS_TOKEN
    • TWITTER_ACCESS_TOKEN_SECRET
  • Slack
    • SLACK_WEBHOOK