Selhosted version Improvements and additional features (#209)

* first commit with test and compile job Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * adding 'prepare' stage Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * updated ci script to include "test" compile phase Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * adding environment variables for connecting to postgresql Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * updated ci config for postgres Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * using non-alpine version of elixir Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * re-using the 'compile' artifacts and added explict env variables for testing Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * removing redundant deps fetching from common code Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * formatting using mix.format -- beware no-code changes! Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * added release config Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * adding consistent env variable for Database Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * more cleaning up of environment variables Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * Adding releases config for enabling releases Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * cleaning up env configs Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * Cleaned up config and prepared config for releases Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * updated CI script with new config for test Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * Added Dockerfile for creating production docker image Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * Adding "docker" build job yay! Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * using non-slim version of debian and installing webpack Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * Adding overlays for migrations on releases Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * restricting the docker built to master branch only Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * typo fix Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * adding "Hosting.md" to explain hosting instructions Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * removed the default comments Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * Added documentation related to env variables * updated documentation and fixed typo Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * updated documentation * Bumping up elixir version as `overlays` are only supported in latest version read release notes: https://github.com/elixir-lang/elixir/releases/tag/v1.10.0 Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * Adding tarball assembly during release Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * updated HOSTING.md Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * Added support for db migration Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * minor corrections Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * initializing admin user Admin user has been added in the "migration" phase. A default user is automatically created in the process. One can provide the related env variables, else a new one will be automatically created for you. Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * Initial base domain update - phase#1 These changes are only meant for correct operating it under self-hosting. There are many other cosmetic changes, that require updates to email, site and other places where the original website and author is used. Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * Using dedicated config variable `base_domain` instead Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * adding base_domain to releases config Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * removing the dedicated config "base_domain", relying on endpoint host Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * Removed the usage of "Mix" in code! It is bad practice to use "mix" module inside the code as in actual release this module is unavailable. Replacing this with a config environment variable Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * Added support for SMTP via Bamboo Smtp Adapter Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * Capturing SMTP errors via Sentry Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * Minor updates Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * Adding junit formatter -- useful for generating test reports Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * adding documentation for default user * Resolve "Gitlab Adoption: Add supported services in "Security & Compliance"" * bumping up the debian version to fix issues fixing some vulnerabilities identified by the scanning tools * More updates for self-hosting Changes in most of the places to suit self-hosting. Although, there are some which have been left-off. Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * quick-dirty-fix! * bumping up the db connect timeout Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * bumping up the db connect timeout Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * bumping up the db connect timeout Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * bumping up timeout - skipping MRs :-/ * removing restrictions on watching for changes this stuff isn't working * Update HOSTING.md * renamed the module name * reverting formatting-whitespace changes Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * reverting the name to release Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * adding docker-compose.yml and related instructions Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * using `plausible_url` instead of assuming `https` this is because, it is much to test in local dev machines and in most cases there's already a layer above which is capable for `https` termination and http -> https upgrade Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * WIP: merging changes from upstream Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * wip: more changes * Pushing in changes from upstream Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * changes to ci for testing Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * cleaning up and finishing clickhouse integration Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * updating readme with hosting details * removing deleted files from upstream * minor config adjustments Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * formatting changes Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * changing the connection strategy for clickhouse during release since clickhouse integration doesn't have an ecto support, we need to prepare the db _before_ the clickhouse migration. One workaround is to connect to a default db on init and then create a db Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * formatting * cleanup and added separated migration to setup * Big improvements to selfhosting - added ability for disabling - authentication completely - registration - landing page - formatting cleanups * Big improvements to selfhosting - added ability for disabling - authentication completely - registration - landing page - formatting cleanups * changing smtp auth to optional Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * removed stale templates and permanently removed landing page Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * removed stale templates and permanently removed landing page Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * removed stale templates and permanently removed landing page Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * WIP Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * fixes form upstream merge Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * added disabling subscription for selfhosted version Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * updated doc Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * formatting Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * Remove reference to file that doesn't exist Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * do not show direct traffic if there's no data Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * addressing PR comments Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me> * formatting Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>
2024-12-22 17:11:36 +03:00 · 2020-07-21 08:58:00 +02:00 · 2020-07-21 08:58:00 +02:00 · f7b37fe9ea
commit f7b37fe9ea
parent 7d0c533a9f
32 changed files with 323 additions and 155 deletions
--- a/.gitignore
+++ b/.gitignore
@ -46,3 +46,9 @@ npm-debug.log
 # Ignore Elixir Language Server files
 .elixir_ls
 plausible-report.xml
+
+*.sh
+.idea
+*.iml
+*.log
+*.code-workspace
--- a/HOSTING.md
+++ b/HOSTING.md
@ -116,6 +116,8 @@ Following are the variables that can be used to configure the availability of th
    - Note: This option is **not recommended** for production deployments.
 - DISABLE_REGISTRATION
  - Disables registration of new users, keep your admin credentials handy ;)  _defaults to `false`_
+- DISABLE_SUBSCRIPTION
+  - Disables changing of subscription and removes the trial notice banner (use with caution!) _defaults to `false`_

 ### Default User Generation
 For self-hosting, a default user is generated during the [Database Migration](#Database Migration) to access Plausible. To be noted that, a default user is a user whose trial period expires in 100 Years ;).
@ -154,7 +156,7 @@ In case of `Bamboo.SMTPAdapter` you need to supply the following variables:

 ### Database

-Plausible uses postgresql as database for storing all the user-data. Use the following the variables to configure it.
+Plausible uses [postgresql as database](https://www.tutorialspoint.com/postgresql/postgresql_environment.htm) for storing all the user-data. Use the following the variables to configure it.

 - DATABASE_URL (*String*)
    - The repo Url as dictated [here](https://hexdocs.pm/ecto/Ecto.Repo.html#module-urls)
@ -163,7 +165,7 @@ Plausible uses postgresql as database for storing all the user-data. Use the fol
 - DATABASE_TLS_ENABLED (*Boolean String*)
    - A flag that says whether to connect to the database via TLS, read [here](https://www.postgresql.org/docs/10/ssl-tcp.html)

-For performance reasons, all the analytics events are stored in clickhouse:
+For performance reasons, all the analytics events are stored in [clickhouse](https://clickhouse.tech/docs/en/getting-started/tutorial/):

 - CLICKHOUSE_DATABASE_HOST (*String*)
 - CLICKHOUSE_DATABASE_NAME (*String*)
--- a/assets/js/dashboard/stats/modals/referrers.js
+++ b/assets/js/dashboard/stats/modals/referrers.js
@ -23,7 +23,7 @@ class ReferrersModal extends React.Component {
    } else {
      const include = this.showExtra() ? 'bounce_rate,visit_duration' : null

-      api.get(`/api/stats/${encodeURIComponent(this.props.site.domain)}/referrers`, this.state.query, {limit: 100, include: include})
+      api.get(`/api/stats/${encodeURIComponent(this.props.site.domain)}/referrers`, this.state.query, { limit: 100, include: include, show_noref: true})
        .then((res) => this.setState({loading: false, referrers: res}))
    }
  }
--- a/assets/package-lock.json
+++ b/assets/package-lock.json
@ -6779,7 +6779,7 @@
    },
    "pretty-hrtime": {
      "version": "1.0.3",
-      "resolved": "http://registry.npmjs.org/pretty-hrtime/-/pretty-hrtime-1.0.3.tgz",
+      "resolved": "https://registry.npmjs.org/pretty-hrtime/-/pretty-hrtime-1.0.3.tgz",
      "integrity": "sha1-t+PqQkNaTJsnWdmeDyAesZWALuE="
    },
    "private": {
@ -7752,7 +7752,7 @@
    },
    "strip-eof": {
      "version": "1.0.0",
-      "resolved": "http://registry.npmjs.org/strip-eof/-/strip-eof-1.0.0.tgz",
+      "resolved": "https://registry.npmjs.org/strip-eof/-/strip-eof-1.0.0.tgz",
      "integrity": "sha1-u0P/VZim6wXYm1n80SnJgzE2Br8="
    },
    "stylehacks": {
--- a/config/config.exs
+++ b/config/config.exs
@ -12,6 +12,7 @@ disable_auth = String.to_existing_atom(System.get_env("DISABLE_AUTH", "false"))

 config :plausible, :selfhost,
  disable_authentication: disable_auth,
+  disable_subscription: String.to_existing_atom(System.get_env("DISABLE_SUBSCRIPTION", "false")),
  disable_registration:
    if(disable_auth,
      do: true,
@ -98,7 +99,7 @@ cron_enabled = String.to_existing_atom(System.get_env("CRON_ENABLED", "false"))

 base_cron = [
  # Daily at midnight
-  {"0 0 * * *", Plausible.Workers.RotateSalts},
+  {"0 0 * * *", Plausible.Workers.RotateSalts}
 ]

 extra_cron = [
@ -117,6 +118,7 @@ extra_cron = [
 ]

 base_queues = [rotate_salts: 1]
+
 extra_queues = [
  provision_ssl_certificates: 1,
  fetch_tweets: 1,
--- a/config/releases.exs
+++ b/config/releases.exs
@ -63,6 +63,7 @@ config :plausible,

 config :plausible, :selfhost,
  disable_authentication: disable_auth,
+  disable_subscription: String.to_existing_atom(System.get_env("DISABLE_SUBSCRIPTION", "false")),
  disable_registration:
    if(!disable_auth,
      do: String.to_existing_atom(System.get_env("DISABLE_REGISTRATION", "false")),
@ -148,7 +149,7 @@ config :plausible, :custom_domain_server,

 base_cron = [
  # Daily at midnight
-  {"0 0 * * *", Plausible.Workers.RotateSalts},
+  {"0 0 * * *", Plausible.Workers.RotateSalts}
 ]

 extra_cron = [
@ -167,6 +168,7 @@ extra_cron = [
 ]

 base_queues = [rotate_salts: 1]
+
 extra_queues = [
  provision_ssl_certificates: 1,
  fetch_tweets: 1,
--- a/lib/plausible/billing/billing.ex
+++ b/lib/plausible/billing/billing.ex
@ -102,9 +102,11 @@ defmodule Plausible.Billing do

  defp subscription_is_active?(%Subscription{status: "active"}), do: true
  defp subscription_is_active?(%Subscription{status: "past_due"}), do: true
+
  defp subscription_is_active?(%Subscription{status: "deleted"} = subscription) do
    subscription.next_bill_date && !Timex.before?(subscription.next_bill_date, Timex.today())
  end
+
  defp subscription_is_active?(_), do: false

  def on_trial?(user), do: trial_days_left(user) >= 0
--- a/lib/plausible/clickhouse.ex
+++ b/lib/plausible/clickhouse.ex
@ -14,8 +14,13 @@ defmodule Plausible.Clickhouse do
    delete_events = "ALTER TABLE events DELETE WHERE domain = ?"
    delete_sessions = "ALTER TABLE sessions DELETE WHERE domain = ?"

-    Clickhousex.query!(:clickhouse, delete_events, [site.domain], log: {Plausible.Clickhouse, :log, []})
-    Clickhousex.query!(:clickhouse, delete_sessions, [site.domain], log: {Plausible.Clickhouse, :log, []})
+    Clickhousex.query!(:clickhouse, delete_events, [site.domain],
+      log: {Plausible.Clickhouse, :log, []}
+    )
+
+    Clickhousex.query!(:clickhouse, delete_sessions, [site.domain],
+      log: {Plausible.Clickhouse, :log, []}
+    )
  end

  def insert_events(events) do
--- a/lib/plausible/session/salts.ex
+++ b/lib/plausible/session/salts.ex
@ -3,19 +3,27 @@ defmodule Plausible.Session.Salts do
  use Plausible.Repo

  def start_link(_opts) do
-    Agent.start_link(fn ->
+    Agent.start_link(
+      fn ->
        clean_old_salts()
-      salts = Repo.all(from s in "salts", select: s.salt, order_by: [desc: s.inserted_at], limit: 2)
+
+        salts =
+          Repo.all(from s in "salts", select: s.salt, order_by: [desc: s.inserted_at], limit: 2)
+
        case salts do
          [current, prev] ->
            %{previous: prev, current: current}
+
          [current] ->
            %{previous: nil, current: current}
+
          [] ->
            new = generate_and_persist_new_salt()
            %{previous: nil, current: new}
        end
-    end, name: __MODULE__)
+      end,
+      name: __MODULE__
+    )
  end

  def fetch() do
@ -41,6 +49,8 @@ defmodule Plausible.Session.Salts do
  end

  defp clean_old_salts() do
-    Repo.delete_all(from s in "salts", where: s.inserted_at < fragment("now() - '48 hours'::interval"))
+    Repo.delete_all(
+      from s in "salts", where: s.inserted_at < fragment("now() - '48 hours'::interval")
+    )
  end
 end
--- a/lib/plausible/stats/clickhouse.ex
+++ b/lib/plausible/stats/clickhouse.ex
@ -2,6 +2,7 @@ defmodule Plausible.Stats.Clickhouse do
  use Plausible.Repo
  alias Plausible.Stats.Query
  alias Plausible.Clickhouse
+  @no_ref "Direct Traffic"

  def compare_pageviews_and_visitors(site, query, {pageviews, visitors}) do
    query = Query.shift_back(query)
@ -256,8 +257,8 @@ defmodule Plausible.Stats.Clickhouse do
    end)
  end

-  def top_referrers(site, query, limit, include) do
-    q =
+  def top_referrers(site, query, limit \\ 5, show_noref \\ false, include \\ []) do
+    referrers =
      from(s in base_session_query(site, query),
        group_by: s.referrer_source,
        where: s.referrer_source != "",
@ -265,9 +266,10 @@ defmodule Plausible.Stats.Clickhouse do
        limit: ^limit
      )

-    q = if "bounce_rate" in include do
+    referrers =
+      if "bounce_rate" in include do
        from(
-          s in q,
+          s in referrers,
          select:
            {fragment("? as name", s.referrer_source), fragment("any(?) as url", s.referrer),
             fragment("uniq(user_id) as count"),
@ -276,17 +278,35 @@ defmodule Plausible.Stats.Clickhouse do
        )
      else
        from(
-          s in q,
+          s in referrers,
          select:
            {fragment("? as name", s.referrer_source), fragment("any(?) as url", s.referrer),
             fragment("uniq(user_id) as count")}
        )
      end

-    Clickhouse.all(q)
+    referrers =
+      Clickhouse.all(referrers)
      |> Enum.map(fn ref ->
        Map.update(ref, "url", nil, fn url -> url && URI.parse("http://" <> url).host end)
      end)
+
+    show_noref = if length(referrers) == 0, do: true, else: show_noref
+
+    if show_noref do
+      no_referrers =
+        Clickhouse.all(
+          from e in base_session_query(site, query),
+            select:
+              {fragment("? as name", @no_ref), fragment("any(?) as url", e.referrer),
+               fragment("uniq(user_id) as count")},
+            where: e.referrer_source == ""
+        )
+
+      if no_referrers |> hd |> Map.get("count") > 0, do: referrers ++ no_referrers, else: []
+    else
+      referrers
+    end
  end

  def visitors_from_referrer(site, query, referrer) do
@ -320,7 +340,20 @@ defmodule Plausible.Stats.Clickhouse do
  end

  def referrer_drilldown(site, query, referrer, include \\ []) do
-    q = from(
+    referrer = if referrer == @no_ref, do: "", else: referrer
+
+    referring_urls =
+      Clickhouse.all(
+        from e in base_session_query(site, query),
+          select: {fragment("? as name", e.referrer), fragment("uniq(user_id) as count")},
+          group_by: e.referrer,
+          where: e.referrer_source == ^referrer,
+          order_by: [desc: fragment("count")],
+          limit: 100
+      )
+
+    q =
+      from(
        s in base_session_query(site, query),
        group_by: s.referrer,
        where: s.referrer_source == ^referrer,
@ -328,17 +361,19 @@ defmodule Plausible.Stats.Clickhouse do
        limit: 100
      )

-    q = if "bounce_rate" in include do
+    q =
+      if "bounce_rate" in include do
        from(
          s in q,
          select:
-            {fragment("? as name", s.referrer),
-             fragment("uniq(user_id) as count"),
+            {fragment("? as name", s.referrer), fragment("uniq(user_id) as count"),
             fragment("round(sum(is_bounce * sign) / sum(sign) * 100) as bounce_rate"),
             fragment("round(avg(duration * sign)) as visit_duration")}
        )
      else
-        from(s in q, select: {fragment("? as name", s.referrer), fragment("uniq(user_id) as count")})
+        from(s in q,
+          select: {fragment("? as name", s.referrer), fragment("uniq(user_id) as count")}
+        )
      end

    referring_urls = Clickhouse.all(q)
@ -391,20 +426,26 @@ defmodule Plausible.Stats.Clickhouse do
  end

  def top_pages(site, query, limit, include) do
-    q = from(
+    q =
+      from(
        e in base_query(site, query),
        group_by: e.pathname,
        order_by: [desc: fragment("count")],
        limit: ^limit
      )

-    q = if "unique_visitors" in include do
+    q =
+      if "unique_visitors" in include do
        from(
          e in q,
-        select: {fragment("? as name", e.pathname), fragment("count(?) as count", e.pathname), fragment("uniq(?) as unique_visitors", e.user_id)}
+          select:
+            {fragment("? as name", e.pathname), fragment("count(?) as count", e.pathname),
+             fragment("uniq(?) as unique_visitors", e.user_id)}
        )
      else
-      from(e in q, select: {fragment("? as name", e.pathname), fragment("count(?) as count", e.pathname)})
+        from(e in q,
+          select: {fragment("? as name", e.pathname), fragment("count(?) as count", e.pathname)}
+        )
      end

    pages = Clickhouse.all(q)
--- a/lib/plausible_release.ex
+++ b/lib/plausible_release.ex
@ -37,6 +37,8 @@ defmodule Plausible.Release do
    prepare()
    Enum.each(repos(), &run_migrations_for/1)
    init_admin()
+    prepare_clickhouse()
+    run_migrations_for_ch()
    IO.puts("Migrations successful!")
  end

@ -44,7 +46,6 @@ defmodule Plausible.Release do
    prepare()
    # Run seed script
    Enum.each(repos(), &run_seeds_for/1)
-
    # Signal shutdown
    IO.puts("Success!")
  end
@ -52,6 +53,8 @@ defmodule Plausible.Release do
  def createdb do
    prepare()
    do_create_db()
+    prepare_clickhouse(:default_db)
+    do_create_ch_db()
    IO.puts("Creation of Db successful!")
  end

@ -117,23 +120,11 @@ defmodule Plausible.Release do
    {:ok, _, _} = Ecto.Migrator.with_repo(repo, &Ecto.Migrator.run(&1, :up, all: true))
  end

-  defp do_create_db do
-    for repo <- repos() do
-      :ok = ensure_repo_created(repo)
-    end
-
-    do_create_ch_db()
-  end
-
-  defp do_create_ch_db() do
-    db_to_create = Keyword.get(Application.get_env(:plausible, :clickhouse), :database)
-
-    IO.puts("create #{inspect(db_to_create)} clickhouse database/tables if it doesn't exist")
-
-    Clickhousex.query(:clickhouse, "CREATE DATABASE IF NOT EXISTS #{db_to_create}", [])
+  defp run_migrations_for_ch() do
+    db = Keyword.get(Application.get_env(:plausible, :clickhouse), :database)

    tb_events = """
-    CREATE TABLE IF NOT EXISTS #{db_to_create}.events (
+    CREATE TABLE IF NOT EXISTS #{db}.events (
      timestamp DateTime,
      name String,
      domain String,
@ -156,7 +147,7 @@ defmodule Plausible.Release do
    Clickhousex.query(:clickhouse, tb_events, [])

    tb_sessions = """
-    CREATE TABLE IF NOT EXISTS #{db_to_create}.sessions (
+    CREATE TABLE IF NOT EXISTS #{db}.sessions (
      session_id UInt64,
      sign Int8,
      domain String,
@ -185,6 +176,18 @@ defmodule Plausible.Release do
    Clickhousex.query(:clickhouse, tb_sessions, [])
  end

+  defp do_create_db do
+    for repo <- repos() do
+      :ok = ensure_repo_created(repo)
+    end
+  end
+
+  defp do_create_ch_db() do
+    db_to_create = Keyword.get(Application.get_env(:plausible, :clickhouse), :database)
+    IO.puts("create #{inspect(db_to_create)} clickhouse database/tables if it doesn't exist")
+    Clickhousex.query(:clickhouse, "CREATE DATABASE IF NOT EXISTS #{db_to_create}", [])
+  end
+
  defp ensure_repo_created(repo) do
    IO.puts("create #{inspect(repo)} database if it doesn't exist")

@ -208,8 +211,6 @@ defmodule Plausible.Release do
    # Load the code for myapp, but don't start it
    :ok = Application.load(@app)

-    prepare_clickhouse()
-
    IO.puts("Starting dependencies..")
    # Start apps necessary for executing migrations
    Enum.each(@start_apps, &Application.ensure_all_started/1)
@ -219,7 +220,8 @@ defmodule Plausible.Release do
    Enum.each(repos(), & &1.start_link(pool_size: 2))
  end

-  defp prepare_clickhouse do
+  # connect to the default db for creating the required db
+  defp prepare_clickhouse(:default_db) do
    Application.ensure_all_started(:db_connection)
    Application.ensure_all_started(:hackney)

@ -228,7 +230,24 @@ defmodule Plausible.Release do
      port: 8123,
      name: :clickhouse,
      database: "default",
-      hostname: Keyword.get(Application.get_env(:plausible, :clickhouse), :hostname)
+      username: "default",
+      hostname: Keyword.get(Application.get_env(:plausible, :clickhouse), :hostname),
+      password: Keyword.get(Application.get_env(:plausible, :clickhouse), :password)
+    )
+  end
+
+  defp prepare_clickhouse() do
+    Application.ensure_all_started(:db_connection)
+    Application.ensure_all_started(:hackney)
+
+    Clickhousex.start_link(
+      scheme: :http,
+      port: 8123,
+      name: :clickhouse,
+      username: Keyword.get(Application.get_env(:plausible, :clickhouse), :username),
+      database: Keyword.get(Application.get_env(:plausible, :clickhouse), :database),
+      hostname: Keyword.get(Application.get_env(:plausible, :clickhouse), :hostname),
+      password: Keyword.get(Application.get_env(:plausible, :clickhouse), :password)
    )
  end

--- a/lib/plausible_web/controllers/api/external_controller.ex
+++ b/lib/plausible_web/controllers/api/external_controller.ex
@ -29,17 +29,20 @@ defmodule PlausibleWeb.Api.ExternalController do
  end

  def health(conn, _params) do
-    postgres_health = case Ecto.Adapters.SQL.query(Plausible.Repo, "SELECT 1", []) do
+    postgres_health =
+      case Ecto.Adapters.SQL.query(Plausible.Repo, "SELECT 1", []) do
        {:ok, _} -> "ok"
-      e -> "error: #{inspect e}"
+        e -> "error: #{inspect(e)}"
      end

-    clickhouse_health = case Clickhousex.query(:clickhouse, "SELECT 1", []) do
+    clickhouse_health =
+      case Clickhousex.query(:clickhouse, "SELECT 1", []) do
        {:ok, _} -> "ok"
-      e -> "error: #{inspect e}"
+        e -> "error: #{inspect(e)}"
      end

-    status = case {postgres_health, clickhouse_health} do
+    status =
+      case {postgres_health, clickhouse_health} do
        {"ok", "ok"} -> 200
        _ -> 500
      end
@ -47,7 +50,7 @@ defmodule PlausibleWeb.Api.ExternalController do
    put_status(conn, status)
    |> json(%{
      postgres: postgres_health,
-      clickhouse: clickhouse_health,
+      clickhouse: clickhouse_health
    })
  end

--- a/lib/plausible_web/controllers/api/stats_controller.ex
+++ b/lib/plausible_web/controllers/api/stats_controller.ex
@ -27,11 +27,11 @@ defmodule PlausibleWeb.Api.StatsController do
    [
      %{
        name: "Active visitors",
-        count: Stats.current_visitors(site),
+        count: Stats.current_visitors(site)
      },
      %{
        name: "Pageviews (last 30 min)",
-        count: Stats.total_pageviews(site, query),
+        count: Stats.total_pageviews(site, query)
      }
    ]
  end
@ -94,7 +94,11 @@ defmodule PlausibleWeb.Api.StatsController do
        change: percent_change(prev_pageviews, pageviews)
      },
      %{name: "Bounce rate", percentage: bounce_rate, change: change_bounce_rate},
-      %{name: "Visit duration", count: visit_duration, change: percent_change(prev_visit_duration, visit_duration)}
+      %{
+        name: "Visit duration",
+        count: visit_duration,
+        change: percent_change(prev_visit_duration, visit_duration)
+      }
    ]
  end

@ -116,8 +120,8 @@ defmodule PlausibleWeb.Api.StatsController do
    query = Query.from(site.timezone, params)
    include = if params["include"], do: String.split(params["include"], ","), else: []
    limit = if params["limit"], do: String.to_integer(params["limit"])
-
-    json(conn, Stats.top_referrers(site, query, limit || 9, include))
+    show_noref = if params["show_noref"], do: "true", else: false
+    json(conn, Stats.top_referrers(site, query, limit || 9, show_noref, include))
  end

  def referrers_for_goal(conn, params) do
--- a/lib/plausible_web/controllers/auth_controller.ex
+++ b/lib/plausible_web/controllers/auth_controller.ex
@ -236,7 +236,11 @@ defmodule PlausibleWeb.AuthController do

  def user_settings(conn, _params) do
    changeset = Auth.User.changeset(conn.assigns[:current_user])
-    render(conn, "user_settings.html", changeset: changeset, subscription: conn.assigns[:current_user].subscription)
+
+    render(conn, "user_settings.html",
+      changeset: changeset,
+      subscription: conn.assigns[:current_user].subscription
+    )
  end

  def save_settings(conn, %{"user" => user_params}) do
--- a/lib/plausible_web/controllers/site_controller.ex
+++ b/lib/plausible_web/controllers/site_controller.ex
@ -160,8 +160,7 @@ defmodule PlausibleWeb.SiteController do
  end

  def reset_stats(conn, %{"website" => website}) do
-    site =
-      Sites.get_for_user!(conn.assigns[:current_user].id, website)
+    site = Sites.get_for_user!(conn.assigns[:current_user].id, website)
    Plausible.Clickhouse.delete_stats!(site)

    conn
@ -417,7 +416,8 @@ defmodule PlausibleWeb.SiteController do
  end

  def delete_custom_domain(conn, %{"website" => website}) do
-    site = Sites.get_for_user!(conn.assigns[:current_user].id, website)
+    site =
+      Sites.get_for_user!(conn.assigns[:current_user].id, website)
      |> Repo.preload(:custom_domain)

    Repo.delete!(site.custom_domain)
--- a/lib/plausible_web/plugs/auth_plug.ex
+++ b/lib/plausible_web/plugs/auth_plug.ex
@ -14,7 +14,10 @@ defmodule PlausibleWeb.AuthPlug do
      id ->
        user =
          Repo.get_by(Plausible.Auth.User, id: id)
-          |> Repo.preload(subscription: from(s in Plausible.Billing.Subscription, order_by: [desc: s.inserted_at]))
+          |> Repo.preload(
+            subscription:
+              from(s in Plausible.Billing.Subscription, order_by: [desc: s.inserted_at])
+          )

        if user do
          Sentry.Context.set_user_context(%{id: user.id, name: user.name})
--- a/lib/plausible_web/plugs/auto_auth_plug.ex
+++ b/lib/plausible_web/plugs/auto_auth_plug.ex
@ -20,7 +20,6 @@ defmodule PlausibleWeb.AutoAuthPlug do
        Plug.Conn.put_session(conn, :login_dest, conn.request_path)
        |> Phoenix.Controller.redirect(to: "/login")
        |> halt
-
    end
  end
 end
--- a/lib/plausible_web/templates/auth/user_settings.html.eex
+++ b/lib/plausible_web/templates/auth/user_settings.html.eex
@ -1,3 +1,4 @@
+<%= if  !Keyword.fetch!(Application.get_env(:plausible, :selfhost), :disable_subscription) do %>
 <div class="max-w-2xl mx-auto bg-white shadow-md rounded rounded-t-none border-t-2 border-orange-200 px-8 pt-6 pb-8 mt-24">
  <div class="flex justify-between">
    <h2 class="text-xl font-black">Subscription Plan</h2>
@ -86,6 +87,7 @@
      </div>
  <% end %>
 </div>
+<% end %>

 <div class="max-w-2xl mx-auto bg-white shadow-md rounded rounded-t-none border-t-2 border-indigo-100 px-8 pt-6 pb-8 mt-16">
  <h2 class="text-xl font-black">Account settings</h2>
--- a/lib/plausible_web/templates/layout/app.html.eex
+++ b/lib/plausible_web/templates/layout/app.html.eex
@ -25,7 +25,7 @@
            <%= cond do %>
              <% @conn.assigns[:current_user] -> %>
                <ul class="flex">
-                  <%= if @conn.assigns[:current_user].subscription == nil do %>
+                  <%= if  !Keyword.fetch!(Application.get_env(:plausible, :selfhost), :disable_subscription) && @conn.assigns[:current_user].subscription == nil  do %>
                    <li class="mr-6 hidden sm:block">
                      <%= link(trial_notificaton(@conn.assigns[:current_user]), to: "/settings", class: "font-bold text-orange-900 rounded p-2 bg-orange-200", style: "line-height: 40px;") %>
                    </li>
--- a/lib/workers/schedule_email_reports.ex
+++ b/lib/workers/schedule_email_reports.ex
@ -16,10 +16,12 @@ defmodule Plausible.Workers.ScheduleEmailReports do
  end

  defp schedule_weekly_emails() do
-    weekly_jobs = from(
+    weekly_jobs =
+      from(
        j in Oban.Job,
-      where: j.worker == "Plausible.Workers.SendEmailReport"
-      and fragment("(? ->> 'interval')", j.args) == "weekly"
+        where:
+          j.worker == "Plausible.Workers.SendEmailReport" and
+            fragment("(? ->> 'interval')", j.args) == "weekly"
      )

    sites =
@ -28,14 +30,17 @@ defmodule Plausible.Workers.ScheduleEmailReports do
          join: wr in Plausible.Site.WeeklyReport,
          on: wr.site_id == s.id,
          left_join: job in subquery(weekly_jobs),
-        on: fragment("(? -> 'site_id')::int", job.args) == s.id and
+          on:
+            fragment("(? -> 'site_id')::int", job.args) == s.id and
              job.state not in ["completed", "discarded"],
          where: is_nil(job),
          preload: [weekly_report: wr]
      )

    for site <- sites do
-      SendEmailReport.new(%{site_id: site.id, interval: "weekly"}, scheduled_at: monday_9am(site.timezone))
+      SendEmailReport.new(%{site_id: site.id, interval: "weekly"},
+        scheduled_at: monday_9am(site.timezone)
+      )
      |> Oban.insert!()
    end

@ -50,10 +55,12 @@ defmodule Plausible.Workers.ScheduleEmailReports do
  end

  defp schedule_monthly_emails() do
-    monthly_jobs = from(
+    monthly_jobs =
+      from(
        j in Oban.Job,
-      where: j.worker == "Plausible.Workers.SendEmailReport"
-      and fragment("(? ->> 'interval')", j.args) == "monthly"
+        where:
+          j.worker == "Plausible.Workers.SendEmailReport" and
+            fragment("(? ->> 'interval')", j.args) == "monthly"
      )

    sites =
@ -70,7 +77,9 @@ defmodule Plausible.Workers.ScheduleEmailReports do
      )

    for site <- sites do
-      SendEmailReport.new(%{site_id: site.id, interval: "monthly"}, scheduled_at: first_of_month_9am(site.timezone))
+      SendEmailReport.new(%{site_id: site.id, interval: "monthly"},
+        scheduled_at: first_of_month_9am(site.timezone)
+      )
      |> Oban.insert!()
    end

--- a/lib/workers/send_email_report.ex
+++ b/lib/workers/send_email_report.ex
@ -16,12 +16,14 @@ defmodule Plausible.Workers.SendEmailReport do

      send_report(email, site, "Weekly", unsubscribe_link, query)
    end
+
    :ok
  end

  @impl Oban.Worker
  def perform(%{"interval" => "monthly", "site_id" => site_id}, _job) do
    site = Repo.get(Plausible.Site, site_id) |> Repo.preload(:monthly_report)
+
    last_month =
      Timex.now(site.timezone)
      |> Timex.shift(months: -1)
@ -40,6 +42,7 @@ defmodule Plausible.Workers.SendEmailReport do

      send_report(email, site, Timex.format!(last_month, "{Mfull}"), unsubscribe_link, query)
    end
+
    :ok
  end

@ -57,7 +60,8 @@ defmodule Plausible.Workers.SendEmailReport do
    user = Plausible.Auth.find_user_by(email: email)
    login_link = user && Plausible.Sites.is_owner?(user.id, site)

-    template = PlausibleWeb.Email.weekly_report(email, site,
+    template =
+      PlausibleWeb.Email.weekly_report(email, site,
        unique_visitors: unique_visitors,
        change_visitors: change_visitors,
        pageviews: pageviews,
--- a/lib/workers/send_trial_notifications.ex
+++ b/lib/workers/send_trial_notifications.ex
@ -1,8 +1,10 @@
 defmodule Plausible.Workers.SendTrialNotifications do
  use Plausible.Repo
+
  use Oban.Worker,
    queue: :trial_notification_emails,
    max_attempts: 1
+
  require Logger

  @impl Oban.Worker
--- a/test/plausible/billing/billing_test.exs
+++ b/test/plausible/billing/billing_test.exs
@ -78,7 +78,13 @@ defmodule Plausible.BillingTest do

    test "is true for a user with a cancelled subscription IF the billing cycle is complete" do
      user = insert(:user, trial_expiry_date: Timex.shift(Timex.today(), days: -1))
-      insert(:subscription, user: user, status: "deleted", next_bill_date: Timex.shift(Timex.today(), days: -1))
+
+      insert(:subscription,
+        user: user,
+        status: "deleted",
+        next_bill_date: Timex.shift(Timex.today(), days: -1)
+      )
+
      user = Repo.preload(user, :subscription)

      assert Billing.needs_to_upgrade?(user)
--- a/test/plausible_web/controllers/api/external_controller_test.exs
+++ b/test/plausible_web/controllers/api/external_controller_test.exs
@ -380,7 +380,6 @@ defmodule PlausibleWeb.Api.ExternalControllerTest do
    assert event["referrer"] == ""
  end

-
  # Fake data is set up in config/test.exs
  test "looks up the country from the ip address", %{conn: conn} do
    params = %{
--- a/test/plausible_web/controllers/api/stats_controller/pages_test.exs
+++ b/test/plausible_web/controllers/api/stats_controller/pages_test.exs
@ -25,9 +25,24 @@ defmodule PlausibleWeb.Api.StatsController.PagesTest do

      assert json_response(conn, 200) == [
               %{"bounce_rate" => 33.0, "count" => 2, "unique_visitors" => 2, "name" => "/"},
-               %{"bounce_rate" => nil, "count" => 2, "unique_visitors" => 2, "name" => "/register"},
-               %{"bounce_rate" => nil, "count" => 1, "unique_visitors" => 1, "name" => "/contact"},
-               %{"bounce_rate" => nil, "count" => 1, "unique_visitors" => 1, "name" => "/irrelevant"}
+               %{
+                 "bounce_rate" => nil,
+                 "count" => 2,
+                 "unique_visitors" => 2,
+                 "name" => "/register"
+               },
+               %{
+                 "bounce_rate" => nil,
+                 "count" => 1,
+                 "unique_visitors" => 1,
+                 "name" => "/contact"
+               },
+               %{
+                 "bounce_rate" => nil,
+                 "count" => 1,
+                 "unique_visitors" => 1,
+                 "name" => "/irrelevant"
+               }
             ]
    end

--- a/test/plausible_web/controllers/api/stats_controller/referrers_test.exs
+++ b/test/plausible_web/controllers/api/stats_controller/referrers_test.exs
@ -29,7 +29,13 @@ defmodule PlausibleWeb.Api.StatsController.ReferrersTest do
                 "visit_duration" => 50,
                 "url" => "10words.com"
               },
-               %{"name" => "Bing", "count" => 1, "bounce_rate" => 0,  "visit_duration" => 100 ,"url" => ""}
+               %{
+                 "name" => "Bing",
+                 "count" => 1,
+                 "bounce_rate" => 0,
+                 "visit_duration" => 100,
+                 "url" => ""
+               }
             ]
    end

@ -99,7 +105,12 @@ defmodule PlausibleWeb.Api.StatsController.ReferrersTest do
      assert json_response(conn, 200) == %{
               "total_visitors" => 2,
               "referrers" => [
-                 %{"name" => "10words.com/page1", "count" => 2, "bounce_rate" => 50.0, "visit_duration" => 50.0}
+                 %{
+                   "name" => "10words.com/page1",
+                   "count" => 2,
+                   "bounce_rate" => 50.0,
+                   "visit_duration" => 50.0
+                 }
               ]
             }
    end
--- a/test/plausible_web/controllers/site_controller_test.exs
+++ b/test/plausible_web/controllers/site_controller_test.exs
@ -153,7 +153,10 @@ defmodule PlausibleWeb.SiteControllerTest do

    test "updates google auth property", %{conn: conn, user: user, site: site} do
      insert(:google_auth, user: user, site: site)
-      put(conn, "/#{site.domain}/settings/google", %{"google_auth" => %{"property" => "some-new-property.com"}})
+
+      put(conn, "/#{site.domain}/settings/google", %{
+        "google_auth" => %{"property" => "some-new-property.com"}
+      })

      updated_auth = Repo.one(Plausible.Site.GoogleAuth)
      assert updated_auth.property == "some-new-property.com"
--- a/test/plausible_web/plugs/auth_plug_test.exs
+++ b/test/plausible_web/plugs/auth_plug_test.exs
@ -15,6 +15,7 @@ defmodule PlausibleWeb.AuthPlugTest do
  test "looks up current user if they are logged in" do
    user = insert(:user)
    subscription = insert(:subscription, user: user)
+
    conn =
      conn(:get, "/")
      |> init_test_session(%{current_user_id: user.id})
@ -26,7 +27,10 @@ defmodule PlausibleWeb.AuthPlugTest do

  test "looks up the latest subscription" do
    user = insert(:user)
-    _old_subscription = insert(:subscription, user: user, inserted_at: Timex.now() |> Timex.shift(days: -1))
+
+    _old_subscription =
+      insert(:subscription, user: user, inserted_at: Timex.now() |> Timex.shift(days: -1))
+
    subscription = insert(:subscription, user: user, inserted_at: Timex.now())

    conn =
--- a/test/support/clickhouse_setup.ex
+++ b/test/support/clickhouse_setup.ex
@ -287,7 +287,7 @@ defmodule Plausible.Test.ClickhouseSetup do
        referrer_source: "Twitter",
        start: ~N[2019-03-01 02:00:00],
        timestamp: ~N[2019-03-01 02:00:00]
-      },
+      }
    ])
  end
 end
--- a/test/workers/schedule_email_reports_test.exs
+++ b/test/workers/schedule_email_reports_test.exs
@ -15,7 +15,11 @@ defmodule Plausible.Workers.ScheduleEmailReportsTest do

      perform(%{})

-      assert_enqueued worker: SendEmailReport, args: %{site_id: site.id, interval: "weekly"}, scheduled_at: ScheduleEmailReports.monday_9am(site.timezone)
+      assert_enqueued(
+        worker: SendEmailReport,
+        args: %{site_id: site.id, interval: "weekly"},
+        scheduled_at: ScheduleEmailReports.monday_9am(site.timezone)
+      )
    end

    test "does not schedule more than one weekly report at a time" do
@ -33,11 +37,10 @@ defmodule Plausible.Workers.ScheduleEmailReportsTest do
      insert(:weekly_report, site: site, recipients: ["user@email.com"])

      perform(%{})
-      Repo.update_all("oban_jobs", [set: [state: "completed"]])
+      Repo.update_all("oban_jobs", set: [state: "completed"])
      assert Enum.count(all_enqueued(worker: SendEmailReport)) == 0
      perform(%{})
      assert Enum.count(all_enqueued(worker: SendEmailReport)) == 1
-
    end
  end

@ -48,7 +51,11 @@ defmodule Plausible.Workers.ScheduleEmailReportsTest do

      perform(%{})

-      assert_enqueued worker: SendEmailReport, args: %{site_id: site.id, interval: "monthly"}, scheduled_at: ScheduleEmailReports.first_of_month_9am(site.timezone)
+      assert_enqueued(
+        worker: SendEmailReport,
+        args: %{site_id: site.id, interval: "monthly"},
+        scheduled_at: ScheduleEmailReports.first_of_month_9am(site.timezone)
+      )
    end

    test "does not schedule more than one monthly report at a time" do
@ -66,11 +73,10 @@ defmodule Plausible.Workers.ScheduleEmailReportsTest do
      insert(:monthly_report, site: site, recipients: ["user@email.com"])

      perform(%{})
-      Repo.update_all("oban_jobs", [set: [state: "completed"]])
+      Repo.update_all("oban_jobs", set: [state: "completed"])
      assert Enum.count(all_enqueued(worker: SendEmailReport)) == 0
      perform(%{})
      assert Enum.count(all_enqueued(worker: SendEmailReport)) == 1
-
    end
  end
 end
--- a/test/workers/send_email_report_test.exs
+++ b/test/workers/send_email_report_test.exs
@ -31,7 +31,12 @@ defmodule Plausible.Workers.SendEmailReportTest do
    test "sends monthly report to all recipients" do
      site = insert(:site, domain: "test-site.com", timezone: "US/Eastern")
      insert(:monthly_report, site: site, recipients: ["user@email.com", "user2@email.com"])
-      last_month = Timex.now(site.timezone) |> Timex.shift(months: -1) |> Timex.beginning_of_month() |> Timex.format!("{Mfull}")
+
+      last_month =
+        Timex.now(site.timezone)
+        |> Timex.shift(months: -1)
+        |> Timex.beginning_of_month()
+        |> Timex.format!("{Mfull}")

      perform(%{"site_id" => site.id, "interval" => "monthly"})