Selhosted version Improvements and additional features (#209)

* first commit with test and compile job

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding 'prepare' stage

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated ci script to include "test" compile phase

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding environment variables for connecting to postgresql

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated ci config for postgres

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* using non-alpine version of elixir

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* re-using the 'compile' artifacts and added explict env variables for testing

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removing redundant deps fetching from common code

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* formatting using mix.format -- beware no-code changes!

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* added release config

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding consistent env variable for Database

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* more cleaning up of environment variables

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding releases config for enabling releases

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* cleaning up env configs

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Cleaned up config and prepared config for releases

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated CI script with new config for test

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added Dockerfile for creating production docker image

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding "docker" build job yay!

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* using non-slim version of debian and installing webpack

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding overlays for migrations on releases

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* restricting the docker built to master branch only

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* typo fix

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding "Hosting.md" to explain hosting instructions

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removed the default comments

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added documentation related to env variables

* updated documentation and fixed typo

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated documentation

* Bumping up elixir version as `overlays` are only supported in latest version

read release notes: https://github.com/elixir-lang/elixir/releases/tag/v1.10.0

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding tarball assembly during release

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated HOSTING.md

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added support for db migration

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* minor corrections

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* initializing admin user

Admin user has been added in the "migration" phase. A default user is automatically created in the process. One can provide the related env variables, else a new one will be automatically created for you.

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Initial base domain update - phase#1

These changes are only meant for correct operating it under self-hosting. There are many other cosmetic changes, that require updates to email, site and other places where the original website and author is used.

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Using dedicated config variable `base_domain` instead

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding base_domain to releases config

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removing the dedicated config "base_domain", relying on endpoint host

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Removed the usage of "Mix" in code!

It is bad practice to use "mix" module inside the code as in actual release this module is unavailable. Replacing this with a config environment variable

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added support for SMTP via Bamboo Smtp Adapter

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Capturing SMTP errors via Sentry

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Minor updates

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding junit formatter -- useful for generating test reports

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding documentation for default user

* Resolve "Gitlab Adoption: Add supported services in "Security & Compliance""

* bumping up the debian version to fix issues

fixing some vulnerabilities identified by the scanning tools

* More updates for self-hosting

Changes in most of the places to suit self-hosting. Although, there are some which have been left-off.

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* quick-dirty-fix!

* bumping up the db connect timeout

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* bumping up the db connect timeout

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* bumping up the db connect timeout

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* bumping up timeout - skipping MRs :-/

* removing restrictions on watching for changes

this stuff isn't working

* Update HOSTING.md

* renamed the module name

* reverting formatting-whitespace changes

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* reverting the name to release

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding docker-compose.yml and related instructions

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* using `plausible_url` instead of assuming `https`

this is because, it is much to test in local dev machines and in most cases there's already a layer above which is capable for `https` termination and http -> https upgrade

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* WIP: merging changes from upstream

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* wip: more changes

* Pushing in changes from upstream

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* changes to ci for testing

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* cleaning up and finishing clickhouse integration

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updating readme with hosting details

* removing deleted files from upstream

* minor config adjustments

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* formatting changes

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* changing the connection strategy for clickhouse during release

since clickhouse integration doesn't have an ecto support, we need to prepare the db _before_ the clickhouse migration. One workaround is to connect to a default db on init and then create a db

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* formatting

* cleanup and added separated migration to setup

* Big improvements to selfhosting

- added ability for disabling
  - authentication completely
  - registration
  - landing page

- formatting cleanups

* Big improvements to selfhosting

- added ability for disabling
  - authentication completely
  - registration
  - landing page

- formatting cleanups

* changing smtp auth  to optional

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removed stale templates and permanently removed landing page

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removed stale templates and permanently removed landing page

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removed stale templates and permanently removed landing page

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* WIP

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* fixes form upstream merge

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* added disabling subscription for selfhosted version

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated doc

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* formatting

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Remove reference to file that doesn't exist

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* do not show direct traffic if there's no data

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* addressing PR comments

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* formatting

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>
This commit is contained in:
Chandra Tungathurthi 2020-07-21 08:58:00 +02:00 committed by GitHub
parent 7d0c533a9f
commit f7b37fe9ea
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
32 changed files with 323 additions and 155 deletions

6
.gitignore vendored
View File

@ -46,3 +46,9 @@ npm-debug.log
# Ignore Elixir Language Server files
.elixir_ls
plausible-report.xml
*.sh
.idea
*.iml
*.log
*.code-workspace

View File

@ -116,6 +116,8 @@ Following are the variables that can be used to configure the availability of th
- Note: This option is **not recommended** for production deployments.
- DISABLE_REGISTRATION
- Disables registration of new users, keep your admin credentials handy ;) _defaults to `false`_
- DISABLE_SUBSCRIPTION
- Disables changing of subscription and removes the trial notice banner (use with caution!) _defaults to `false`_
### Default User Generation
For self-hosting, a default user is generated during the [Database Migration](#Database Migration) to access Plausible. To be noted that, a default user is a user whose trial period expires in 100 Years ;).
@ -154,7 +156,7 @@ In case of `Bamboo.SMTPAdapter` you need to supply the following variables:
### Database
Plausible uses postgresql as database for storing all the user-data. Use the following the variables to configure it.
Plausible uses [postgresql as database](https://www.tutorialspoint.com/postgresql/postgresql_environment.htm) for storing all the user-data. Use the following the variables to configure it.
- DATABASE_URL (*String*)
- The repo Url as dictated [here](https://hexdocs.pm/ecto/Ecto.Repo.html#module-urls)
@ -163,7 +165,7 @@ Plausible uses postgresql as database for storing all the user-data. Use the fol
- DATABASE_TLS_ENABLED (*Boolean String*)
- A flag that says whether to connect to the database via TLS, read [here](https://www.postgresql.org/docs/10/ssl-tcp.html)
For performance reasons, all the analytics events are stored in clickhouse:
For performance reasons, all the analytics events are stored in [clickhouse](https://clickhouse.tech/docs/en/getting-started/tutorial/):
- CLICKHOUSE_DATABASE_HOST (*String*)
- CLICKHOUSE_DATABASE_NAME (*String*)

View File

@ -23,7 +23,7 @@ class ReferrersModal extends React.Component {
} else {
const include = this.showExtra() ? 'bounce_rate,visit_duration' : null
api.get(`/api/stats/${encodeURIComponent(this.props.site.domain)}/referrers`, this.state.query, {limit: 100, include: include})
api.get(`/api/stats/${encodeURIComponent(this.props.site.domain)}/referrers`, this.state.query, { limit: 100, include: include, show_noref: true})
.then((res) => this.setState({loading: false, referrers: res}))
}
}

View File

@ -6779,7 +6779,7 @@
},
"pretty-hrtime": {
"version": "1.0.3",
"resolved": "http://registry.npmjs.org/pretty-hrtime/-/pretty-hrtime-1.0.3.tgz",
"resolved": "https://registry.npmjs.org/pretty-hrtime/-/pretty-hrtime-1.0.3.tgz",
"integrity": "sha1-t+PqQkNaTJsnWdmeDyAesZWALuE="
},
"private": {
@ -7752,7 +7752,7 @@
},
"strip-eof": {
"version": "1.0.0",
"resolved": "http://registry.npmjs.org/strip-eof/-/strip-eof-1.0.0.tgz",
"resolved": "https://registry.npmjs.org/strip-eof/-/strip-eof-1.0.0.tgz",
"integrity": "sha1-u0P/VZim6wXYm1n80SnJgzE2Br8="
},
"stylehacks": {

View File

@ -12,6 +12,7 @@ disable_auth = String.to_existing_atom(System.get_env("DISABLE_AUTH", "false"))
config :plausible, :selfhost,
disable_authentication: disable_auth,
disable_subscription: String.to_existing_atom(System.get_env("DISABLE_SUBSCRIPTION", "false")),
disable_registration:
if(disable_auth,
do: true,
@ -98,7 +99,7 @@ cron_enabled = String.to_existing_atom(System.get_env("CRON_ENABLED", "false"))
base_cron = [
# Daily at midnight
{"0 0 * * *", Plausible.Workers.RotateSalts},
{"0 0 * * *", Plausible.Workers.RotateSalts}
]
extra_cron = [
@ -117,6 +118,7 @@ extra_cron = [
]
base_queues = [rotate_salts: 1]
extra_queues = [
provision_ssl_certificates: 1,
fetch_tweets: 1,

View File

@ -34,7 +34,7 @@ ck_host = System.get_env("CLICKHOUSE_DATABASE_HOST", "localhost")
ck_db = System.get_env("CLICKHOUSE_DATABASE_NAME", "plausible_dev")
ck_db_user = System.get_env("CLICKHOUSE_DATABASE_USER")
ck_db_pwd = System.get_env("CLICKHOUSE_DATABASE_PASSWORD")
ck_db_pool = String.to_integer(System.get_env("CLICKHOUSE_DATABASE_POOLSIZE", "10"))
ck_db_pool = String.to_integer(System.get_env("CLICKHOUSE_DATABASE_POOLSIZE", "10"))
### Mandatory params End
sentry_dsn = System.get_env("SENTRY_DSN")
@ -63,6 +63,7 @@ config :plausible,
config :plausible, :selfhost,
disable_authentication: disable_auth,
disable_subscription: String.to_existing_atom(System.get_env("DISABLE_SUBSCRIPTION", "false")),
disable_registration:
if(!disable_auth,
do: String.to_existing_atom(System.get_env("DISABLE_REGISTRATION", "false")),
@ -148,7 +149,7 @@ config :plausible, :custom_domain_server,
base_cron = [
# Daily at midnight
{"0 0 * * *", Plausible.Workers.RotateSalts},
{"0 0 * * *", Plausible.Workers.RotateSalts}
]
extra_cron = [
@ -167,6 +168,7 @@ extra_cron = [
]
base_queues = [rotate_salts: 1]
extra_queues = [
provision_ssl_certificates: 1,
fetch_tweets: 1,

View File

@ -102,9 +102,11 @@ defmodule Plausible.Billing do
defp subscription_is_active?(%Subscription{status: "active"}), do: true
defp subscription_is_active?(%Subscription{status: "past_due"}), do: true
defp subscription_is_active?(%Subscription{status: "deleted"} = subscription) do
subscription.next_bill_date && !Timex.before?(subscription.next_bill_date, Timex.today())
end
defp subscription_is_active?(_), do: false
def on_trial?(user), do: trial_days_left(user) >= 0

View File

@ -14,8 +14,13 @@ defmodule Plausible.Clickhouse do
delete_events = "ALTER TABLE events DELETE WHERE domain = ?"
delete_sessions = "ALTER TABLE sessions DELETE WHERE domain = ?"
Clickhousex.query!(:clickhouse, delete_events, [site.domain], log: {Plausible.Clickhouse, :log, []})
Clickhousex.query!(:clickhouse, delete_sessions, [site.domain], log: {Plausible.Clickhouse, :log, []})
Clickhousex.query!(:clickhouse, delete_events, [site.domain],
log: {Plausible.Clickhouse, :log, []}
)
Clickhousex.query!(:clickhouse, delete_sessions, [site.domain],
log: {Plausible.Clickhouse, :log, []}
)
end
def insert_events(events) do

View File

@ -3,19 +3,27 @@ defmodule Plausible.Session.Salts do
use Plausible.Repo
def start_link(_opts) do
Agent.start_link(fn ->
clean_old_salts()
salts = Repo.all(from s in "salts", select: s.salt, order_by: [desc: s.inserted_at], limit: 2)
case salts do
[current, prev] ->
%{previous: prev, current: current}
[current] ->
%{previous: nil, current: current}
[] ->
new = generate_and_persist_new_salt()
%{previous: nil, current: new}
end
end, name: __MODULE__)
Agent.start_link(
fn ->
clean_old_salts()
salts =
Repo.all(from s in "salts", select: s.salt, order_by: [desc: s.inserted_at], limit: 2)
case salts do
[current, prev] ->
%{previous: prev, current: current}
[current] ->
%{previous: nil, current: current}
[] ->
new = generate_and_persist_new_salt()
%{previous: nil, current: new}
end
end,
name: __MODULE__
)
end
def fetch() do
@ -41,6 +49,8 @@ defmodule Plausible.Session.Salts do
end
defp clean_old_salts() do
Repo.delete_all(from s in "salts", where: s.inserted_at < fragment("now() - '48 hours'::interval"))
Repo.delete_all(
from s in "salts", where: s.inserted_at < fragment("now() - '48 hours'::interval")
)
end
end

View File

@ -2,6 +2,7 @@ defmodule Plausible.Stats.Clickhouse do
use Plausible.Repo
alias Plausible.Stats.Query
alias Plausible.Clickhouse
@no_ref "Direct Traffic"
def compare_pageviews_and_visitors(site, query, {pageviews, visitors}) do
query = Query.shift_back(query)
@ -256,8 +257,8 @@ defmodule Plausible.Stats.Clickhouse do
end)
end
def top_referrers(site, query, limit, include) do
q =
def top_referrers(site, query, limit \\ 5, show_noref \\ false, include \\ []) do
referrers =
from(s in base_session_query(site, query),
group_by: s.referrer_source,
where: s.referrer_source != "",
@ -265,9 +266,10 @@ defmodule Plausible.Stats.Clickhouse do
limit: ^limit
)
q = if "bounce_rate" in include do
referrers =
if "bounce_rate" in include do
from(
s in q,
s in referrers,
select:
{fragment("? as name", s.referrer_source), fragment("any(?) as url", s.referrer),
fragment("uniq(user_id) as count"),
@ -276,17 +278,35 @@ defmodule Plausible.Stats.Clickhouse do
)
else
from(
s in q,
s in referrers,
select:
{fragment("? as name", s.referrer_source), fragment("any(?) as url", s.referrer),
fragment("uniq(user_id) as count")}
{fragment("? as name", s.referrer_source), fragment("any(?) as url", s.referrer),
fragment("uniq(user_id) as count")}
)
end
Clickhouse.all(q)
|> Enum.map(fn ref ->
Map.update(ref, "url", nil, fn url -> url && URI.parse("http://" <> url).host end)
end)
referrers =
Clickhouse.all(referrers)
|> Enum.map(fn ref ->
Map.update(ref, "url", nil, fn url -> url && URI.parse("http://" <> url).host end)
end)
show_noref = if length(referrers) == 0, do: true, else: show_noref
if show_noref do
no_referrers =
Clickhouse.all(
from e in base_session_query(site, query),
select:
{fragment("? as name", @no_ref), fragment("any(?) as url", e.referrer),
fragment("uniq(user_id) as count")},
where: e.referrer_source == ""
)
if no_referrers |> hd |> Map.get("count") > 0, do: referrers ++ no_referrers, else: []
else
referrers
end
end
def visitors_from_referrer(site, query, referrer) do
@ -320,25 +340,40 @@ defmodule Plausible.Stats.Clickhouse do
end
def referrer_drilldown(site, query, referrer, include \\ []) do
q = from(
s in base_session_query(site, query),
group_by: s.referrer,
where: s.referrer_source == ^referrer,
order_by: [desc: fragment("count")],
limit: 100
)
referrer = if referrer == @no_ref, do: "", else: referrer
q = if "bounce_rate" in include do
referring_urls =
Clickhouse.all(
from e in base_session_query(site, query),
select: {fragment("? as name", e.referrer), fragment("uniq(user_id) as count")},
group_by: e.referrer,
where: e.referrer_source == ^referrer,
order_by: [desc: fragment("count")],
limit: 100
)
q =
from(
s in base_session_query(site, query),
group_by: s.referrer,
where: s.referrer_source == ^referrer,
order_by: [desc: fragment("count")],
limit: 100
)
q =
if "bounce_rate" in include do
from(
s in q,
select:
{fragment("? as name", s.referrer),
fragment("uniq(user_id) as count"),
{fragment("? as name", s.referrer), fragment("uniq(user_id) as count"),
fragment("round(sum(is_bounce * sign) / sum(sign) * 100) as bounce_rate"),
fragment("round(avg(duration * sign)) as visit_duration")}
)
else
from(s in q, select: {fragment("? as name", s.referrer), fragment("uniq(user_id) as count")})
from(s in q,
select: {fragment("? as name", s.referrer), fragment("uniq(user_id) as count")}
)
end
referring_urls = Clickhouse.all(q)
@ -391,21 +426,27 @@ defmodule Plausible.Stats.Clickhouse do
end
def top_pages(site, query, limit, include) do
q = from(
e in base_query(site, query),
group_by: e.pathname,
order_by: [desc: fragment("count")],
limit: ^limit
)
q = if "unique_visitors" in include do
q =
from(
e in q,
select: {fragment("? as name", e.pathname), fragment("count(?) as count", e.pathname), fragment("uniq(?) as unique_visitors", e.user_id)}
e in base_query(site, query),
group_by: e.pathname,
order_by: [desc: fragment("count")],
limit: ^limit
)
else
from(e in q, select: {fragment("? as name", e.pathname), fragment("count(?) as count", e.pathname)})
end
q =
if "unique_visitors" in include do
from(
e in q,
select:
{fragment("? as name", e.pathname), fragment("count(?) as count", e.pathname),
fragment("uniq(?) as unique_visitors", e.user_id)}
)
else
from(e in q,
select: {fragment("? as name", e.pathname), fragment("count(?) as count", e.pathname)}
)
end
pages = Clickhouse.all(q)

View File

@ -37,6 +37,8 @@ defmodule Plausible.Release do
prepare()
Enum.each(repos(), &run_migrations_for/1)
init_admin()
prepare_clickhouse()
run_migrations_for_ch()
IO.puts("Migrations successful!")
end
@ -44,7 +46,6 @@ defmodule Plausible.Release do
prepare()
# Run seed script
Enum.each(repos(), &run_seeds_for/1)
# Signal shutdown
IO.puts("Success!")
end
@ -52,6 +53,8 @@ defmodule Plausible.Release do
def createdb do
prepare()
do_create_db()
prepare_clickhouse(:default_db)
do_create_ch_db()
IO.puts("Creation of Db successful!")
end
@ -117,23 +120,11 @@ defmodule Plausible.Release do
{:ok, _, _} = Ecto.Migrator.with_repo(repo, &Ecto.Migrator.run(&1, :up, all: true))
end
defp do_create_db do
for repo <- repos() do
:ok = ensure_repo_created(repo)
end
do_create_ch_db()
end
defp do_create_ch_db() do
db_to_create = Keyword.get(Application.get_env(:plausible, :clickhouse), :database)
IO.puts("create #{inspect(db_to_create)} clickhouse database/tables if it doesn't exist")
Clickhousex.query(:clickhouse, "CREATE DATABASE IF NOT EXISTS #{db_to_create}", [])
defp run_migrations_for_ch() do
db = Keyword.get(Application.get_env(:plausible, :clickhouse), :database)
tb_events = """
CREATE TABLE IF NOT EXISTS #{db_to_create}.events (
CREATE TABLE IF NOT EXISTS #{db}.events (
timestamp DateTime,
name String,
domain String,
@ -156,7 +147,7 @@ defmodule Plausible.Release do
Clickhousex.query(:clickhouse, tb_events, [])
tb_sessions = """
CREATE TABLE IF NOT EXISTS #{db_to_create}.sessions (
CREATE TABLE IF NOT EXISTS #{db}.sessions (
session_id UInt64,
sign Int8,
domain String,
@ -185,6 +176,18 @@ defmodule Plausible.Release do
Clickhousex.query(:clickhouse, tb_sessions, [])
end
defp do_create_db do
for repo <- repos() do
:ok = ensure_repo_created(repo)
end
end
defp do_create_ch_db() do
db_to_create = Keyword.get(Application.get_env(:plausible, :clickhouse), :database)
IO.puts("create #{inspect(db_to_create)} clickhouse database/tables if it doesn't exist")
Clickhousex.query(:clickhouse, "CREATE DATABASE IF NOT EXISTS #{db_to_create}", [])
end
defp ensure_repo_created(repo) do
IO.puts("create #{inspect(repo)} database if it doesn't exist")
@ -208,8 +211,6 @@ defmodule Plausible.Release do
# Load the code for myapp, but don't start it
:ok = Application.load(@app)
prepare_clickhouse()
IO.puts("Starting dependencies..")
# Start apps necessary for executing migrations
Enum.each(@start_apps, &Application.ensure_all_started/1)
@ -219,7 +220,8 @@ defmodule Plausible.Release do
Enum.each(repos(), & &1.start_link(pool_size: 2))
end
defp prepare_clickhouse do
# connect to the default db for creating the required db
defp prepare_clickhouse(:default_db) do
Application.ensure_all_started(:db_connection)
Application.ensure_all_started(:hackney)
@ -228,7 +230,24 @@ defmodule Plausible.Release do
port: 8123,
name: :clickhouse,
database: "default",
hostname: Keyword.get(Application.get_env(:plausible, :clickhouse), :hostname)
username: "default",
hostname: Keyword.get(Application.get_env(:plausible, :clickhouse), :hostname),
password: Keyword.get(Application.get_env(:plausible, :clickhouse), :password)
)
end
defp prepare_clickhouse() do
Application.ensure_all_started(:db_connection)
Application.ensure_all_started(:hackney)
Clickhousex.start_link(
scheme: :http,
port: 8123,
name: :clickhouse,
username: Keyword.get(Application.get_env(:plausible, :clickhouse), :username),
database: Keyword.get(Application.get_env(:plausible, :clickhouse), :database),
hostname: Keyword.get(Application.get_env(:plausible, :clickhouse), :hostname),
password: Keyword.get(Application.get_env(:plausible, :clickhouse), :password)
)
end

View File

@ -29,25 +29,28 @@ defmodule PlausibleWeb.Api.ExternalController do
end
def health(conn, _params) do
postgres_health = case Ecto.Adapters.SQL.query(Plausible.Repo, "SELECT 1", []) do
{:ok, _} -> "ok"
e -> "error: #{inspect e}"
end
postgres_health =
case Ecto.Adapters.SQL.query(Plausible.Repo, "SELECT 1", []) do
{:ok, _} -> "ok"
e -> "error: #{inspect(e)}"
end
clickhouse_health = case Clickhousex.query(:clickhouse, "SELECT 1", []) do
{:ok, _} -> "ok"
e -> "error: #{inspect e}"
end
clickhouse_health =
case Clickhousex.query(:clickhouse, "SELECT 1", []) do
{:ok, _} -> "ok"
e -> "error: #{inspect(e)}"
end
status = case {postgres_health, clickhouse_health} do
{"ok", "ok"} ->200
_ -> 500
end
status =
case {postgres_health, clickhouse_health} do
{"ok", "ok"} -> 200
_ -> 500
end
put_status(conn, status)
|> json(%{
postgres: postgres_health,
clickhouse: clickhouse_health,
clickhouse: clickhouse_health
})
end

View File

@ -27,11 +27,11 @@ defmodule PlausibleWeb.Api.StatsController do
[
%{
name: "Active visitors",
count: Stats.current_visitors(site),
count: Stats.current_visitors(site)
},
%{
name: "Pageviews (last 30 min)",
count: Stats.total_pageviews(site, query),
count: Stats.total_pageviews(site, query)
}
]
end
@ -94,7 +94,11 @@ defmodule PlausibleWeb.Api.StatsController do
change: percent_change(prev_pageviews, pageviews)
},
%{name: "Bounce rate", percentage: bounce_rate, change: change_bounce_rate},
%{name: "Visit duration", count: visit_duration, change: percent_change(prev_visit_duration, visit_duration)}
%{
name: "Visit duration",
count: visit_duration,
change: percent_change(prev_visit_duration, visit_duration)
}
]
end
@ -116,8 +120,8 @@ defmodule PlausibleWeb.Api.StatsController do
query = Query.from(site.timezone, params)
include = if params["include"], do: String.split(params["include"], ","), else: []
limit = if params["limit"], do: String.to_integer(params["limit"])
json(conn, Stats.top_referrers(site, query, limit || 9, include))
show_noref = if params["show_noref"], do: "true", else: false
json(conn, Stats.top_referrers(site, query, limit || 9, show_noref, include))
end
def referrers_for_goal(conn, params) do

View File

@ -236,7 +236,11 @@ defmodule PlausibleWeb.AuthController do
def user_settings(conn, _params) do
changeset = Auth.User.changeset(conn.assigns[:current_user])
render(conn, "user_settings.html", changeset: changeset, subscription: conn.assigns[:current_user].subscription)
render(conn, "user_settings.html",
changeset: changeset,
subscription: conn.assigns[:current_user].subscription
)
end
def save_settings(conn, %{"user" => user_params}) do

View File

@ -8,7 +8,7 @@ defmodule PlausibleWeb.PageController do
user = conn.assigns[:current_user] |> Repo.preload(:sites)
render(conn, "sites.html", sites: user.sites)
else
render(conn, "index.html" )
render(conn, "index.html")
end
end
end

View File

@ -160,8 +160,7 @@ defmodule PlausibleWeb.SiteController do
end
def reset_stats(conn, %{"website" => website}) do
site =
Sites.get_for_user!(conn.assigns[:current_user].id, website)
site = Sites.get_for_user!(conn.assigns[:current_user].id, website)
Plausible.Clickhouse.delete_stats!(site)
conn
@ -417,8 +416,9 @@ defmodule PlausibleWeb.SiteController do
end
def delete_custom_domain(conn, %{"website" => website}) do
site = Sites.get_for_user!(conn.assigns[:current_user].id, website)
|> Repo.preload(:custom_domain)
site =
Sites.get_for_user!(conn.assigns[:current_user].id, website)
|> Repo.preload(:custom_domain)
Repo.delete!(site.custom_domain)

View File

@ -14,7 +14,10 @@ defmodule PlausibleWeb.AuthPlug do
id ->
user =
Repo.get_by(Plausible.Auth.User, id: id)
|> Repo.preload(subscription: from(s in Plausible.Billing.Subscription, order_by: [desc: s.inserted_at]))
|> Repo.preload(
subscription:
from(s in Plausible.Billing.Subscription, order_by: [desc: s.inserted_at])
)
if user do
Sentry.Context.set_user_context(%{id: user.id, name: user.name})

View File

@ -20,7 +20,6 @@ defmodule PlausibleWeb.AutoAuthPlug do
Plug.Conn.put_session(conn, :login_dest, conn.request_path)
|> Phoenix.Controller.redirect(to: "/login")
|> halt
end
end
end

View File

@ -1,3 +1,4 @@
<%= if !Keyword.fetch!(Application.get_env(:plausible, :selfhost), :disable_subscription) do %>
<div class="max-w-2xl mx-auto bg-white shadow-md rounded rounded-t-none border-t-2 border-orange-200 px-8 pt-6 pb-8 mt-24">
<div class="flex justify-between">
<h2 class="text-xl font-black">Subscription Plan</h2>
@ -86,6 +87,7 @@
</div>
<% end %>
</div>
<% end %>
<div class="max-w-2xl mx-auto bg-white shadow-md rounded rounded-t-none border-t-2 border-indigo-100 px-8 pt-6 pb-8 mt-16">
<h2 class="text-xl font-black">Account settings</h2>

View File

@ -25,7 +25,7 @@
<%= cond do %>
<% @conn.assigns[:current_user] -> %>
<ul class="flex">
<%= if @conn.assigns[:current_user].subscription == nil do %>
<%= if !Keyword.fetch!(Application.get_env(:plausible, :selfhost), :disable_subscription) && @conn.assigns[:current_user].subscription == nil do %>
<li class="mr-6 hidden sm:block">
<%= link(trial_notificaton(@conn.assigns[:current_user]), to: "/settings", class: "font-bold text-orange-900 rounded p-2 bg-orange-200", style: "line-height: 40px;") %>
</li>

View File

@ -16,26 +16,31 @@ defmodule Plausible.Workers.ScheduleEmailReports do
end
defp schedule_weekly_emails() do
weekly_jobs = from(
j in Oban.Job,
where: j.worker == "Plausible.Workers.SendEmailReport"
and fragment("(? ->> 'interval')", j.args) == "weekly"
)
weekly_jobs =
from(
j in Oban.Job,
where:
j.worker == "Plausible.Workers.SendEmailReport" and
fragment("(? ->> 'interval')", j.args) == "weekly"
)
sites =
Repo.all(
from s in Plausible.Site,
join: wr in Plausible.Site.WeeklyReport,
on: wr.site_id == s.id,
left_join: job in subquery(weekly_jobs),
on: fragment("(? -> 'site_id')::int", job.args) == s.id and
job.state not in ["completed", "discarded"],
join: wr in Plausible.Site.WeeklyReport,
on: wr.site_id == s.id,
left_join: job in subquery(weekly_jobs),
on:
fragment("(? -> 'site_id')::int", job.args) == s.id and
job.state not in ["completed", "discarded"],
where: is_nil(job),
preload: [weekly_report: wr]
)
for site <- sites do
SendEmailReport.new(%{site_id: site.id, interval: "weekly"}, scheduled_at: monday_9am(site.timezone))
SendEmailReport.new(%{site_id: site.id, interval: "weekly"},
scheduled_at: monday_9am(site.timezone)
)
|> Oban.insert!()
end
@ -50,11 +55,13 @@ defmodule Plausible.Workers.ScheduleEmailReports do
end
defp schedule_monthly_emails() do
monthly_jobs = from(
j in Oban.Job,
where: j.worker == "Plausible.Workers.SendEmailReport"
and fragment("(? ->> 'interval')", j.args) == "monthly"
)
monthly_jobs =
from(
j in Oban.Job,
where:
j.worker == "Plausible.Workers.SendEmailReport" and
fragment("(? ->> 'interval')", j.args) == "monthly"
)
sites =
Repo.all(
@ -70,7 +77,9 @@ defmodule Plausible.Workers.ScheduleEmailReports do
)
for site <- sites do
SendEmailReport.new(%{site_id: site.id, interval: "monthly"}, scheduled_at: first_of_month_9am(site.timezone))
SendEmailReport.new(%{site_id: site.id, interval: "monthly"},
scheduled_at: first_of_month_9am(site.timezone)
)
|> Oban.insert!()
end

View File

@ -16,12 +16,14 @@ defmodule Plausible.Workers.SendEmailReport do
send_report(email, site, "Weekly", unsubscribe_link, query)
end
:ok
end
@impl Oban.Worker
def perform(%{"interval" => "monthly", "site_id" => site_id}, _job) do
site = Repo.get(Plausible.Site, site_id) |> Repo.preload(:monthly_report)
last_month =
Timex.now(site.timezone)
|> Timex.shift(months: -1)
@ -40,6 +42,7 @@ defmodule Plausible.Workers.SendEmailReport do
send_report(email, site, Timex.format!(last_month, "{Mfull}"), unsubscribe_link, query)
end
:ok
end
@ -57,20 +60,21 @@ defmodule Plausible.Workers.SendEmailReport do
user = Plausible.Auth.find_user_by(email: email)
login_link = user && Plausible.Sites.is_owner?(user.id, site)
template = PlausibleWeb.Email.weekly_report(email, site,
unique_visitors: unique_visitors,
change_visitors: change_visitors,
pageviews: pageviews,
change_pageviews: change_pageviews,
bounce_rate: bounce_rate,
change_bounce_rate: change_bounce_rate,
referrers: referrers,
unsubscribe_link: unsubscribe_link,
login_link: login_link,
pages: pages,
query: query,
name: name
)
template =
PlausibleWeb.Email.weekly_report(email, site,
unique_visitors: unique_visitors,
change_visitors: change_visitors,
pageviews: pageviews,
change_pageviews: change_pageviews,
bounce_rate: bounce_rate,
change_bounce_rate: change_bounce_rate,
referrers: referrers,
unsubscribe_link: unsubscribe_link,
login_link: login_link,
pages: pages,
query: query,
name: name
)
try do
Plausible.Mailer.send_email(template)

View File

@ -1,8 +1,10 @@
defmodule Plausible.Workers.SendTrialNotifications do
use Plausible.Repo
use Oban.Worker,
queue: :trial_notification_emails,
max_attempts: 1
require Logger
@impl Oban.Worker

View File

@ -78,7 +78,13 @@ defmodule Plausible.BillingTest do
test "is true for a user with a cancelled subscription IF the billing cycle is complete" do
user = insert(:user, trial_expiry_date: Timex.shift(Timex.today(), days: -1))
insert(:subscription, user: user, status: "deleted", next_bill_date: Timex.shift(Timex.today(), days: -1))
insert(:subscription,
user: user,
status: "deleted",
next_bill_date: Timex.shift(Timex.today(), days: -1)
)
user = Repo.preload(user, :subscription)
assert Billing.needs_to_upgrade?(user)

View File

@ -380,7 +380,6 @@ defmodule PlausibleWeb.Api.ExternalControllerTest do
assert event["referrer"] == ""
end
# Fake data is set up in config/test.exs
test "looks up the country from the ip address", %{conn: conn} do
params = %{

View File

@ -25,9 +25,24 @@ defmodule PlausibleWeb.Api.StatsController.PagesTest do
assert json_response(conn, 200) == [
%{"bounce_rate" => 33.0, "count" => 2, "unique_visitors" => 2, "name" => "/"},
%{"bounce_rate" => nil, "count" => 2, "unique_visitors" => 2, "name" => "/register"},
%{"bounce_rate" => nil, "count" => 1, "unique_visitors" => 1, "name" => "/contact"},
%{"bounce_rate" => nil, "count" => 1, "unique_visitors" => 1, "name" => "/irrelevant"}
%{
"bounce_rate" => nil,
"count" => 2,
"unique_visitors" => 2,
"name" => "/register"
},
%{
"bounce_rate" => nil,
"count" => 1,
"unique_visitors" => 1,
"name" => "/contact"
},
%{
"bounce_rate" => nil,
"count" => 1,
"unique_visitors" => 1,
"name" => "/irrelevant"
}
]
end

View File

@ -29,7 +29,13 @@ defmodule PlausibleWeb.Api.StatsController.ReferrersTest do
"visit_duration" => 50,
"url" => "10words.com"
},
%{"name" => "Bing", "count" => 1, "bounce_rate" => 0, "visit_duration" => 100 ,"url" => ""}
%{
"name" => "Bing",
"count" => 1,
"bounce_rate" => 0,
"visit_duration" => 100,
"url" => ""
}
]
end
@ -99,7 +105,12 @@ defmodule PlausibleWeb.Api.StatsController.ReferrersTest do
assert json_response(conn, 200) == %{
"total_visitors" => 2,
"referrers" => [
%{"name" => "10words.com/page1", "count" => 2, "bounce_rate" => 50.0, "visit_duration" => 50.0}
%{
"name" => "10words.com/page1",
"count" => 2,
"bounce_rate" => 50.0,
"visit_duration" => 50.0
}
]
}
end

View File

@ -153,7 +153,10 @@ defmodule PlausibleWeb.SiteControllerTest do
test "updates google auth property", %{conn: conn, user: user, site: site} do
insert(:google_auth, user: user, site: site)
put(conn, "/#{site.domain}/settings/google", %{"google_auth" => %{"property" => "some-new-property.com"}})
put(conn, "/#{site.domain}/settings/google", %{
"google_auth" => %{"property" => "some-new-property.com"}
})
updated_auth = Repo.one(Plausible.Site.GoogleAuth)
assert updated_auth.property == "some-new-property.com"

View File

@ -15,6 +15,7 @@ defmodule PlausibleWeb.AuthPlugTest do
test "looks up current user if they are logged in" do
user = insert(:user)
subscription = insert(:subscription, user: user)
conn =
conn(:get, "/")
|> init_test_session(%{current_user_id: user.id})
@ -26,7 +27,10 @@ defmodule PlausibleWeb.AuthPlugTest do
test "looks up the latest subscription" do
user = insert(:user)
_old_subscription = insert(:subscription, user: user, inserted_at: Timex.now() |> Timex.shift(days: -1))
_old_subscription =
insert(:subscription, user: user, inserted_at: Timex.now() |> Timex.shift(days: -1))
subscription = insert(:subscription, user: user, inserted_at: Timex.now())
conn =

View File

@ -287,7 +287,7 @@ defmodule Plausible.Test.ClickhouseSetup do
referrer_source: "Twitter",
start: ~N[2019-03-01 02:00:00],
timestamp: ~N[2019-03-01 02:00:00]
},
}
])
end
end

View File

@ -15,7 +15,11 @@ defmodule Plausible.Workers.ScheduleEmailReportsTest do
perform(%{})
assert_enqueued worker: SendEmailReport, args: %{site_id: site.id, interval: "weekly"}, scheduled_at: ScheduleEmailReports.monday_9am(site.timezone)
assert_enqueued(
worker: SendEmailReport,
args: %{site_id: site.id, interval: "weekly"},
scheduled_at: ScheduleEmailReports.monday_9am(site.timezone)
)
end
test "does not schedule more than one weekly report at a time" do
@ -33,11 +37,10 @@ defmodule Plausible.Workers.ScheduleEmailReportsTest do
insert(:weekly_report, site: site, recipients: ["user@email.com"])
perform(%{})
Repo.update_all("oban_jobs", [set: [state: "completed"]])
Repo.update_all("oban_jobs", set: [state: "completed"])
assert Enum.count(all_enqueued(worker: SendEmailReport)) == 0
perform(%{})
assert Enum.count(all_enqueued(worker: SendEmailReport)) == 1
end
end
@ -48,7 +51,11 @@ defmodule Plausible.Workers.ScheduleEmailReportsTest do
perform(%{})
assert_enqueued worker: SendEmailReport, args: %{site_id: site.id, interval: "monthly"}, scheduled_at: ScheduleEmailReports.first_of_month_9am(site.timezone)
assert_enqueued(
worker: SendEmailReport,
args: %{site_id: site.id, interval: "monthly"},
scheduled_at: ScheduleEmailReports.first_of_month_9am(site.timezone)
)
end
test "does not schedule more than one monthly report at a time" do
@ -66,11 +73,10 @@ defmodule Plausible.Workers.ScheduleEmailReportsTest do
insert(:monthly_report, site: site, recipients: ["user@email.com"])
perform(%{})
Repo.update_all("oban_jobs", [set: [state: "completed"]])
Repo.update_all("oban_jobs", set: [state: "completed"])
assert Enum.count(all_enqueued(worker: SendEmailReport)) == 0
perform(%{})
assert Enum.count(all_enqueued(worker: SendEmailReport)) == 1
end
end
end

View File

@ -31,7 +31,12 @@ defmodule Plausible.Workers.SendEmailReportTest do
test "sends monthly report to all recipients" do
site = insert(:site, domain: "test-site.com", timezone: "US/Eastern")
insert(:monthly_report, site: site, recipients: ["user@email.com", "user2@email.com"])
last_month = Timex.now(site.timezone) |> Timex.shift(months: -1) |> Timex.beginning_of_month() |> Timex.format!("{Mfull}")
last_month =
Timex.now(site.timezone)
|> Timex.shift(months: -1)
|> Timex.beginning_of_month()
|> Timex.format!("{Mfull}")
perform(%{"site_id" => site.id, "interval" => "monthly"})