analytics/lib/plausible_web/controllers/api/stats_controller.ex
Chandra Tungathurthi f7b37fe9ea
Selhosted version Improvements and additional features (#209)
* first commit with test and compile job

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding 'prepare' stage

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated ci script to include "test" compile phase

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding environment variables for connecting to postgresql

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated ci config for postgres

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* using non-alpine version of elixir

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* re-using the 'compile' artifacts and added explict env variables for testing

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removing redundant deps fetching from common code

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* formatting using mix.format -- beware no-code changes!

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* added release config

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding consistent env variable for Database

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* more cleaning up of environment variables

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding releases config for enabling releases

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* cleaning up env configs

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Cleaned up config and prepared config for releases

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated CI script with new config for test

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added Dockerfile for creating production docker image

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding "docker" build job yay!

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* using non-slim version of debian and installing webpack

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding overlays for migrations on releases

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* restricting the docker built to master branch only

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* typo fix

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding "Hosting.md" to explain hosting instructions

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removed the default comments

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added documentation related to env variables

* updated documentation and fixed typo

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated documentation

* Bumping up elixir version as `overlays` are only supported in latest version

read release notes: https://github.com/elixir-lang/elixir/releases/tag/v1.10.0

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding tarball assembly during release

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated HOSTING.md

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added support for db migration

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* minor corrections

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* initializing admin user

Admin user has been added in the "migration" phase. A default user is automatically created in the process. One can provide the related env variables, else a new one will be automatically created for you.

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Initial base domain update - phase#1

These changes are only meant for correct operating it under self-hosting. There are many other cosmetic changes, that require updates to email, site and other places where the original website and author is used.

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Using dedicated config variable `base_domain` instead

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding base_domain to releases config

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removing the dedicated config "base_domain", relying on endpoint host

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Removed the usage of "Mix" in code!

It is bad practice to use "mix" module inside the code as in actual release this module is unavailable. Replacing this with a config environment variable

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added support for SMTP via Bamboo Smtp Adapter

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Capturing SMTP errors via Sentry

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Minor updates

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding junit formatter -- useful for generating test reports

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding documentation for default user

* Resolve "Gitlab Adoption: Add supported services in "Security & Compliance""

* bumping up the debian version to fix issues

fixing some vulnerabilities identified by the scanning tools

* More updates for self-hosting

Changes in most of the places to suit self-hosting. Although, there are some which have been left-off.

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* quick-dirty-fix!

* bumping up the db connect timeout

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* bumping up the db connect timeout

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* bumping up the db connect timeout

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* bumping up timeout - skipping MRs :-/

* removing restrictions on watching for changes

this stuff isn't working

* Update HOSTING.md

* renamed the module name

* reverting formatting-whitespace changes

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* reverting the name to release

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding docker-compose.yml and related instructions

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* using `plausible_url` instead of assuming `https`

this is because, it is much to test in local dev machines and in most cases there's already a layer above which is capable for `https` termination and http -> https upgrade

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* WIP: merging changes from upstream

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* wip: more changes

* Pushing in changes from upstream

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* changes to ci for testing

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* cleaning up and finishing clickhouse integration

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updating readme with hosting details

* removing deleted files from upstream

* minor config adjustments

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* formatting changes

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* changing the connection strategy for clickhouse during release

since clickhouse integration doesn't have an ecto support, we need to prepare the db _before_ the clickhouse migration. One workaround is to connect to a default db on init and then create a db

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* formatting

* cleanup and added separated migration to setup

* Big improvements to selfhosting

- added ability for disabling
  - authentication completely
  - registration
  - landing page

- formatting cleanups

* Big improvements to selfhosting

- added ability for disabling
  - authentication completely
  - registration
  - landing page

- formatting cleanups

* changing smtp auth  to optional

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removed stale templates and permanently removed landing page

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removed stale templates and permanently removed landing page

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removed stale templates and permanently removed landing page

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* WIP

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* fixes form upstream merge

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* added disabling subscription for selfhosted version

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated doc

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* formatting

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Remove reference to file that doesn't exist

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* do not show direct traffic if there's no data

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* addressing PR comments

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* formatting

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>
2020-07-21 09:58:00 +03:00

238 lines
7.4 KiB
Elixir

defmodule PlausibleWeb.Api.StatsController do
use PlausibleWeb, :controller
use Plausible.Repo
alias Plausible.Stats.Clickhouse, as: Stats
alias Plausible.Stats.Query
plug PlausibleWeb.AuthorizeStatsPlug
def main_graph(conn, params) do
site = conn.assigns[:site]
query = Query.from(site.timezone, params)
plot_task = Task.async(fn -> Stats.calculate_plot(site, query) end)
top_stats = fetch_top_stats(site, query)
{plot, compare_plot, labels, present_index} = Task.await(plot_task)
json(conn, %{
plot: plot,
compare_plot: compare_plot,
labels: labels,
present_index: present_index,
top_stats: top_stats,
interval: query.step_type
})
end
defp fetch_top_stats(site, %Query{period: "realtime"} = query) do
[
%{
name: "Active visitors",
count: Stats.current_visitors(site)
},
%{
name: "Pageviews (last 30 min)",
count: Stats.total_pageviews(site, query)
}
]
end
defp fetch_top_stats(site, %Query{filters: %{"goal" => goal}} = query) when is_binary(goal) do
prev_query = Query.shift_back(query)
total_visitors = Stats.unique_visitors(site, %{query | filters: %{}})
prev_total_visitors = Stats.unique_visitors(site, %{prev_query | filters: %{}})
converted_visitors = Stats.unique_visitors(site, query)
prev_converted_visitors = Stats.unique_visitors(site, prev_query)
conversion_rate =
if total_visitors > 0,
do: Float.round(converted_visitors / total_visitors * 100, 1),
else: 0.0
prev_conversion_rate =
if prev_total_visitors > 0,
do: Float.round(prev_converted_visitors / prev_total_visitors * 100, 1),
else: 0.0
[
%{
name: "Total visitors",
count: total_visitors,
change: percent_change(prev_total_visitors, total_visitors)
},
%{
name: "Converted visitors",
count: converted_visitors,
change: percent_change(prev_converted_visitors, converted_visitors)
},
%{
name: "Conversion rate",
percentage: conversion_rate,
change: percent_change(prev_conversion_rate, conversion_rate)
}
]
end
defp fetch_top_stats(site, query) do
prev_query = Query.shift_back(query)
{pageviews, visitors} = Stats.pageviews_and_visitors(site, query)
{prev_pageviews, prev_visitors} = Stats.pageviews_and_visitors(site, prev_query)
bounce_rate = Stats.bounce_rate(site, query)
prev_bounce_rate = Stats.bounce_rate(site, prev_query)
change_bounce_rate = if prev_bounce_rate > 0, do: bounce_rate - prev_bounce_rate
visit_duration = Stats.visit_duration(site, query)
prev_visit_duration = Stats.visit_duration(site, prev_query)
[
%{
name: "Unique visitors",
count: visitors,
change: percent_change(prev_visitors, visitors)
},
%{
name: "Total pageviews",
count: pageviews,
change: percent_change(prev_pageviews, pageviews)
},
%{name: "Bounce rate", percentage: bounce_rate, change: change_bounce_rate},
%{
name: "Visit duration",
count: visit_duration,
change: percent_change(prev_visit_duration, visit_duration)
}
]
end
defp percent_change(old_count, new_count) do
cond do
old_count == 0 and new_count > 0 ->
100
old_count == 0 and new_count == 0 ->
0
true ->
round((new_count - old_count) / old_count * 100)
end
end
def referrers(conn, params) do
site = conn.assigns[:site]
query = Query.from(site.timezone, params)
include = if params["include"], do: String.split(params["include"], ","), else: []
limit = if params["limit"], do: String.to_integer(params["limit"])
show_noref = if params["show_noref"], do: "true", else: false
json(conn, Stats.top_referrers(site, query, limit || 9, show_noref, include))
end
def referrers_for_goal(conn, params) do
site = conn.assigns[:site]
query = Query.from(site.timezone, params)
json(conn, Stats.top_referrers_for_goal(site, query, params["limit"] || 9))
end
@google_api Application.fetch_env!(:plausible, :google_api)
def referrer_drilldown(conn, %{"referrer" => "Google"} = params) do
site = conn.assigns[:site] |> Repo.preload(:google_auth)
query = Query.from(site.timezone, params)
search_terms =
if site.google_auth && site.google_auth.property && !query.filters["goal"] do
@google_api.fetch_stats(site.google_auth, query)
end
case search_terms do
nil ->
total_visitors = Stats.visitors_from_referrer(site, query, "Google")
user_id = get_session(conn, :current_user_id)
is_owner = user_id && Plausible.Sites.is_owner?(user_id, site)
json(conn, %{not_configured: true, is_owner: is_owner, total_visitors: total_visitors})
{:ok, terms} ->
total_visitors = Stats.visitors_from_referrer(site, query, "Google")
json(conn, %{search_terms: terms, total_visitors: total_visitors})
{:error, e} ->
put_status(conn, 500)
|> json(%{error: e})
end
end
def referrer_drilldown(conn, %{"referrer" => referrer} = params) do
site = conn.assigns[:site]
query = Query.from(site.timezone, params)
include = if params["include"], do: String.split(params["include"], ","), else: []
referrers = Stats.referrer_drilldown(site, query, referrer, include)
total_visitors = Stats.visitors_from_referrer(site, query, referrer)
json(conn, %{referrers: referrers, total_visitors: total_visitors})
end
def referrer_drilldown_for_goal(conn, %{"referrer" => referrer} = params) do
site = conn.assigns[:site]
query = Query.from(site.timezone, params)
referrers = Stats.referrer_drilldown_for_goal(site, query, referrer)
total_visitors = Stats.conversions_from_referrer(site, query, referrer)
json(conn, %{referrers: referrers, total_visitors: total_visitors})
end
def pages(conn, params) do
site = conn.assigns[:site]
query = Query.from(site.timezone, params)
include = if params["include"], do: String.split(params["include"], ","), else: []
limit = if params["limit"], do: String.to_integer(params["limit"])
json(conn, Stats.top_pages(site, query, limit || 9, include))
end
def countries(conn, params) do
site = conn.assigns[:site]
query = Query.from(site.timezone, params)
json(conn, Stats.countries(site, query))
end
def browsers(conn, params) do
site = conn.assigns[:site]
query = Query.from(site.timezone, params)
json(conn, Stats.browsers(site, query, parse_integer(params["limit"]) || 9))
end
def operating_systems(conn, params) do
site = conn.assigns[:site]
query = Query.from(site.timezone, params)
json(conn, Stats.operating_systems(site, query, parse_integer(params["limit"]) || 9))
end
def screen_sizes(conn, params) do
site = conn.assigns[:site]
query = Query.from(site.timezone, params)
json(conn, Stats.top_screen_sizes(site, query))
end
def conversions(conn, params) do
site = conn.assigns[:site]
query = Query.from(site.timezone, params)
json(conn, Stats.goal_conversions(site, query))
end
def current_visitors(conn, _) do
json(conn, Stats.current_visitors(conn.assigns[:site]))
end
defp parse_integer(nil), do: nil
defp parse_integer(nr) do
case Integer.parse(nr) do
{number, ""} -> number
_ -> nil
end
end
end