analytics/test/plausible_web/controllers/api/external_controller_test.exs
Chandra Tungathurthi f7b37fe9ea
Selhosted version Improvements and additional features (#209)
* first commit with test and compile job

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding 'prepare' stage

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated ci script to include "test" compile phase

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding environment variables for connecting to postgresql

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated ci config for postgres

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* using non-alpine version of elixir

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* re-using the 'compile' artifacts and added explict env variables for testing

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removing redundant deps fetching from common code

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* formatting using mix.format -- beware no-code changes!

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* added release config

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding consistent env variable for Database

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* more cleaning up of environment variables

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding releases config for enabling releases

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* cleaning up env configs

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Cleaned up config and prepared config for releases

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated CI script with new config for test

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added Dockerfile for creating production docker image

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding "docker" build job yay!

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* using non-slim version of debian and installing webpack

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding overlays for migrations on releases

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* restricting the docker built to master branch only

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* typo fix

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding "Hosting.md" to explain hosting instructions

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removed the default comments

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added documentation related to env variables

* updated documentation and fixed typo

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated documentation

* Bumping up elixir version as `overlays` are only supported in latest version

read release notes: https://github.com/elixir-lang/elixir/releases/tag/v1.10.0

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding tarball assembly during release

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated HOSTING.md

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added support for db migration

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* minor corrections

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* initializing admin user

Admin user has been added in the "migration" phase. A default user is automatically created in the process. One can provide the related env variables, else a new one will be automatically created for you.

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Initial base domain update - phase#1

These changes are only meant for correct operating it under self-hosting. There are many other cosmetic changes, that require updates to email, site and other places where the original website and author is used.

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Using dedicated config variable `base_domain` instead

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding base_domain to releases config

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removing the dedicated config "base_domain", relying on endpoint host

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Removed the usage of "Mix" in code!

It is bad practice to use "mix" module inside the code as in actual release this module is unavailable. Replacing this with a config environment variable

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added support for SMTP via Bamboo Smtp Adapter

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Capturing SMTP errors via Sentry

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Minor updates

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding junit formatter -- useful for generating test reports

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding documentation for default user

* Resolve "Gitlab Adoption: Add supported services in "Security & Compliance""

* bumping up the debian version to fix issues

fixing some vulnerabilities identified by the scanning tools

* More updates for self-hosting

Changes in most of the places to suit self-hosting. Although, there are some which have been left-off.

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* quick-dirty-fix!

* bumping up the db connect timeout

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* bumping up the db connect timeout

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* bumping up the db connect timeout

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* bumping up timeout - skipping MRs :-/

* removing restrictions on watching for changes

this stuff isn't working

* Update HOSTING.md

* renamed the module name

* reverting formatting-whitespace changes

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* reverting the name to release

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding docker-compose.yml and related instructions

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* using `plausible_url` instead of assuming `https`

this is because, it is much to test in local dev machines and in most cases there's already a layer above which is capable for `https` termination and http -> https upgrade

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* WIP: merging changes from upstream

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* wip: more changes

* Pushing in changes from upstream

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* changes to ci for testing

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* cleaning up and finishing clickhouse integration

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updating readme with hosting details

* removing deleted files from upstream

* minor config adjustments

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* formatting changes

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* changing the connection strategy for clickhouse during release

since clickhouse integration doesn't have an ecto support, we need to prepare the db _before_ the clickhouse migration. One workaround is to connect to a default db on init and then create a db

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* formatting

* cleanup and added separated migration to setup

* Big improvements to selfhosting

- added ability for disabling
  - authentication completely
  - registration
  - landing page

- formatting cleanups

* Big improvements to selfhosting

- added ability for disabling
  - authentication completely
  - registration
  - landing page

- formatting cleanups

* changing smtp auth  to optional

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removed stale templates and permanently removed landing page

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removed stale templates and permanently removed landing page

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removed stale templates and permanently removed landing page

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* WIP

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* fixes form upstream merge

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* added disabling subscription for selfhosted version

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated doc

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* formatting

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Remove reference to file that doesn't exist

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* do not show direct traffic if there's no data

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* addressing PR comments

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* formatting

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>
2020-07-21 09:58:00 +03:00

421 lines
12 KiB
Elixir

defmodule PlausibleWeb.Api.ExternalControllerTest do
use PlausibleWeb.ConnCase
use Plausible.Repo
defp get_event(domain) do
Plausible.Event.WriteBuffer.flush()
events =
Plausible.Clickhouse.all(
from e in Plausible.ClickhouseEvent,
where: e.domain == ^domain,
order_by: [desc: e.timestamp],
limit: 1
)
List.first(events)
end
@user_agent "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36"
describe "POST /api/event" do
test "records the event", %{conn: conn} do
params = %{
domain: "external-controller-test-1.com",
name: "pageview",
url: "http://gigride.live/",
referrer: "http://m.facebook.com/",
screen_width: 1440
}
conn =
conn
|> put_req_header("content-type", "text/plain")
|> put_req_header("user-agent", @user_agent)
|> post("/api/event", Jason.encode!(params))
pageview = get_event("external-controller-test-1.com")
assert response(conn, 202) == ""
assert pageview["hostname"] == "gigride.live"
assert pageview["domain"] == "external-controller-test-1.com"
assert pageview["pathname"] == "/"
end
test "www. is stripped from domain", %{conn: conn} do
params = %{
name: "custom event",
url: "http://gigride.live/",
domain: "www.external-controller-test-2.com"
}
conn
|> put_req_header("content-type", "text/plain")
|> post("/api/event", Jason.encode!(params))
pageview = get_event("external-controller-test-2.com")
assert pageview["domain"] == "external-controller-test-2.com"
end
test "www. is stripped from hostname", %{conn: conn} do
params = %{
name: "pageview",
url: "http://www.example.com/",
domain: "external-controller-test-3.com"
}
conn
|> put_req_header("content-type", "text/plain")
|> post("/api/event", Jason.encode!(params))
pageview = get_event("external-controller-test-3.com")
assert pageview["hostname"] == "example.com"
end
test "empty path defaults to /", %{conn: conn} do
params = %{
name: "pageview",
url: "http://www.example.com",
domain: "external-controller-test-4.com"
}
conn
|> put_req_header("content-type", "text/plain")
|> post("/api/event", Jason.encode!(params))
pageview = get_event("external-controller-test-4.com")
assert pageview["pathname"] == "/"
end
test "bots and crawlers are ignored", %{conn: conn} do
params = %{
name: "pageview",
url: "http://www.example.com/",
domain: "external-controller-test-5.com"
}
conn
|> put_req_header("content-type", "text/plain")
|> put_req_header("user-agent", "generic crawler")
|> post("/api/event", Jason.encode!(params))
assert get_event("external-controller-test-5.com") == nil
end
test "parses user_agent", %{conn: conn} do
params = %{
name: "pageview",
url: "http://gigride.live/",
domain: "external-controller-test-6.com"
}
conn =
conn
|> put_req_header("content-type", "text/plain")
|> put_req_header("user-agent", @user_agent)
|> post("/api/event", Jason.encode!(params))
pageview = get_event("external-controller-test-6.com")
assert response(conn, 202) == ""
assert pageview["operating_system"] == "Mac"
assert pageview["browser"] == "Chrome"
end
test "parses referrer", %{conn: conn} do
params = %{
name: "pageview",
url: "http://gigride.live/",
referrer: "https://facebook.com",
domain: "external-controller-test-7.com"
}
conn =
conn
|> put_req_header("content-type", "text/plain")
|> put_req_header("user-agent", @user_agent)
|> post("/api/event", Jason.encode!(params))
pageview = get_event("external-controller-test-7.com")
assert response(conn, 202) == ""
assert pageview["referrer_source"] == "Facebook"
end
test "strips trailing slash from referrer", %{conn: conn} do
params = %{
name: "pageview",
url: "http://gigride.live/",
referrer: "https://facebook.com/page/",
domain: "external-controller-test-8.com"
}
conn =
conn
|> put_req_header("content-type", "text/plain")
|> put_req_header("user-agent", @user_agent)
|> post("/api/event", Jason.encode!(params))
pageview = get_event("external-controller-test-8.com")
assert response(conn, 202) == ""
assert pageview["referrer"] == "facebook.com/page"
assert pageview["referrer_source"] == "Facebook"
end
test "ignores when referrer is internal", %{conn: conn} do
params = %{
name: "pageview",
url: "http://gigride.live/",
referrer: "https://gigride.live",
domain: "external-controller-test-9.com"
}
conn =
conn
|> put_req_header("content-type", "text/plain")
|> put_req_header("user-agent", @user_agent)
|> post("/api/event", Jason.encode!(params))
pageview = get_event("external-controller-test-9.com")
assert response(conn, 202) == ""
assert pageview["referrer_source"] == ""
end
test "ignores localhost referrer", %{conn: conn} do
params = %{
name: "pageview",
url: "http://gigride.live/",
referrer: "http://localhost:4000/",
domain: "external-controller-test-10.com"
}
conn =
conn
|> put_req_header("content-type", "text/plain")
|> put_req_header("user-agent", @user_agent)
|> post("/api/event", Jason.encode!(params))
pageview = get_event("external-controller-test-10.com")
assert response(conn, 202) == ""
assert pageview["referrer_source"] == ""
end
test "parses subdomain referrer", %{conn: conn} do
params = %{
name: "pageview",
url: "http://gigride.live/",
referrer: "https://blog.gigride.live",
domain: "external-controller-test-11.com"
}
conn =
conn
|> put_req_header("content-type", "text/plain")
|> put_req_header("user-agent", @user_agent)
|> post("/api/event", Jason.encode!(params))
pageview = get_event("external-controller-test-11.com")
assert response(conn, 202) == ""
assert pageview["referrer_source"] == "blog.gigride.live"
end
test "referrer is cleaned", %{conn: conn} do
params = %{
name: "pageview",
url: "http://www.example.com/",
referrer: "https://www.indiehackers.com/page?query=param#hash",
domain: "external-controller-test-12.com"
}
conn
|> put_req_header("content-type", "text/plain")
|> post("/api/event", Jason.encode!(params))
pageview = get_event("external-controller-test-12.com")
assert pageview["referrer"] == "indiehackers.com/page"
end
test "source param controls the referrer source", %{conn: conn} do
params = %{
name: "pageview",
url: "http://www.example.com/",
referrer: "https://betalist.com/my-produxct",
source: "betalist",
domain: "external-controller-test-13.com"
}
conn
|> put_req_header("content-type", "text/plain")
|> post("/api/event", Jason.encode!(params))
pageview = get_event("external-controller-test-13.com")
assert pageview["referrer_source"] == "betalist"
end
test "if it's an :unknown referrer, just the domain is used", %{conn: conn} do
params = %{
name: "pageview",
url: "http://gigride.live/",
referrer: "https://www.indiehackers.com/landing-page-feedback",
domain: "external-controller-test-14.com"
}
conn =
conn
|> put_req_header("content-type", "text/plain")
|> put_req_header("user-agent", @user_agent)
|> post("/api/event", Jason.encode!(params))
pageview = get_event("external-controller-test-14.com")
assert response(conn, 202) == ""
assert pageview["referrer_source"] == "indiehackers.com"
end
test "if the referrer is not http or https, it is ignored", %{conn: conn} do
params = %{
name: "pageview",
url: "http://gigride.live/",
referrer: "android-app://com.google.android.gm",
domain: "external-controller-test-15.com"
}
conn =
conn
|> put_req_header("content-type", "text/plain")
|> put_req_header("user-agent", @user_agent)
|> post("/api/event", Jason.encode!(params))
pageview = get_event("external-controller-test-15.com")
assert response(conn, 202) == ""
assert pageview["referrer_source"] == ""
end
end
test "screen size is calculated from screen_width", %{conn: conn} do
params = %{
name: "pageview",
url: "http://gigride.live/",
screen_width: 480,
domain: "external-controller-test-16.com"
}
conn =
conn
|> put_req_header("content-type", "text/plain")
|> put_req_header("user-agent", @user_agent)
|> post("/api/event", Jason.encode!(params))
pageview = get_event("external-controller-test-16.com")
assert response(conn, 202) == ""
assert pageview["screen_size"] == "Mobile"
end
test "screen size is nil if screen_width is missing", %{conn: conn} do
params = %{
name: "pageview",
url: "http://gigride.live/",
domain: "external-controller-test-17.com"
}
conn =
conn
|> put_req_header("content-type", "text/plain")
|> put_req_header("user-agent", @user_agent)
|> post("/api/event", Jason.encode!(params))
pageview = get_event("external-controller-test-17.com")
assert response(conn, 202) == ""
assert pageview["screen_size"] == ""
end
test "can trigger a custom event", %{conn: conn} do
params = %{
name: "custom event",
url: "http://gigride.live/",
domain: "external-controller-test-18.com"
}
conn =
conn
|> put_req_header("content-type", "text/plain")
|> put_req_header("user-agent", @user_agent)
|> post("/api/event", Jason.encode!(params))
event = get_event("external-controller-test-18.com")
assert response(conn, 202) == ""
assert event["name"] == "custom event"
end
test "ignores a malformed referrer URL", %{conn: conn} do
params = %{
name: "pageview",
url: "http://gigride.live/",
referrer: "https:://twitter.com",
domain: "external-controller-test-19.com"
}
conn =
conn
|> put_req_header("content-type", "text/plain")
|> put_req_header("user-agent", @user_agent)
|> post("/api/event", Jason.encode!(params))
event = get_event("external-controller-test-19.com")
assert response(conn, 202) == ""
assert event["referrer"] == ""
end
# Fake data is set up in config/test.exs
test "looks up the country from the ip address", %{conn: conn} do
params = %{
name: "pageview",
domain: "external-controller-test-20.com",
url: "http://gigride.live/"
}
conn
|> put_req_header("content-type", "text/plain")
|> put_req_header("x-forwarded-for", "1.1.1.1")
|> post("/api/event", Jason.encode!(params))
pageview = get_event("external-controller-test-20.com")
assert pageview["country_code"] == "US"
end
test "responds 400 when required fields are missing", %{conn: conn} do
params = %{}
conn =
conn
|> put_req_header("content-type", "text/plain")
|> put_req_header("user-agent", @user_agent)
|> post("/api/event", Jason.encode!(params))
assert response(conn, 400) == ""
end
describe "GET /api/health" do
test "returns 200 OK", %{conn: conn} do
conn = get(conn, "/api/health")
assert conn.status == 200
end
end
end