Commit Graph

1926 Commits

Author SHA1 Message Date
Adam Rutkowski
867dad6da7
Implement ingest counters (#2693)
* Clickhouse migration: add ingest_counters table

* Configure ingest counters per MIX_ENV

* Emit telemetry for ingest events with rich metadata

* Allow building Request.t() with fake now() - for testing purposes

* Use clickhousex branch where session_id is assigned to each connection

* Add helper function for getting site id via cache

* Add Ecto schema for `ingest_counters` table

* Implement metrics buffer

* Implement buffering handler for `Plausible.Ingestion.Event` telemetry

* Implement periodic metrics aggregation

* Update counters docs

* Add toStartOfMinute() to ordering key

* Reset the sync connection state in `after` clause

* Flush counters on app termination

* Use separate Repo with async settings enabled at config level

* Switch to clickhouse_settings repo root config key

* Add AsyncInsertRepo module
2023-02-23 14:34:24 +01:00
Adam Rutkowski
7dec454a23
Don't rely on Mix.env check in runtime (#2703) 2023-02-23 10:35:46 +01:00
Adam Rutkowski
043e3ed572
Clickhouse migration: add ingest_counters table (#2692)
* Clickhouse migration: add ingest_counters table

* Add toStartOfMinute() to ordering key

* Explicitly include column to be summarized
2023-02-23 09:34:44 +01:00
RobertJoonas
bc557b78fd
Add visits metric under a feature flag (#2687)
* Add visits metric and make it graphable

* include visits metric in csv export (visitors.csv)

* put visits under a feature flag (CSV export)

* feature flag for displaying visits on the dashboard

* fix formatting

* add visits metric to top stats (fix)

* fix imported_test to expect visits metric included

* fix formatting
2023-02-22 18:10:18 +02:00
Vini Brasil
858687c77b
Fix graph z-index property (#2699)
* Revert "Fix graph element z-index (#2698)"

This reverts commit 35ab26308d.

* Revert "Fix interval selector z-index (#2696)"

This reverts commit c307057386.

* Set graph z-index to zero

* Add as={Fragment} to Transition component
2023-02-22 15:54:24 +02:00
Vini Brasil
35ab26308d
Fix graph element z-index (#2698)
Previously, the graph element was overlapping with other UI elements
such as the period input. This commit fixes this issue by setting the
z-index of the graph element to 0, which moves it behind other elements
in the stacking order.
2023-02-22 09:23:38 -03:00
Adam Rutkowski
ecb552479f
Upgrade erlang/elixir/alpine stack (#2695)
* Dockerfile: pin elixir-1.14.3, erlang-25.2.3, alpine-3.17.0

* asdf: pin erlang 25.2.3, elixir 1.14.3-otp-25

* Remove nonsense

* Remove erlang dupe

* Remove erlang, it's in the base image already

* Remove empty file

* Truncate seconds in NaiveDateTime instances

* Bump cache key

* Mitigate Error:error:0308010C:digital envelope routines::unsupported

See: https://www.newline.co/@kchan/how-to-fix-the-error-errorerror0308010cdigital-envelope-routinesunsupported--0f8d3f17

* Upgrade to latest alpine image and pin to an exact digest

* Revert "Upgrade to latest alpine image and pin to an exact digest"

This reverts commit cdf6624efa.

* Pin alpine image to an exact digest

---------

Co-authored-by: Cenk Kücük <c@cenk.me>
2023-02-22 09:49:43 +01:00
Vini Brasil
c307057386
Fix interval selector z-index (#2696)
This commit fixes a bug where the interval selector was not clickable
because it was behind the graph.
2023-02-21 13:57:44 -03:00
Vini Brasil
db9b4b81fb
Refactor: Graph Dataset functions (#2690)
* Refactor graph dataset building function

This commit refactors the graph dataset building functions into three
separate functions, one for each plot: main plot, comparison plot, and
dashed plot.

* Merge default graph dataset options

This commit moves default options from graph building functions to the
main function.

* Remove truncateToPresentIndex function

* Include dashed part when comparing periods

This commit fixes a bug where some period + interval combinations were
not working properly with comparisons. This is because `present_index`
is not available in every period, e.g. Last Month.
2023-02-21 11:36:26 -03:00
Adam Rutkowski
ef894b0546
Fail the CI if tests emit log errors/warnings (#2694)
* Make CI fail if tests emit warning/error logs

* Swallow log output in GA test case
2023-02-21 10:38:22 +01:00
Uku Taht
3ed5e9d27d
Refactor visits metric internals (#2691)
* Stop relying on implicitly selected `visits` metric

* Make the `visits` metric more explicit in imported data queries

* Rename internally used `visits` to `__internal_visits`
2023-02-20 14:40:36 +02:00
Uku Taht
531dfb114b
Refactor: Use HeadlessUI for search-select component (#2676)
* Use HeadlessUI for search select box

* Remove downshift from package.json

* More consistent API for Combobox component

* Combine toFilterType and valueWithoutPrefix into a single function

* Rename MyCombobox -> PlausibleCombobox

* Update webpack-cli

* Disable cache for build

* Revert "Disable cache for build"

This reverts commit aa130541f8.

* Disable cache for build

* Update webpack dependencies

* Remove glob from webpack config

* Webpack is required by package.json

* Require autoprefixer in postcss config

* Revert build changes

* Fix styling for dark mode
2023-02-20 11:10:11 +02:00
Vini Brasil
565036c9b2
Refactor: Graph Tooltip (#2685)
* Move GraphTooltip to its own file

* Convert graph tooltip module to JSX

* Improve graph tooltip code

* Change graph tooltip element to `<aside>`

* Add arrows to comparisons percentage difference

* Revert graph tooltip JSX change
2023-02-16 10:11:58 -03:00
Uku Taht
412e8df41b
Validate property key in breakdown API endpoint (#2686)
* Validate property key in breakdown API endpoint

* Add missing properties

* Do not allow empty custom prop
2023-02-16 14:34:11 +02:00
Uku Taht
66d3025b72
Use hooks instead of classes for DatePicker (#2673) 2023-02-16 13:30:37 +02:00
Vini Brasil
ce03b5ebd7
Move CH migration from clickhouse_repo/ to ingest_repo/ (#2683)
This commit moves a migration that was created in `clickhouse_repo/` to
its correct folder, `ingest_repo/`. The migration was created in
1cb07efe6d prior the read/write
separation.
2023-02-16 09:00:26 +01:00
Vini Brasil
535874be6f
Reject unknown imported cities from queries (#2675)
* Reject unknown imported cities from queries

This commit fixes a bug where the city report returned `N/A` entries.
The functions that build imported data queries were using SQL
`COALESCE`, assuming city data is `NULL` when unknown, when actually its
unknown value is `0`.

This commit addresses the problem using SQL `NULLIF` combined with the
previous `COALESCE` call. With this change both `0` and `NULL` are
treated as unknown.

Since 1cb07efe6d cities can be `NULL`, but
previously we saved `0` as unknown.

Closes #1960

* Add entry to CHANGELOG

* Ignore cyclomatic complexity Credo check
2023-02-15 07:35:35 -03:00
RobertJoonas
6f4d479852
Refactor top stats and graph positioning (#2662)
* Position the graph component relatively

* lose graph container height flicker (resized canvas)

* rename variable

* prevent flicker in the loading process

* remove unused import

* disable cursor-pointer class for minute interval

* change var names and loading state conditions

* refactor showLoader condition

* fix UI bug with interval change

set `graphData` to `null` so that the component would fade out when
loader becomes visible. Before, the spinner was just displayed on top of
the graph.
2023-02-14 09:59:25 -03:00
Vini Brasil
1cb07efe6d
Save city name when importing from GA (#2608)
This commit adds city data to imported records from Google Analytics. The
current implementation sets city to 0 because GA does not use the GeoNames
database.

Google Analytics Reporting API uses [Geographical IDs](https://developers.google.com/analytics/devguides/collection/protocol/v1/geoid)
to identify cities and countries. Plausible uses
[GeoNames](https://geonames.org/) and I couldn't find databases corelating the
two.

Fortunately, GA also returns the city name and this commit uses the city name
and the country ISO code to find the Geoname ID. To avoid making expensive ETS
searches, I created another ETS table in the Location library that uses
{country, city} as a key.

Related PR: https://github.com/plausible/location/pull/3
2023-02-14 09:32:18 -03:00
Uku Taht
8fcf4d3304
Fix 'field key does not exist' error (#2674) 2023-02-14 08:57:38 -03:00
Aviral
0db52ff977
Add year to X axis of multi-year graph (#2607)
* Add year to X axis of multi-year graph

* Remove hardcoded condition for rendering year string, render if multiple years present in graph view

* Apply new argument contract to dateFormatter usage in graph-util.js

* Better defensive runtime type guard when deriving hasMultipleYears

* Remove console log

* Remove unrelated JSDoc change

---------

Co-authored-by: Vini Brasil <vini@hey.com>
2023-02-14 08:54:22 -03:00
Davy Landman
0026487fb5
Added certs to runtime container to enable SSL for SMTP (#2537) 2023-02-14 12:21:37 +02:00
ruslandoga
e9ba60c8b4
improve transfer ownership error message (#2651)
* improve transfer ownership error message

* add changelog

* simplify

* revert changeset invitation error message

* more descriptive error message

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2023-02-13 15:25:17 +01:00
Kevin Gugelmann
35e7a3d699
Fix README typo (#2667) 2023-02-13 10:46:29 +02:00
Adam Rutkowski
8f85b110aa
Split Clickhouse pools into Read-Only and Read/Write (dedicated to writes) (#2661)
* Configure ingest repo access/pool size

If I'm not mistaken 3 is a sane default, the only
inserts we're doing are:

  - session buffer dump
  - events buffer dump
  - GA import dump

And all are serializable within their scopes?

* Add IngestRepo

* Start IngestRepo

* Use IngestRepo for inserts

* Annotate ClickhouseRepo as read_only

So no insert* functions are expanded

* Update moduledoc

* rename alias

* Fix default env var value so it can be casted

* Use IngestRepo for migrations

* Set default ingest pool size from 3 to 5

in case conns are restarting or else...

* Ensure all Repo prometheus metrics are collected
2023-02-12 17:50:57 +01:00
Uku Taht
4494252cce
Add data-domain to tracking script (#2663) 2023-02-09 15:12:46 -03:00
Uku Taht
3b73836136
Fix unknown bind meta (#2654)
* Fix 'unknown bind meta' error

* Changelog

* Use one case instead of nested ifs

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2023-02-09 14:56:21 +01:00
Uku Taht
1cc5ac0b11
Fix bug with multiple ARRAY JOIN query (#2653)
* Fix bug with multiple ARRAY JOIN query

* Add changelog entry

* Remove 'bug' label from test description

* Simplify pattern match

Co-authored-by: Adam Rutkowski <hq@mtod.org>

* Rename has_join_with_table -> joins_table?

Co-authored-by: Adam Rutkowski <hq@mtod.org>

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2023-02-09 14:14:09 +01:00
Adam Rutkowski
ac1f44f89a
Bump clickhouse images to 22.8.13.20-alpine (prod/CI) (#2660)
* Bump clickhouse images to 22.8.13.20-alpine (prod/CI)

* !fixup

* Remove clickhouse-ci target
2023-02-09 08:11:03 +01:00
ruslandoga
c3e736aae6
add flash ttl (#2652) 2023-02-07 10:08:17 -03:00
Vini Brasil
5159e683cc
First pass at Comparisons (#2650)
This commit adds support for comparing the actual showed period on the main graph with the previous one. This is a first pass and it's hidden under a feature flag because it's not feature complete yet as we want to support other comparison modes.
2023-02-07 10:00:49 -03:00
ruslandoga
7b2f4c99ee
Support alternative mailing services (Mailgun, Mandrill, Sendgrid) (#2649)
* more bamboo adapters

* add changelog

* add tests
2023-02-07 12:56:47 +01:00
Adam Rutkowski
8a2df294f5
Replace empty OS/screen size/Browser with (not set) in UI/API/exports (#2646)
* Make Device section components aware of (not set)

So that no extra sub-filters are possible when the unset
top item is selected.

* Support '(not set)' in breakdown/filters

* Update expectations for export tests

* Add extra tests for returning/filtering by '(not set)'

* Add changelog entry

* Remove ListReport conditional render

* Prevent redundant sub-filters

* Fix filter text rendering

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
2023-02-07 12:47:54 +01:00
Marko Saric
4ea519fd11
Some changes to the "site taken" copy (#2647)
* Some changes to the "site taken" copy

* Update site.ex

* Fix tests

---------

Co-authored-by: Vinicius Brasil <vini@hey.com>
2023-02-06 15:33:45 -03:00
Adam Rutkowski
912c9374d4
Bump clickhousex (per-query timeout override) (#2645)
https://github.com/plausible/clickhousex/pull/21
2023-02-06 12:30:39 +01:00
Adam Rutkowski
482249ffba
Better error message on site creation error (#2644)
* Ensure clear error message on site creation error

* Update colons in API error messages
2023-02-06 12:20:48 +01:00
Cenk Kücük
f6ee17a400
Use hostname for server_name (#2642) 2023-02-03 08:51:32 -03:00
Vini Brasil
a730763838
Add city name to imported_locations table (#2633)
See also #2608
2023-02-02 14:25:07 -03:00
Adam Rutkowski
f678cfab25
Prevent error reports from being sent with short feedback (#2635) 2023-02-01 05:36:14 -03:00
Vini Brasil
061cb6ec83
Remove the ability to collapse the main graph + transition bug fix (#2627)
* Remove the ability to collpase the top graph

This commit removes the ability to collapse the top graph. The graph
collapsed whenever `metric` was falsy. I removed all related code to
that. Metric now defaults to visitors.

We want to add new items to top stats, and this commit will make it
easier to change it. Also, there's currently a bug where top stats is
randomly collapsing, which should be fixed by this commit.

* Refactor graph and top stats loading state

The graph loading state shows and hides the graph conditionally
depending on whether the data is loaded, loading or refreshing.

The current code is a bit difficult to read because its big
conditionals. This commit refactors the loading state making it easier
to read.

This commit also fixes a bug where the graph wasn't fading out when
changing metrics.
2023-01-31 16:11:51 -03:00
Adam Rutkowski
8f9f032968
Delay stats deletions (#2632)
* Implement Site removal transaction

* Implement Stats removal Oban worker

* Configure site removal queue

* Call Site.Removal.run() instead of Purge.delete_site!

* Test site/stats removal

* Remove FIXME - filed a ticket

* Over-communicate lenghty deletion process to the users
2023-01-31 16:11:04 -03:00
Vini Brasil
1b9e6d9ae5
Add city geolocation data to seeds (#2626) 2023-01-31 16:15:01 +01:00
Marko Saric
66c65e0071
Changes to the site settings danger zone (#2629)
* Changes to the site settings danger zone

The idea was to:

1. Show a transfer site ownership message to owners of the site only. This will make ownership transfer feature more discoverable. Will also help people to transfer ownership of a site rather than deleting the whole thing when that makes more sense
2. Add a note that site deletion is a delayed operation
3. Make some other tiny copy changes

What did I break? :)

* Fix markup so the long text wraps sooner

* Fix indent

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2023-01-30 09:43:53 +01:00
Adam Rutkowski
ad12e1ef31
Show user feedback form on server errors (#2617)
* Move Endpoint errors setup to common config

* Implement naive Sentry link resolver

* Implement error report e-mail

* Delete static sentry script

* Implement user feedback form on server errors

* Re-arrange pipe

* Use Sentry.Config.dsn() where applicable

* Fix typo

* Use Map.replace/3
2023-01-25 15:15:41 +01:00
Adam Rutkowski
5bb53c7e35
Revert "parse date labels by local timezone instead of UTC (#2601)" (#2619)
This reverts commit 7a7076ed20.
2023-01-24 10:43:43 +01:00
Marko Saric
c1d174d0fa
Some changes to the "locked site" copy (#2615)
Some changes to be more consistent with the emails we send. Also "valid" subscription rather than "active" subscription fits better for the different cases where this screen is shown
2023-01-23 09:02:21 -03:00
RobertJoonas
7a7076ed20
parse date labels by local timezone instead of UTC (#2601) 2023-01-23 12:32:12 +02:00
RobertJoonas
bd0de97521
Fix returning more pageviews with a visit property filter (#2612)
* fix subquery for sessions in base_event_query/2

As the 'sessions' table is using the CollapsingMergeTree engine, we have
to select session_id's distinctively. Otherwise we will get multiple rows
(with sign -1 and 1) as long as the background merge hasn't happened.

* update changelog

* use GROUP BY instead of SELECT DISTINCT

* remove comma
2023-01-23 12:14:27 +02:00
ruslandoga
8640ba499f
fix clickhouse-ci rule in Makefile (#2614) 2023-01-20 10:52:39 -03:00
ruslandoga
e3f095ade7
fix class name typo and include it in safelist (#2610) 2023-01-20 10:41:04 -03:00