- `:verification` flag is no longer needed,
however we'll keep the old "awaiting first pageview" flow
for Community Edition where verification is not available
by default
- `:traffic_drop_notifications` have been tested and should
be enabled for everyone
* Expose current visitors 12h aggregate
* Remove unused site association
* Distinct drop/spike notification factories
* Rename modules accordingly + implement drop handling
* Rename periodic oban service
* Implement drop email
* Rest of the owl
* Update changelog
* Update moduledoc
* Update moduledoc
* Min threshold to 1
* Threshold 1
* Remove merge artifact
* Put panel behind a feature flag
* Format
Some users are reporting wrong results for stats querying from events table
but with session filters.
This seems to be caused by sampling: Namely both tables end up sampled and the
incorrect sample rate seems to get used.
This is sadly impossible to test for in our test suite, but reveals itself in
production with the following query:
```sql
SELECT
toUInt64 (round(uniq (se0.user_id) * any(_sample_factor))) AS visitors
FROM
events_v2 AS se0 SAMPLE 20000000
INNER JOIN (
SELECT
sss0.session_id AS session_id,
any(_sample_factor) AS _sample_factor
FROM
sessions_v2 AS sss0 SAMPLE 20000000
WHERE
(sss0.site_id = _CAST (8195000, 'Int64'))
AND (sss0.timestamp >= _CAST (1720497600, 'DateTime'))
AND (sss0.start < _CAST (1720584000, 'DateTime'))
AND (sss0.entry_page IN ('/'))
AND (sss0.sign = 1)
GROUP BY
sss0.session_id
) AS ss1 ON se0.session_id = ss1.session_id
WHERE
(se0.site_id = _CAST (8195000, 'Int64'))
AND (se0.timestamp >= _CAST (1720497600, 'DateTime'))
AND (se0.timestamp < _CAST (1720584000, 'DateTime'))
```
By removing the inner query SAMPLE clause the result changes >50%
Related old PR: https://github.com/plausible/analytics/pull/2962
This will likely require more work in the future so looking forward to that -
after my vacation.
* Refactor Expression.dimension to accept q
* Handle quarter- and half-hour timezones
Previously APIv2 output didn't start at a full hour for these time zones
and main graph was blank
The core reasoning is that ClickHouse `timeSlots` is not time-zone
aware and works off of unix epoch - meaning that in time zones which
have an offset of 5:45 the "hours" reported would start at minute :45.
The fix is kind of silly - we now divide each hour into 4 and handle
things that way.
Related basecamp issue: https://3.basecamp.com/5308029/buckets/36789884/card_tables/cards/7590936581
* Fix test typo
Some comparisons return __blank__ values. After recent time series fix,
this blew up when trying to parse __blank__ as a date.
This fixes the issue and adds a test for a period with __blank__ values
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
Previously, revenue metrics were reported as 0.0$ when imports were included with
APIv2. This is because the metrics were not properly included/forwarded by the
imports query.
Not sure how long this bug has persisted, but eh.
* Fix filter ["contains", "visit:country", ["E"]]
This blew up since FixedString columns dont natively support matching functions for some reason
* Dont blow up for 3-letter country codes, instead fail validation
* Allow sending numbers for visit:city along with strings
This simplifies querying as we pass the list of cities as numbers to clients
* use &is_integer/1
* Remove redundancy
* Remove a dead method
* Move select_event/session_metrics to within QueryBuilder
* Make a method private
* Move page_regex util around
* Move utc_boundaries helper around
* Fixups for utc_boundaries
* Add legacy notices
* Move Stats.query method around
* include_sentry_replay_info consistently
* Move filters out of select_group_fields
* Collapse conditions under select_group_fields
* Shorten some imported methods
* Use dimension over dim in group_by
* Separate SQL query building from imported.ex
* props.ex -> legacy_dimensions.ex
* Move some query building out of Query.ex
* Remove unneeded method
* put_filter -> add_filter
* Remove some query setters
* Moduledoc
* Split out validations and import tests from query_test
* Move tests around
* Split event:goal tests from query_test
* Remove redundant filters
* Remove dead code
* Split special metrics tests from query_test
* Legacy module
* Move fragments module under Plausible.Stats.SQL
* Introduce select_merge_as macro
This simplifies some select_merge calls
* Simplify select_join_fields
* Remove a needless dynamic
* wrap_select_columns macro
* Move metrics from base.ex to expression.ex
* Move WhereBuilder under Plausible.Stats.SQL
* Moduledoc
* Improved macros
* Wrap more code
* select_merge_as more
* Move defp to the end
* include.time_labels parsing
* include.time_labels in result
Note that the previous implementation of the labels from TimeSeries.ex was broken
* Apply consistent function in imports and timeseries.ex
* Remove boilerplate
* WIP: Limited support for timeseries-with-querybuilder
* time:week dimension
* cleanup: property -> dimension
* Make querying with time series work
* Refactor: Move special metrics (percentage, conversion rate) to own module
* Explicitly format datetimes
* Consistent include_imported in special metrics
* Solve week-related crash
* conversion_rate hacking
* Keep include_imported consistent after splitting the query
* Simplify do_decide_tables
* Handle time dimensions in imports cleaner
* Allow time dimensions in custom property queries
* time:week handling continued
* cast_revenue_metrics_to_money
* fix `full_intervals` support
* Handle minute/realtime graphs
* experimental_session_count? with timeseries
This becomes required as we try to include visits from sessions by default
* Support hourly data in imports
* Update bounce_rate in more csv tests
* Update some time-series query tests
* Fix for meta.warning being included incorrectly
* Simplify imported.ex
* experimental_session_count flag removal
* moduledoc
* Split interval and time modules
* Move fragments module under Plausible.Stats.SQL
* Introduce select_merge_as macro
This simplifies some select_merge calls
* Simplify select_join_fields
* Remove a needless dynamic
* wrap_select_columns macro
* Move metrics from base.ex to expression.ex
* Move WhereBuilder under Plausible.Stats.SQL
* Moduledoc
* Improved macros
* Wrap more code
* select_merge_as more
* Move defp to the end
* wrap_alias
* Refactor and unify auth plugs for Stats and Sites APIs
* Expose get site Sites API endpoint to all API keys
* Test the new plug
* Add test for endpoint with modified scope
* Fix typos
Co-authored-by: hq1 <hq@mtod.org>
* Rename plug for consistency (h/t @aerosol)
---------
Co-authored-by: hq1 <hq@mtod.org>
* Revert "Revert "APIv2: Replace breakdown module with QueryBuilder (#4283)" (#4292)"
This reverts commit ef5e0e0382.
* Allow querying events and pageviews from sessions table
This is not strictly accurate, especially with shorter time frames, but
is useful for a fallback mechanism. I'll figure out something around
shorter time frames in the future.
See also: https://github.com/plausible/analytics/pull/4292
* Only query events and pageviews in legacy breakdowns
* WIP: Breakdown using QueryBuilder
* Revert "Remove problematic test"
This reverts commit b442bb5d1f.
* Get more breakdown tests passing
* Preload goals, sort when dealing with time_on_page
* Handle conversion_rate in breakdowns
* Simplify ordering by using selected_as consistently for dimensions
* Get breakdown tests passing
* Strings to atoms in keys for StatsController.transform_keys calls to work
* Handle revenue metrics removal
* Add test for nil-removal case
* Include percentage metric
* Fix and test with imported locations
* Fixup time-on-page
* Fix country/region automatic filters
* Handle multiple imports (os/browser version) in importsv2
* Filter goals
* Default to ordering by page as well
* Calculate conversion rate on sessions if needed
* Order by event dimensions - handles event:page special case
* Update tests
* Update more tests, handle goal=0 case in imports
* Handle event:goal breakdowns correctly with filters
* Revenue to money
* Improved table deciding
* Also update event:page filters on event:page breakdown
* bounce_rate to 0
Previous behavior relied on two queries being made - new query leads to 0 naturally
* Update pagination test
* dont count non-pageviews as path goal completions
* Make revenue logic breakdown-specific
Its hard to fit into the new schema and likely needs a rethink for apiv2
* Retain previous behavior for TimeSeries module
* Get GA4 test passing
Most failures are related to ordering, pageviews shouldnt be read off of sessions
* Clean up old methods
* Simplify imported.ex
* Dont crash on garbage filters
* Reflect ordering-related change in test
* Fix test data
* Update table_decider
* Re-simplify get_revenue_tracking_currency
* Revert revenue changes
* Use Query.set
* Remove a TODO
* csv importer: no pageviews
Pageviews were incorrectly fetched from sessions table before, causing issues
* csv importer tweaking
* Remove use Plausible
* to_existing_atom
* Add some aggregates tests
* Port aggregates tests to do with filtering
* Session metrics can be queried with event: filters
* Solve a typo
* Update a validation message
* Add validations for views_per_visit
* Port an aggregation/imports test
* Optimize time dimension, add tests
* Add first timeseries test, update parsing tests
* Docs for SQL.Expression
* Test timeseries more
* Allow time explicitly in order_by
* Add multiple breakdowns test
* Refactor QueryOptimizer not to care about time dimension placement in dimensions array
* Add test breaking down by event:hostname
* Add hostname filtering logic to QueryOptimizer, unblock some tests
* WIP: Breakdown by goal
* conversion rate logic for query api
* Update more tests
* Set default order_by
* dimension_label
* preloaded_goals in tests
* inline load_goals
* Use Date functions over Timex
* Comments
* is_binary
* Remove special form used in tests
* Fix defmodule
* WIP: Fix memory leak, event:page breakdown logic
* Enable more tests, fix for group_conversion_rate without explicit visitors metric
* Re-enable a partially commented test
* Re-enable a partially commented test
* Get last test passing
* No imports order_by in apiv2
* Add a TODO
* Remove redundant Util call
* Update aggregate.ex
* Remove problematic test
* add notice about pending ownerships counting towards usage
* Refactor feature usage to take site_ids too
* Add option to query usage from pending ownerships
* count pending ownerships towards usage on the choose-plan page
* turn 'eligible_for_upgrade?' into a predicate
* fix variable names on ce
* fix unused alias on ce
* fix ce test
* Use max `pageSize` when fetching data for import from UA
* Update test fixture
---------
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
* WIP new querying
* WIP: Move some aggregate code under new command
* WIP: Add joins, handling less metrics
* join events table to sessions if needed
* Merge imported results with built query
* Remove dead code
* WIP: /api/v2/query
* Allow grouping by time
* Use JOIN for main query
* Build query result
* update parse_time
* Make joinless order by work
* First test
* more breakdown tests
* Serialize event:goal filters in an json-encodable way/reflection
* Handle inner vs outer ORDER BY clauses properly
* Handle single conversion_rate metric
* Update more tests
* Get parsing tests passing again
* Validate filtered goal filter is configured
* Enable more validation tests
* Enable more event:name breakdown tests
* Enable more breakdown tests
* Validate site has access to custom props
* Validate conversion_rate metric which is only allowed in some situations
* Validate that empty event:props: is not valid
* handle query.dimensions properly in table_decider
* test more validations on metrics/dimensions
* Validate session metrics in combination with event dimension(s)
* Tests cleanup
* Parse include.imports
* Get imports working with new querying
* Make more imports tests work
* Make event:props:path imports-adjacent test work
* Get query imports warning-related tests running
* Remove dead pagination tests
* Solve dead import
* Solve some warnings
* Update aggregate metrics tests
* credo
* Improve test naming
* Lazy goal loading
* Use datetime methods
* Ecto -> SQL module name
* Remove Expression.dimension mode option
* Make goals accept wildcards in :is queries
* Add support for contains/does_not_contains in the backend
* Support all operations for event custom properties
* Support does_not_contain on the frontend
* Changelog entry
* Render filter operations nicely for does not contain
Found 3 extra pixels for operation dropdown
* Remove multiple_filters feature flag
* Prioritize CSP over GTM error in case both detected
* Outline better message for persistent headless timeout
* Bump headless waiting times slightly
* Add generic type of error
For known cases, that we rather not reveal much
details about.
* Add the malformed script tag case
It turns out Kaffy uses pretty crude solution for large select widget, where
jQuery's `.val(value)` is called on input element. This in turn does not trigger
any events and the behavior is not extensible in any sane way. The prefill
callback is now called periodically to ensure any change in the input gets
picked up and acted upon eventually when conditions are met.
* Set default selected tab to last cycle when rendering usage
...And never disable this tab.
* rename Ongoing cycle to Upcoming cycle
* fix tests
* mix format
Kaffy uses inconsistent field ID when changing between select and standard input for large datasets.
This PR accounts for that change when applying prefill.
* Implement autoprefill of enterprise plan fields on user change
* Implement sanitizing input attrs in enterprise plan CRM form
* Implement number formatting for monthly pageview limit input in CRM form
* Link subscription status to Paddle profile whenever available
* Display usage info directly in user form in CRM
* Handle pageview limit fallback gracefully when rendering subscription quota
* Remove usage page link from users list view in CRM
* Add tests for usage page
* Apply `raw/1` only to the one (and only) element of the list
* pull last_bill_date from paddle sandbox in mix task
* move cycle usage checks to Quota module
* move quota.ex to a subfolder
* split up Quota module
* set choose-plan pageview slider according to usage
* silence credo
