* migration: add scroll_depth to events_v2
* (cherry-pick) ingest scroll depth
* replace convoluted test with more concise ones
* QueryParser: parse internal scroll_depth metric + validation
* turn QueryComparisonsTest into QueryInternalTest
* rename file
* (cherry pick) query scroll depth 15b14d3
...and move the tests into `internal_query_test.exs`
* review feedback
* Get rid of unnecessary separation between aggregate and group scroll depth
* Drop irrelevant other metrics in tests
* add test ensuring scroll depth unavailable in Stats API v1
* Put scroll depth on the dashboard
* Top Stats
* Main Graph
* Top Pages > Details
* feature flag for dashboard scroll depth access
* ignore credo warning
* enable scroll_depth flag in tests
* remove duplication
* write timestamps explicitly in a test
* revert moving tests around
* Add query_comparisons_test back
* Move scroll_depth tests into query_test
* Delete query_internal_test
* rename setup util (got updated on master)
* use pageleave_factory where applicable
* Use the correct generated query-api.d.ts
* npm format
* Switch upgrade to enterprise plan view to teams schema behind FF
* Switch change plan preview action to teams schema behind FF
* Switch remaining billing controller actions to teams schema behind FF
* Switch on teams schema in choose plan view behind FF
* Proxy via Read adapter where applicable for billing context
* Proxy remaining plan-related functions
* Switch enterprise_configured?/1 tests to use the adapter
* Format
* Update SiteLocker tests
* Actually use `has_active_subscription?/1` billing adapter
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Populate `current_team` to site's team and make site and subscription preloads consistent
* Accept only full `User` struct in `Users.get_for_user(!)`
* Make all uses of `Sites.get_for_user(!)` switch on team schema
* Remove redundant preloads for funnel/props settings
* Use adapter transitions in subscription settings
* Use team's schema subscription when listing invoices
* Fix typespec
* Turn owned site IDs into a specific query
* Add clauses for when FF is on but no team has been created
* Fix formatting
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Clamp dates on both directions
We could still get warnings about negative date ranges when the date
being queried is in the future.
This could happen if the users local time is in the future for some
reason or they manually edit the url.
* Update cond
* Update acquisition channel UDF to prioritize display over paid search
* Remove migration
Will run this manually together with a backfill, self-hosted will get this for free.
* Add test
---------
Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
* Expose site limit, usage, ensure_can_add_new_site via Adapter
* Print to stdout if TEST_READ_TEAM_SCHEMAS is enabled
* Add factory wrappers for remaining subscription types
* Ensure consistent ordering when fetching latest subscription
* Switch creating new site to read team schemas
* Dedup code based on read team schemas switching
* Switch to transitional factory where necessary
* Update yet another test requiring transitional factory
* Modify test utils to use teams test factories
* Implement alternative routes for updating and removing membership
* Implement teams read adapter for listing site members and invitees
* Use new teams read adapter for Settings > People view
* Add `invitation_id` column to `guest_invitations` schema
* Add `invitation_id` to `GuestInvitation` schema and populate it
* Sync guest invitation's invitation ID instead of team invitation
* Expose guest invitation's invitation ID in sites list
* Sync guest invitation invitation ID instead of team invitation in backfill
* Update team consistency check to account for guest invitation IDs
* Remove workaround for no invitation ID on guest invitation in `list_people`
* Test listing pending invitations
* Test listing memberships
* Format
* Test membership changes via new routes
* Remove old membership altering routes
* Clean up
* Revert "Modify test utils to use teams test factories"
This reverts commit 5eb8754782.
* Ensure test setup provisions teams for people listing
* See if we can avoid exposing user id
* Revert "See if we can avoid exposing user id"
This reverts commit 672429b9d1.
* Fix faulty member label in people list
* Fix sites listing for a case of pending invite with existing pin
---------
Co-authored-by: hq1 <hq@mtod.org>
* Solve noisy warnings about `a negative range was inferred for Date.range/2`
query.now is in utc but the date range was in the querys timezone
This is visible in the pattern on the graph:
* Add tests
ON CLUSTER fails since it tries to create conflicting
dll entries on each node.
Error:
```Cannot execute replicated DDL query, maximum retries exceeded.
(UNFINISHED)```
* Channels: Migration to add column, backfill code
This change adds `acqusition_channel` columns to events_v2 and
sessions_v2 tables. These columns are materialized - we don't ingest
into them directly. Instead they're calculated based on other columns.
The data migration changes now allow to also backfill the column.
Tested the ability to change definitions by changing the function
definitions and re-running the migration with backfill. Confirmed that
the underlying data changed as expected.
* quiet option
* Exclude data migrations from validation
* Migration consistency
* Channels: Fix cluster behavior
CREATE TABLE AS SELECT syntax did not work on cluster.
Instead, let's do a normal insert. For safety and to avoid timing
issues, ensure that INSERT waits for data to be inserted on all active
replicas.
* Proper replicated tables
* Fix interpolation in data_migration.ex
* Speed up calculating acquisition_channel in clickhouse
The previous `has` queries proved to be problematic and causing a lot of
CPU overhead.
Benchmarked via this query:
```sql
SELECT
channel,
count(),
countIf(acquisition_channel(referrer_source, utm_medium, utm_campaign, utm_source, click_id_param) = channel) AS matches
FROM events_v2
WHERE timestamp > now() - toIntervalHour(48)
GROUP BY channel
ORDER BY count() desc
```
Before this fix:
```
query_duration_ms: 57960
DiskReadElapsedMs: 374.712
RealTimeMs: 2891200.667
UserTimeMs: 2704024.783
SystemTimeMs: 1693.265
OSCPUWaitMs: 90.253
OSCPUVirtualTimeMs: 2705709.58
```
After this fix:
```
query_duration_ms: 4367
DiskReadElapsedMs: 454.356
RealTimeMs: 213892.207
UserTimeMs: 199363.485
SystemTimeMs: 1479.364
OSCPUWaitMs: 13.739
OSCPUVirtualTimeMs: 200837.37
```
Note that the new tables are not tracked in our schema as usual as
they're pretty much temporary tables to create the dictionary without
needing to upload files to clickhouse servers.
* CREATE OR REPLACE table with SELECT
* Implement user owning existing and pending sites check for teams
* Add predicate checking whether user has any existin or pending sites via teams
* Check need to upgrade for sites list via teams when FF is up
* Backfill teams for users on trial without a team
* Create team for users who register with trial started
* Replicate trial start logic on user create in team factories
* Make `ensure_can_take_ownership` in sites LV work via teams too
* Dispatch feature access check from `/sites` to Teams-schema reads
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Remove query.v2 flag
This was originally used for making sure queries use the right table as
migrating to APIv2. This is no longer needed
* Remove experimental_reduced_joins flag
* Expose a few data migration functions, add quiet option to do_run
* Create functions and test acquisition channel logic in clickhouse
Tests were lifted from test/plausible_web/controllers/api/external_controller_test.exs
* Clean up test code a bit
* Property test for acquisition channels
* Handle empty strings properly in reference implementation
* Fix spelling, minor issues
* Revert "Property test for acquisition channels"
This reverts commit 3fa0e0e4eb.
* Only test clickhouse functions
* Solve minor code issue
* update channels logic
* Revert "Only test clickhouse functions"
This reverts commit e12784031a.
* Add more tests
* Add small result assertion
* Make query options explicit in data migrations
* Move multi-query running logic to within datamigration lib
* Unbreak numeric ids migration
* Named params directly to Clickhouse
* Update reference test implementation
---------
Co-authored-by: Uku Taht <uku.taht@gmail.com>
