* Update acquisition channel UDF to prioritize display over paid search
* Remove migration
Will run this manually together with a backfill, self-hosted will get this for free.
* Add test
---------
Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
* Expose site limit, usage, ensure_can_add_new_site via Adapter
* Print to stdout if TEST_READ_TEAM_SCHEMAS is enabled
* Add factory wrappers for remaining subscription types
* Ensure consistent ordering when fetching latest subscription
* Switch creating new site to read team schemas
* Dedup code based on read team schemas switching
* Switch to transitional factory where necessary
* Update yet another test requiring transitional factory
* Modify test utils to use teams test factories
* Implement alternative routes for updating and removing membership
* Implement teams read adapter for listing site members and invitees
* Use new teams read adapter for Settings > People view
* Add `invitation_id` column to `guest_invitations` schema
* Add `invitation_id` to `GuestInvitation` schema and populate it
* Sync guest invitation's invitation ID instead of team invitation
* Expose guest invitation's invitation ID in sites list
* Sync guest invitation invitation ID instead of team invitation in backfill
* Update team consistency check to account for guest invitation IDs
* Remove workaround for no invitation ID on guest invitation in `list_people`
* Test listing pending invitations
* Test listing memberships
* Format
* Test membership changes via new routes
* Remove old membership altering routes
* Clean up
* Revert "Modify test utils to use teams test factories"
This reverts commit 5eb8754782.
* Ensure test setup provisions teams for people listing
* See if we can avoid exposing user id
* Revert "See if we can avoid exposing user id"
This reverts commit 672429b9d1.
* Fix faulty member label in people list
* Fix sites listing for a case of pending invite with existing pin
---------
Co-authored-by: hq1 <hq@mtod.org>
* Solve noisy warnings about `a negative range was inferred for Date.range/2`
query.now is in utc but the date range was in the querys timezone
This is visible in the pattern on the graph:
* Add tests
ON CLUSTER fails since it tries to create conflicting
dll entries on each node.
Error:
```Cannot execute replicated DDL query, maximum retries exceeded.
(UNFINISHED)```
* Channels: Migration to add column, backfill code
This change adds `acqusition_channel` columns to events_v2 and
sessions_v2 tables. These columns are materialized - we don't ingest
into them directly. Instead they're calculated based on other columns.
The data migration changes now allow to also backfill the column.
Tested the ability to change definitions by changing the function
definitions and re-running the migration with backfill. Confirmed that
the underlying data changed as expected.
* quiet option
* Exclude data migrations from validation
* Migration consistency
* Channels: Fix cluster behavior
CREATE TABLE AS SELECT syntax did not work on cluster.
Instead, let's do a normal insert. For safety and to avoid timing
issues, ensure that INSERT waits for data to be inserted on all active
replicas.
* Proper replicated tables
* Fix interpolation in data_migration.ex
* Speed up calculating acquisition_channel in clickhouse
The previous `has` queries proved to be problematic and causing a lot of
CPU overhead.
Benchmarked via this query:
```sql
SELECT
channel,
count(),
countIf(acquisition_channel(referrer_source, utm_medium, utm_campaign, utm_source, click_id_param) = channel) AS matches
FROM events_v2
WHERE timestamp > now() - toIntervalHour(48)
GROUP BY channel
ORDER BY count() desc
```
Before this fix:
```
query_duration_ms: 57960
DiskReadElapsedMs: 374.712
RealTimeMs: 2891200.667
UserTimeMs: 2704024.783
SystemTimeMs: 1693.265
OSCPUWaitMs: 90.253
OSCPUVirtualTimeMs: 2705709.58
```
After this fix:
```
query_duration_ms: 4367
DiskReadElapsedMs: 454.356
RealTimeMs: 213892.207
UserTimeMs: 199363.485
SystemTimeMs: 1479.364
OSCPUWaitMs: 13.739
OSCPUVirtualTimeMs: 200837.37
```
Note that the new tables are not tracked in our schema as usual as
they're pretty much temporary tables to create the dictionary without
needing to upload files to clickhouse servers.
* CREATE OR REPLACE table with SELECT
* Implement user owning existing and pending sites check for teams
* Add predicate checking whether user has any existin or pending sites via teams
* Check need to upgrade for sites list via teams when FF is up
* Backfill teams for users on trial without a team
* Create team for users who register with trial started
* Replicate trial start logic on user create in team factories
* Make `ensure_can_take_ownership` in sites LV work via teams too
* Dispatch feature access check from `/sites` to Teams-schema reads
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Remove query.v2 flag
This was originally used for making sure queries use the right table as
migrating to APIv2. This is no longer needed
* Remove experimental_reduced_joins flag
* Expose a few data migration functions, add quiet option to do_run
* Create functions and test acquisition channel logic in clickhouse
Tests were lifted from test/plausible_web/controllers/api/external_controller_test.exs
* Clean up test code a bit
* Property test for acquisition channels
* Handle empty strings properly in reference implementation
* Fix spelling, minor issues
* Revert "Property test for acquisition channels"
This reverts commit 3fa0e0e4eb.
* Only test clickhouse functions
* Solve minor code issue
* update channels logic
* Revert "Only test clickhouse functions"
This reverts commit e12784031a.
* Add more tests
* Add small result assertion
* Make query options explicit in data migrations
* Move multi-query running logic to within datamigration lib
* Unbreak numeric ids migration
* Named params directly to Clickhouse
* Update reference test implementation
---------
Co-authored-by: Uku Taht <uku.taht@gmail.com>
* Run all tests with `read_team_schemas` flag up and down on CI
Replacing the existing setup for `experimental_reduced_joins` which
wasn't actively used and will be removed some time in the future.
This way we don't make CI run longer than necessary.
* Add one more test case for listing invitations on /sites LV
* Intentionally reintroduce regression to test CI
* Revert "Intentionally reintroduce regression to test CI"
This reverts commit cdb83c1494.
* Run tests for `experimental_reduced_joins` together with `read_team_schemas`
* Hard-code sample rate based on fractional_hardcoded_sample_rate flag
We found cases where using a numeric sample rate would cause issues when
joining two tables due to different _sample_factor. Ref: https://3.basecamp.com/5308029/buckets/26383192/card_tables/cards/7973456592#__recording_7978780711
The proper fix is to use fractional sample rates everywhere, but this is
a whole project due to not wanting to sample small sites. For now, hard-code
sample rate for specific sites having the issue while we work on a
larger fix.
* is_number
