* Solve noisy warnings about `a negative range was inferred for Date.range/2`
query.now is in utc but the date range was in the querys timezone
This is visible in the pattern on the graph:
* Add tests
ON CLUSTER fails since it tries to create conflicting
dll entries on each node.
Error:
```Cannot execute replicated DDL query, maximum retries exceeded.
(UNFINISHED)```
* Channels: Migration to add column, backfill code
This change adds `acqusition_channel` columns to events_v2 and
sessions_v2 tables. These columns are materialized - we don't ingest
into them directly. Instead they're calculated based on other columns.
The data migration changes now allow to also backfill the column.
Tested the ability to change definitions by changing the function
definitions and re-running the migration with backfill. Confirmed that
the underlying data changed as expected.
* quiet option
* Exclude data migrations from validation
* Migration consistency
* Channels: Fix cluster behavior
CREATE TABLE AS SELECT syntax did not work on cluster.
Instead, let's do a normal insert. For safety and to avoid timing
issues, ensure that INSERT waits for data to be inserted on all active
replicas.
* Proper replicated tables
* Fix interpolation in data_migration.ex
* Speed up calculating acquisition_channel in clickhouse
The previous `has` queries proved to be problematic and causing a lot of
CPU overhead.
Benchmarked via this query:
```sql
SELECT
channel,
count(),
countIf(acquisition_channel(referrer_source, utm_medium, utm_campaign, utm_source, click_id_param) = channel) AS matches
FROM events_v2
WHERE timestamp > now() - toIntervalHour(48)
GROUP BY channel
ORDER BY count() desc
```
Before this fix:
```
query_duration_ms: 57960
DiskReadElapsedMs: 374.712
RealTimeMs: 2891200.667
UserTimeMs: 2704024.783
SystemTimeMs: 1693.265
OSCPUWaitMs: 90.253
OSCPUVirtualTimeMs: 2705709.58
```
After this fix:
```
query_duration_ms: 4367
DiskReadElapsedMs: 454.356
RealTimeMs: 213892.207
UserTimeMs: 199363.485
SystemTimeMs: 1479.364
OSCPUWaitMs: 13.739
OSCPUVirtualTimeMs: 200837.37
```
Note that the new tables are not tracked in our schema as usual as
they're pretty much temporary tables to create the dictionary without
needing to upload files to clickhouse servers.
* CREATE OR REPLACE table with SELECT
* Implement user owning existing and pending sites check for teams
* Add predicate checking whether user has any existin or pending sites via teams
* Check need to upgrade for sites list via teams when FF is up
* Backfill teams for users on trial without a team
* Create team for users who register with trial started
* Replicate trial start logic on user create in team factories
* Make `ensure_can_take_ownership` in sites LV work via teams too
* Dispatch feature access check from `/sites` to Teams-schema reads
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Remove query.v2 flag
This was originally used for making sure queries use the right table as
migrating to APIv2. This is no longer needed
* Remove experimental_reduced_joins flag
* Expose a few data migration functions, add quiet option to do_run
* Create functions and test acquisition channel logic in clickhouse
Tests were lifted from test/plausible_web/controllers/api/external_controller_test.exs
* Clean up test code a bit
* Property test for acquisition channels
* Handle empty strings properly in reference implementation
* Fix spelling, minor issues
* Revert "Property test for acquisition channels"
This reverts commit 3fa0e0e4eb.
* Only test clickhouse functions
* Solve minor code issue
* update channels logic
* Revert "Only test clickhouse functions"
This reverts commit e12784031a.
* Add more tests
* Add small result assertion
* Make query options explicit in data migrations
* Move multi-query running logic to within datamigration lib
* Unbreak numeric ids migration
* Named params directly to Clickhouse
* Update reference test implementation
---------
Co-authored-by: Uku Taht <uku.taht@gmail.com>
* Run all tests with `read_team_schemas` flag up and down on CI
Replacing the existing setup for `experimental_reduced_joins` which
wasn't actively used and will be removed some time in the future.
This way we don't make CI run longer than necessary.
* Add one more test case for listing invitations on /sites LV
* Intentionally reintroduce regression to test CI
* Revert "Intentionally reintroduce regression to test CI"
This reverts commit cdb83c1494.
* Run tests for `experimental_reduced_joins` together with `read_team_schemas`
* Hard-code sample rate based on fractional_hardcoded_sample_rate flag
We found cases where using a numeric sample rate would cause issues when
joining two tables due to different _sample_factor. Ref: https://3.basecamp.com/5308029/buckets/26383192/card_tables/cards/7973456592#__recording_7978780711
The proper fix is to use fractional sample rates everywhere, but this is
a whole project due to not wanting to sample small sites. For now, hard-code
sample rate for specific sites having the issue while we work on a
larger fix.
* is_number
* Simplify "needs_to_upgrade" check in Sites LV
* Use new schema for listing sites (WIP)
* wip: trying to filter out redundant site entries
* Move site listing logic around and continue fixing tests
* Fix Plausible.SiteTest
Expectations in some tests were adjusted. For instance,
we do not show invitations for sites where user is already
a member or owner.
* 1st stab at high-level factories covering old/new models
* Switch live /sites tests to transitional factories
* Improve prosthetics and make the remaining /sites tests pass
* Put new sites list logic behind a feature flag
* Fix listing pinned sites
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Increase max_parser_depth setting
This is currently causing issues in production with exports w/ comparisons, but
would blow up in other cases with sufficient data.
The reason this is now blowing up is that we create a large AND/OR condition
for breakdown comparisons. ecto-ch nests each conditional in its separate parentheses
causing us to reach the default 1000 max_parser_depth limit
We can improve this in a couple of ways:
- Not add unneccessary parentheses in ecto-ch
- Adding a new operator to make the AND/OR nesting require less conditionals, using
arrays and tuples instead
* Don't query comparisons for exports
csv exports don't actually output any comparison data, so querying for
it just causes issues.
* Sync removing and rejecting invitations and site transfers
* Sync invitation/transfer ID with each new invite
* Accept only subset of guest invitations on sync
* Pass test
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Revert "Revert "Remove no change icon, better alignment""
This reverts commit a69d0a14a7fe038fe110907571240ad99a2ba6a4.
* different arrows
* Render graph tooltips in react
* Use ChangeArrow in graph tooltips
* Only display a single date if viewing a single date
* Tests for breakdown responses with labels
* Simplify code a bit
* strokeClass
* refactor ChangeArrow
* Play with stroke widths
* jest tests
* Silence logging in jest tests
* not variable stroke width
* prettier
* Conversion rate widths
* Update a test
* Trim trailing dates for label when comparing running month
* Reformat file
* Update expect.toHaveTextContent for multiline
* Update date range labelling to be more consistent with the frontend
* reformat
* move util function to util module
* move playwright config file to tracker dir root
* update Playwright + add gitignores
* Playwright: enable reuseExistingServer (non-CI env)
* store tracker src copies to avoid re-compilation in dev env
* test pageleave on simple navigation
* fix test util fn
* rename local_test npm script
* make test util able to expect multiple requests
* test pageleaves in SPAs
* test pageleave with manual URL
* test pageleave not sent in manual when pageview not triggered
* extend util fn to refute event requests
* test pageleaves not sent in excluded hash pages
* store hashes instead of file copies to detect /tracker/src changes
* drop async
* speed up test suite
* Clean site transfers after 48 hours
* Sync accepting site transfers and invitations within transaction
* Add dry run mode to teams backfill and make it a default
* Extend invitation clean worker tests
