* Solve noisy warnings about `a negative range was inferred for Date.range/2`
query.now is in utc but the date range was in the querys timezone
This is visible in the pattern on the graph:
* Add tests
* Channels: Migration to add column, backfill code
This change adds `acqusition_channel` columns to events_v2 and
sessions_v2 tables. These columns are materialized - we don't ingest
into them directly. Instead they're calculated based on other columns.
The data migration changes now allow to also backfill the column.
Tested the ability to change definitions by changing the function
definitions and re-running the migration with backfill. Confirmed that
the underlying data changed as expected.
* quiet option
* Exclude data migrations from validation
* Migration consistency
* Channels: Fix cluster behavior
CREATE TABLE AS SELECT syntax did not work on cluster.
Instead, let's do a normal insert. For safety and to avoid timing
issues, ensure that INSERT waits for data to be inserted on all active
replicas.
* Proper replicated tables
* Fix interpolation in data_migration.ex
* Speed up calculating acquisition_channel in clickhouse
The previous `has` queries proved to be problematic and causing a lot of
CPU overhead.
Benchmarked via this query:
```sql
SELECT
channel,
count(),
countIf(acquisition_channel(referrer_source, utm_medium, utm_campaign, utm_source, click_id_param) = channel) AS matches
FROM events_v2
WHERE timestamp > now() - toIntervalHour(48)
GROUP BY channel
ORDER BY count() desc
```
Before this fix:
```
query_duration_ms: 57960
DiskReadElapsedMs: 374.712
RealTimeMs: 2891200.667
UserTimeMs: 2704024.783
SystemTimeMs: 1693.265
OSCPUWaitMs: 90.253
OSCPUVirtualTimeMs: 2705709.58
```
After this fix:
```
query_duration_ms: 4367
DiskReadElapsedMs: 454.356
RealTimeMs: 213892.207
UserTimeMs: 199363.485
SystemTimeMs: 1479.364
OSCPUWaitMs: 13.739
OSCPUVirtualTimeMs: 200837.37
```
Note that the new tables are not tracked in our schema as usual as
they're pretty much temporary tables to create the dictionary without
needing to upload files to clickhouse servers.
* CREATE OR REPLACE table with SELECT
* Implement user owning existing and pending sites check for teams
* Add predicate checking whether user has any existin or pending sites via teams
* Check need to upgrade for sites list via teams when FF is up
* Backfill teams for users on trial without a team
* Create team for users who register with trial started
* Replicate trial start logic on user create in team factories
* Make `ensure_can_take_ownership` in sites LV work via teams too
* Dispatch feature access check from `/sites` to Teams-schema reads
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Remove query.v2 flag
This was originally used for making sure queries use the right table as
migrating to APIv2. This is no longer needed
* Remove experimental_reduced_joins flag
* Expose a few data migration functions, add quiet option to do_run
* Create functions and test acquisition channel logic in clickhouse
Tests were lifted from test/plausible_web/controllers/api/external_controller_test.exs
* Clean up test code a bit
* Property test for acquisition channels
* Handle empty strings properly in reference implementation
* Fix spelling, minor issues
* Revert "Property test for acquisition channels"
This reverts commit 3fa0e0e4eb.
* Only test clickhouse functions
* Solve minor code issue
* update channels logic
* Revert "Only test clickhouse functions"
This reverts commit e12784031a.
* Add more tests
* Add small result assertion
* Make query options explicit in data migrations
* Move multi-query running logic to within datamigration lib
* Unbreak numeric ids migration
* Named params directly to Clickhouse
* Update reference test implementation
---------
Co-authored-by: Uku Taht <uku.taht@gmail.com>
* Hard-code sample rate based on fractional_hardcoded_sample_rate flag
We found cases where using a numeric sample rate would cause issues when
joining two tables due to different _sample_factor. Ref: https://3.basecamp.com/5308029/buckets/26383192/card_tables/cards/7973456592#__recording_7978780711
The proper fix is to use fractional sample rates everywhere, but this is
a whole project due to not wanting to sample small sites. For now, hard-code
sample rate for specific sites having the issue while we work on a
larger fix.
* is_number
* Simplify "needs_to_upgrade" check in Sites LV
* Use new schema for listing sites (WIP)
* wip: trying to filter out redundant site entries
* Move site listing logic around and continue fixing tests
* Fix Plausible.SiteTest
Expectations in some tests were adjusted. For instance,
we do not show invitations for sites where user is already
a member or owner.
* 1st stab at high-level factories covering old/new models
* Switch live /sites tests to transitional factories
* Improve prosthetics and make the remaining /sites tests pass
* Put new sites list logic behind a feature flag
* Fix listing pinned sites
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Sync removing and rejecting invitations and site transfers
* Sync invitation/transfer ID with each new invite
* Accept only subset of guest invitations on sync
* Pass test
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Revert "Revert "Remove no change icon, better alignment""
This reverts commit a69d0a14a7fe038fe110907571240ad99a2ba6a4.
* different arrows
* Render graph tooltips in react
* Use ChangeArrow in graph tooltips
* Only display a single date if viewing a single date
* Tests for breakdown responses with labels
* Simplify code a bit
* strokeClass
* refactor ChangeArrow
* Play with stroke widths
* jest tests
* Silence logging in jest tests
* not variable stroke width
* prettier
* Conversion rate widths
* Update a test
* Trim trailing dates for label when comparing running month
* Reformat file
* Update expect.toHaveTextContent for multiline
* Update date range labelling to be more consistent with the frontend
* reformat
* Clean site transfers after 48 hours
* Sync accepting site transfers and invitations within transaction
* Add dry run mode to teams backfill and make it a default
* Extend invitation clean worker tests
* Lost tapes: bring back premium feature notices in tiles
* Ensure that e-mail CTA is only presented to business/enterprise customers
* Format
* Fix typespecs
* Add t() type to EnterprisePlan
* Reduce Plan.t() definition
Found no reason for it to be there.
* Extend team consistency checks
* Sync team against user right after creating the site
* Prune orphaned team guest memberships and invitations on site removal
* new endpoint for custom prop value filter suggestions
* add test coverage for searching prop suggestions
* plug the new endpoint into FE
* update changelog
* format JS
* Update CHANGELOG.md
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
---------
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Ensure team is present before use in sync logic
* Ensure teams backfill works against partially assigned sites
* Associate site with team on creation
* Associate site with team on sync
* Reuse alias
* Add tests for invitation creation sync
* Move team assertions to a helper module
* Format
* Test team creation on site creation via Sites context module
* Add tests for teams sync on subscription changes
* Tag tests
* Test grace period start syncing up with teams
* Test grace period manual lock sycning w/ teams
* Test grace period end sycing up w/ teams
* Test clearing grace period sync with teams
* Update moduledoc
* Fix missing preloads and wrong result pattern matching in sync logic
* Test sync on accepting invites and site transfers
* Test sync on membership role update and member removal
* transfer async fix WIP
* Stop privisioning team in site factory
* Remove unused relationship from Site schema
* Ensure consistent parsing of `passthrough` from Paddle webhook
* Update team passthrough notification tests & logic
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Log warnings on HTTP GA4 import errors and add Sentry context on socket errors
* Treat socket and 5xx errors as recoverable (up for retry in ~1hr) in GA4 imports
* Add filter clauses for each main result filter
This handles the case where main query has a limit and results change. Doesnt handle metrics like percentage.
* Fix percentage calculations by ignoring breakdown-related filters in totals queries
* Refactor comparisons test suite
* Move comparisons logic to comparisons module
* New route for internal query tests
Only to be used in testing
* Support comparison queries with imports/breakdowns
* time dimension predicate extraction
* Clean up a test
* Update docstring
* Update route test
* fix a typo
* Comparisons: Move code to LegacyQueryBuilder
* WIP: Return comparison results to frontend
* refactor: remove useless param
* Different result format
* Pass object to metric.renderValue
* remove dead code
* Fixup response format
* Comparison in a tooltip
Not perfect at all, but a good start. Problems arise with money etc.
* Simple change arrow
* Extract metric entry to ts
* popper attempt WIP
* Slightly nicer content
* Solve warning
* Unified changeArrow in app
* Remove needless spanning
* Always set `graph_metric` in top stats.
FE already has business logic around whether a given metric is graphable
* Remove dead code
* Move Money module under dashboard utils, keep in build
* change <Metric /> definition to take in a `formatter` and store default formatters in another (typed) const
* Use standard system for formatting numbers
* Arrows only in table
* remove dead import
* Inline renderValue
* Render metric name in tooltip
* numberFormatter -> numberShortFormatter
* numberShortFormatter update
* Separate long/short formatters
* Use long vs short formatters
* Put column name into tooltip
* Slightly improved label handling for percentages, conversion rate
* Improved boundary handling in tooltip.js
* Iterate tooltips, no tooltip for - revenue
* Update top stats tests after graph_metric change
* Change revenue metrics stats API return structure
Conversion now happens earlier in query pipeline, we return float for comparison purposes
* useQueryContext in a component
* graph_metric for current visitors to fix realtime view
* No tooltips if fully - row
* renderValue as a proper function
* Simplify MetricEntry
* Use common const
* tooltip to typescript
* More explicit return structure
* metric-entry -> metric-value
* Restore some files
* ChangeArrow
* Restore MoreLink
* Fix typing in MoreLink
* <MetricValue />
* Tests for MetricValue and ChangeArrow
* details modal fixups
* re-add space between arrow and percentage
* Solve stylelint issues
* Update test
* Format
* Add flag `breakdown_comparisons_ui`
* reformat
* Remove no change icon, better alignment
* Revert "Remove no change icon, better alignment"
This reverts commit a8d62b6383.
* number-formatter.ts
* numberLongFormatter refactor
* useMemo dependency
* Handle nulls/undefined in top stats
---------
Co-authored-by: Uku Taht <uku.taht@gmail.com>
