* fractional_sample_rate: Vary sample rate according to traffic in the past 30 days
Our old sampling mechanism used SAMPLE 20000000 syntax. This was
wonderful since it allowed essentially dynamic sampling based on the
data being queried. However this ran into many issues relating to JOINs
and sample rate being different for different tables.
Instead, we now start to dynamically vary sample rates fractionally.
At query time we check the time window being queried and estimate how
many rows this query might reach. For large queries, we then dynamically
decide the sample rate.
For getting the traffic estimate for a site, we have a new SampingCache class which queries `ingest_counters`.
The query being cached is slightly expensive and can be sped up with a
ClickHouse projection.
* ce behavior
* Fix test
* Update test/plausible/stats/sampling_cache_test.exs
Co-authored-by: hq1 <hq@mtod.org>
* Update tests
* Update tests
---------
Co-authored-by: hq1 <hq@mtod.org>
* Reduce "choose plan" reliance on the old schema
* Fix team members usage computation on teams schema
* Switch CreateInvitation to reading from team schemas behind FF (WIP)
* Allow test-inviting one guest into multiple sites
* Convert another test case where team members count is wrong
cc @zoldar
* WIP: support site transfer notification e-mails
* Even more strict SiteTransfer fetching
* Make skipping permissions work
* Make CreateInvitation read from team schemas behind FF fully
* Fix passing options to `check_invitation_permissions`
* Fix allowance check for pageview usage for active or recently ended trial case
* Fix `check_invitation_permissions`
* Remove no longer relevant invite implementations for Teams
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* migration: add scroll_depth to events_v2
* (cherry-pick) ingest scroll depth
* replace convoluted test with more concise ones
* QueryParser: parse internal scroll_depth metric + validation
* turn QueryComparisonsTest into QueryInternalTest
* rename file
* (cherry pick) query scroll depth 15b14d3
...and move the tests into `internal_query_test.exs`
* review feedback
* Get rid of unnecessary separation between aggregate and group scroll depth
* Drop irrelevant other metrics in tests
* add test ensuring scroll depth unavailable in Stats API v1
* Put scroll depth on the dashboard
* Top Stats
* Main Graph
* Top Pages > Details
* feature flag for dashboard scroll depth access
* ignore credo warning
* enable scroll_depth flag in tests
* remove duplication
* write timestamps explicitly in a test
* revert moving tests around
* Add query_comparisons_test back
* Move scroll_depth tests into query_test
* Delete query_internal_test
* rename setup util (got updated on master)
* use pageleave_factory where applicable
* Use the correct generated query-api.d.ts
* npm format
* Switch upgrade to enterprise plan view to teams schema behind FF
* Switch change plan preview action to teams schema behind FF
* Switch remaining billing controller actions to teams schema behind FF
* Switch on teams schema in choose plan view behind FF
* Proxy via Read adapter where applicable for billing context
* Proxy remaining plan-related functions
* Switch enterprise_configured?/1 tests to use the adapter
* Format
* Update SiteLocker tests
* Actually use `has_active_subscription?/1` billing adapter
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Populate `current_team` to site's team and make site and subscription preloads consistent
* Accept only full `User` struct in `Users.get_for_user(!)`
* Make all uses of `Sites.get_for_user(!)` switch on team schema
* Remove redundant preloads for funnel/props settings
* Use adapter transitions in subscription settings
* Use team's schema subscription when listing invoices
* Fix typespec
* Turn owned site IDs into a specific query
* Add clauses for when FF is on but no team has been created
* Fix formatting
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Clamp dates on both directions
We could still get warnings about negative date ranges when the date
being queried is in the future.
This could happen if the users local time is in the future for some
reason or they manually edit the url.
* Update cond
* Update acquisition channel UDF to prioritize display over paid search
* Remove migration
Will run this manually together with a backfill, self-hosted will get this for free.
* Add test
---------
Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
* Expose site limit, usage, ensure_can_add_new_site via Adapter
* Print to stdout if TEST_READ_TEAM_SCHEMAS is enabled
* Add factory wrappers for remaining subscription types
* Ensure consistent ordering when fetching latest subscription
* Switch creating new site to read team schemas
* Dedup code based on read team schemas switching
* Switch to transitional factory where necessary
* Update yet another test requiring transitional factory
* Modify test utils to use teams test factories
* Implement alternative routes for updating and removing membership
* Implement teams read adapter for listing site members and invitees
* Use new teams read adapter for Settings > People view
* Add `invitation_id` column to `guest_invitations` schema
* Add `invitation_id` to `GuestInvitation` schema and populate it
* Sync guest invitation's invitation ID instead of team invitation
* Expose guest invitation's invitation ID in sites list
* Sync guest invitation invitation ID instead of team invitation in backfill
* Update team consistency check to account for guest invitation IDs
* Remove workaround for no invitation ID on guest invitation in `list_people`
* Test listing pending invitations
* Test listing memberships
* Format
* Test membership changes via new routes
* Remove old membership altering routes
* Clean up
* Revert "Modify test utils to use teams test factories"
This reverts commit 5eb8754782.
* Ensure test setup provisions teams for people listing
* See if we can avoid exposing user id
* Revert "See if we can avoid exposing user id"
This reverts commit 672429b9d1.
* Fix faulty member label in people list
* Fix sites listing for a case of pending invite with existing pin
---------
Co-authored-by: hq1 <hq@mtod.org>
* Solve noisy warnings about `a negative range was inferred for Date.range/2`
query.now is in utc but the date range was in the querys timezone
This is visible in the pattern on the graph:
* Add tests
* Implement user owning existing and pending sites check for teams
* Add predicate checking whether user has any existin or pending sites via teams
* Check need to upgrade for sites list via teams when FF is up
* Backfill teams for users on trial without a team
* Create team for users who register with trial started
* Replicate trial start logic on user create in team factories
* Make `ensure_can_take_ownership` in sites LV work via teams too
* Dispatch feature access check from `/sites` to Teams-schema reads
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Remove query.v2 flag
This was originally used for making sure queries use the right table as
migrating to APIv2. This is no longer needed
* Remove experimental_reduced_joins flag
* Expose a few data migration functions, add quiet option to do_run
* Create functions and test acquisition channel logic in clickhouse
Tests were lifted from test/plausible_web/controllers/api/external_controller_test.exs
* Clean up test code a bit
* Property test for acquisition channels
* Handle empty strings properly in reference implementation
* Fix spelling, minor issues
* Revert "Property test for acquisition channels"
This reverts commit 3fa0e0e4eb.
* Only test clickhouse functions
* Solve minor code issue
* update channels logic
* Revert "Only test clickhouse functions"
This reverts commit e12784031a.
* Add more tests
* Add small result assertion
* Make query options explicit in data migrations
* Move multi-query running logic to within datamigration lib
* Unbreak numeric ids migration
* Named params directly to Clickhouse
* Update reference test implementation
---------
Co-authored-by: Uku Taht <uku.taht@gmail.com>
* Run all tests with `read_team_schemas` flag up and down on CI
Replacing the existing setup for `experimental_reduced_joins` which
wasn't actively used and will be removed some time in the future.
This way we don't make CI run longer than necessary.
* Add one more test case for listing invitations on /sites LV
* Intentionally reintroduce regression to test CI
* Revert "Intentionally reintroduce regression to test CI"
This reverts commit cdb83c1494.
* Run tests for `experimental_reduced_joins` together with `read_team_schemas`
* Simplify "needs_to_upgrade" check in Sites LV
* Use new schema for listing sites (WIP)
* wip: trying to filter out redundant site entries
* Move site listing logic around and continue fixing tests
* Fix Plausible.SiteTest
Expectations in some tests were adjusted. For instance,
we do not show invitations for sites where user is already
a member or owner.
* 1st stab at high-level factories covering old/new models
* Switch live /sites tests to transitional factories
* Improve prosthetics and make the remaining /sites tests pass
* Put new sites list logic behind a feature flag
* Fix listing pinned sites
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Sync removing and rejecting invitations and site transfers
* Sync invitation/transfer ID with each new invite
* Accept only subset of guest invitations on sync
* Pass test
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Revert "Revert "Remove no change icon, better alignment""
This reverts commit a69d0a14a7fe038fe110907571240ad99a2ba6a4.
* different arrows
* Render graph tooltips in react
* Use ChangeArrow in graph tooltips
* Only display a single date if viewing a single date
* Tests for breakdown responses with labels
* Simplify code a bit
* strokeClass
* refactor ChangeArrow
* Play with stroke widths
* jest tests
* Silence logging in jest tests
* not variable stroke width
* prettier
* Conversion rate widths
* Update a test
* Trim trailing dates for label when comparing running month
* Reformat file
* Update expect.toHaveTextContent for multiline
* Update date range labelling to be more consistent with the frontend
* reformat
* Clean site transfers after 48 hours
* Sync accepting site transfers and invitations within transaction
* Add dry run mode to teams backfill and make it a default
* Extend invitation clean worker tests
* Lost tapes: bring back premium feature notices in tiles
* Ensure that e-mail CTA is only presented to business/enterprise customers
* Format
* Fix typespecs
* Add t() type to EnterprisePlan
* Reduce Plan.t() definition
Found no reason for it to be there.
* Extend team consistency checks
* Sync team against user right after creating the site
* Prune orphaned team guest memberships and invitations on site removal
* new endpoint for custom prop value filter suggestions
* add test coverage for searching prop suggestions
* plug the new endpoint into FE
* update changelog
* format JS
* Update CHANGELOG.md
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
---------
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Ensure team is present before use in sync logic
* Ensure teams backfill works against partially assigned sites
* Associate site with team on creation
* Associate site with team on sync
* Reuse alias
* Add tests for invitation creation sync
* Move team assertions to a helper module
* Format
* Test team creation on site creation via Sites context module
* Add tests for teams sync on subscription changes
* Tag tests
* Test grace period start syncing up with teams
* Test grace period manual lock sycning w/ teams
* Test grace period end sycing up w/ teams
* Test clearing grace period sync with teams
* Update moduledoc
* Fix missing preloads and wrong result pattern matching in sync logic
* Test sync on accepting invites and site transfers
* Test sync on membership role update and member removal
* transfer async fix WIP
* Stop privisioning team in site factory
* Remove unused relationship from Site schema
* Ensure consistent parsing of `passthrough` from Paddle webhook
* Update team passthrough notification tests & logic
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Log warnings on HTTP GA4 import errors and add Sentry context on socket errors
* Treat socket and 5xx errors as recoverable (up for retry in ~1hr) in GA4 imports
* Add filter clauses for each main result filter
This handles the case where main query has a limit and results change. Doesnt handle metrics like percentage.
* Fix percentage calculations by ignoring breakdown-related filters in totals queries
* Refactor comparisons test suite
* Move comparisons logic to comparisons module
* New route for internal query tests
Only to be used in testing
* Support comparison queries with imports/breakdowns
* time dimension predicate extraction
* Clean up a test
* Update docstring
* Update route test
* fix a typo
* Comparisons: Move code to LegacyQueryBuilder
* WIP: Return comparison results to frontend
* refactor: remove useless param
* Different result format
* Pass object to metric.renderValue
* remove dead code
* Fixup response format
* Comparison in a tooltip
Not perfect at all, but a good start. Problems arise with money etc.
* Simple change arrow
* Extract metric entry to ts
* popper attempt WIP
* Slightly nicer content
* Solve warning
* Unified changeArrow in app
* Remove needless spanning
* Always set `graph_metric` in top stats.
FE already has business logic around whether a given metric is graphable
* Remove dead code
* Move Money module under dashboard utils, keep in build
* change <Metric /> definition to take in a `formatter` and store default formatters in another (typed) const
* Use standard system for formatting numbers
* Arrows only in table
* remove dead import
* Inline renderValue
* Render metric name in tooltip
* numberFormatter -> numberShortFormatter
* numberShortFormatter update
* Separate long/short formatters
* Use long vs short formatters
* Put column name into tooltip
* Slightly improved label handling for percentages, conversion rate
* Improved boundary handling in tooltip.js
* Iterate tooltips, no tooltip for - revenue
* Update top stats tests after graph_metric change
* Change revenue metrics stats API return structure
Conversion now happens earlier in query pipeline, we return float for comparison purposes
* useQueryContext in a component
* graph_metric for current visitors to fix realtime view
* No tooltips if fully - row
* renderValue as a proper function
* Simplify MetricEntry
* Use common const
* tooltip to typescript
* More explicit return structure
* metric-entry -> metric-value
* Restore some files
* ChangeArrow
* Restore MoreLink
* Fix typing in MoreLink
* <MetricValue />
* Tests for MetricValue and ChangeArrow
* details modal fixups
* re-add space between arrow and percentage
* Solve stylelint issues
* Update test
* Format
* Add flag `breakdown_comparisons_ui`
* reformat
* Remove no change icon, better alignment
* Revert "Remove no change icon, better alignment"
This reverts commit a8d62b6383.
* number-formatter.ts
* numberLongFormatter refactor
* useMemo dependency
* Handle nulls/undefined in top stats
---------
Co-authored-by: Uku Taht <uku.taht@gmail.com>
* Extend schemas with new fields and relationships for teams
* Implement listing sites and sites with invitations with teams
* Implement creating invitations with teams
* Implement accepting invites with teams
* Add `Teams.SiteTransfer` schema
* Implement creating ownership transfers
* Implement accepting site transfer between teams
* Make results shapes from `Teams.Memberships` role functions more consistent
* Remove :team relation from ApiKey schema
* Pass and provision team on subscription creation
* Pass and provision team on enterprise plan creation
* Implement creating site for a team
* Keep team in sync during legacy ownership transfer and invitations
* Resolve conflict in `Teams.get_or_create` without transaction
* Abstract `GracePeriod` manipulation behind `Plausible.Users`
* Put `User.start_trial` behind `Plausible.Users` API
* Sync team fields on user update, if team exists
* Sync cleaning invitations, updating and removing members
* Transfer invitations too
* Implement backfill script
* Allow separate pg repo for backfill script
* Rollback purposefully at the end
* Update backfill script with parallel processing
* Use `IS DISTINCT FROM` when comparing nullable fields
* Handle no teams to backfill case gracefully when reporting
* Parallelize guest memberships backfill
* Remove transaction wrapping and query timeouts
* Make team sync check more granular and fix formatting
* Wrap single team backfill in a transatction for consistent restarts
* Make invitation and site transfer backfills preserve invitation ID
* Update migration repo config for easier dev access
* Backfill teams for users with subscriptions without sites
* Log timestamps
* Put teams sync behind a compile-time flag
* Keep timestamps in sync and fix subscriptions backfill
* Fix formatting
* Make credo happy
* Don't `use Plausible.Migration` to avoid dialyzer complaining
None of the tooling from there is used anywhere and `@repo` can
be defined directly in the migration script.
* Drop SSL workarounds in the backfill script
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
