* Comment out legacy fields and relationships
* WIP
* WIP 2
* WIP 3
* wip
* Remove teams backfill and consistency check scripts
* WIP 3
* Fix CheckUsage tests
* Update billing/subscription tests
* WIP 4
* Make site transfer fail if some invitation already exists
* Fixup: do symmetric invitation/site transfer check
* Update UI bugs: make listing sites/inviting admins work like before
* Fix Sites test
* Fix external sites controller test
* Fix live sites tests
* Fix props availability lookup
* Fix site controller tests
* Fix billing controller tests
* WIP - accept invitation tests
* Another round of test fixes + invitations logic bugs
* users_test -> teams_test
* Update registration via invitation
Here, we still rely on "polymorphic" invitation structures,
hence the "unified by id" helper.
For now, it'll remain local unless we discover it's
needed in the broader `Teams.Invitations` context.
cc @zoldar
* Yet another round of test and bugfixes along the way
* Include team in site setup success e-mail
* Fix send_site_setup_emails worker
* Fixed almost all tests except CRM ones
* Update enterprise plan admin test
* Fix CRM + remaining tests
* Address credo warnings (modulo one FIXME)
* Remove last FIXME and rephrase the invitation test case description
* Set Team fields via User CRM transparently
* Map user reference in Enterprise Plan CRM via team owner
* Fix resource actions in user CRM
* Get rid of warning when opening create form in API keys CRM
* Stop emitting warnings when editing Enterprise Plans via CRM
* Tests: Bump await_clickhouse_count interval
* Remove XXX marker
* Fix register from invitation link in email sent for ownership transfer
* Simplify fetching all pending site ownership site IDs
* Remove commented out schema fields
* Remove unused functions
* Address flakiness in ingest counter tests
* Remove unused `Teams.Sites.create`
* Don't restart trial on team with subscription when creating site
* Account for cases of legacy teams with empty trial expiry date
* Revert "Address flakiness in ingest counter tests"
This reverts commit 60dc1e4115.
* Fix flaky ingest counters tests under load
* Attempt 2
* Pre-emptively hardcode site ids in sampling cache test
to avoid supplying the same IDs alongside with counters test,
that inserts through another repo (async).
what we're observing is, clickhouse not summing mergetree columns fast
enough, even though we wait quite a bit.
* Fix ingest counter tests by accounting for delayed summation
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* ingest missing pageleave as 255 for pageleave events
* return scroll depth as nil when no valid pageleave data in range
* also set empty comparison value as nil instead of 0
* add data migration
* Improve report performance in cases where site has a lot of unique pathnames
Ref: https://3.basecamp.com/5308029/buckets/39750953/card_tables/cards/8052057081
JOINs in ClickHouse are slow. In one degenerate case I found a user had
over 20 million unique paths in an import, which resulted in extremely slow
JOINs. This introduces a sort-of hacky solution to it by limiting the
amount of data analyzed.
Query timing without this change:
```
9 rows in set. Elapsed: 11.383 sec. Processed 49.16 million rows, 5.75 GB (4.32 million rows/s., 505.29 MB/s.)
Peak memory usage: 14.75 GiB.
```
After:
```
9 rows in set. Elapsed: 0.572 sec. Processed 49.18 million rows, 5.75 GB (86.03 million rows/s., 10.06 GB/s.)
Peak memory usage: 9.01 GiB.
```
* Splitting should no longer remove pagination. Handle special cases in special_metrics.ex
* select_merge_as in imports
This sets up selected_as aliases which will be used in a subsequent commit
* Add explicit ORDER BY to import
* Rewrite comment
* quoting
* merge conflict
* Split test
* Merge conflict fail fix
* WIP: Optional modifiers to queries
* WIP: Modifiers v2
* Use preloaded_goals when determining whether imports can be included
This was previously broken with conversion_rate totals metrics since it removed event:goal
filters but did not update preloaded_goals
* Preload goals according to modifiers
* Make case_sensitive: false work for is/contains operators
* Make modals send { case_sensitive: false } to backend for search
* CHANGELOG.md
* Typegen
* Prettier
* Refactor: more DRY where_builder for case sensitivity
* Support case_sensitive modifier for is_not/contains_not
* Cleanup
* credo
* remove defaults
* negating a previously set filter
* Pre-emptively introduce `site.team_owner` relation
* Drop null constraint on user_id from subscriptions and enterprise_plans
* Temporarily remove populating old schemas in Teams.Test
* Point to site.owner via new schema
* Switch more reads to teams schema WIP
* Fix AuhtorizeSiteAccess test
There's no need to translate `admin`<->`editor` here,
the redundancy is inlined wherever the plug is initialized.
* Fix regions test
* Fix main graph test
* Fix authorization test
* Try to rely on team for subscription/plans where applicable
* Test fixes
* Fix plans test
* Prep for CheckUsage changes
* Skip remaining CheckUsage tests for now
* Fix user deletion to account for team relations
* Fix HelpScout tests
* 💀 Modify ingestion to read team schemas
* Made all tests green except skipped ones
* Mute warnings about transferring site with no order
By making artificial site membership struct,
when reading data off team membership schema.
* Fix site removal test case
* Re-enable locked site tests, that don't have to rely on `SiteLocker`
* Format
* Revert "Mute warnings about transferring site with no order"
This reverts commit 0e45f8c9d9.
* Re-enable old models and fix remaining tests
* Use new factories in a long running minio test
* FIXME->TODO
* Fix remaining tests in legacy mode (no FF raised)
* oof
cc @zoldar
* Add missing definitions of editor role in FE code
* Remove no longer relevant comment about roles
* Fix JS formatting
* Always prioritize site transfers over memberships in sites list
* Fix misaligned "Reject" invitation button
* Fix site pinning when user is guest in multiple sites in team
* Fix subscription settings controller tests
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Move `bulk_transfer_ownership_direct` under `AcceptInvitation`
* [WIP] Switch ownership transfer operations to read from team schemas behind FF
* Fix usage test regression
* Semantics - current user; ownership is not necessarily involved
* Perform remaining read via adapter; remove obsolete test
* Properly list site with pending site transfer while being guest on a team
* Account for pending site transfers in Settings > People list
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* fractional_sample_rate: Vary sample rate according to traffic in the past 30 days
Our old sampling mechanism used SAMPLE 20000000 syntax. This was
wonderful since it allowed essentially dynamic sampling based on the
data being queried. However this ran into many issues relating to JOINs
and sample rate being different for different tables.
Instead, we now start to dynamically vary sample rates fractionally.
At query time we check the time window being queried and estimate how
many rows this query might reach. For large queries, we then dynamically
decide the sample rate.
For getting the traffic estimate for a site, we have a new SampingCache class which queries `ingest_counters`.
The query being cached is slightly expensive and can be sped up with a
ClickHouse projection.
* ce behavior
* Fix test
* Update test/plausible/stats/sampling_cache_test.exs
Co-authored-by: hq1 <hq@mtod.org>
* Update tests
* Update tests
---------
Co-authored-by: hq1 <hq@mtod.org>
* Reduce "choose plan" reliance on the old schema
* Fix team members usage computation on teams schema
* Switch CreateInvitation to reading from team schemas behind FF (WIP)
* Allow test-inviting one guest into multiple sites
* Convert another test case where team members count is wrong
cc @zoldar
* WIP: support site transfer notification e-mails
* Even more strict SiteTransfer fetching
* Make skipping permissions work
* Make CreateInvitation read from team schemas behind FF fully
* Fix passing options to `check_invitation_permissions`
* Fix allowance check for pageview usage for active or recently ended trial case
* Fix `check_invitation_permissions`
* Remove no longer relevant invite implementations for Teams
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* migration: add scroll_depth to events_v2
* (cherry-pick) ingest scroll depth
* replace convoluted test with more concise ones
* QueryParser: parse internal scroll_depth metric + validation
* turn QueryComparisonsTest into QueryInternalTest
* rename file
* (cherry pick) query scroll depth 15b14d3
...and move the tests into `internal_query_test.exs`
* review feedback
* Get rid of unnecessary separation between aggregate and group scroll depth
* Drop irrelevant other metrics in tests
* add test ensuring scroll depth unavailable in Stats API v1
* Put scroll depth on the dashboard
* Top Stats
* Main Graph
* Top Pages > Details
* feature flag for dashboard scroll depth access
* ignore credo warning
* enable scroll_depth flag in tests
* remove duplication
* write timestamps explicitly in a test
* revert moving tests around
* Add query_comparisons_test back
* Move scroll_depth tests into query_test
* Delete query_internal_test
* rename setup util (got updated on master)
* use pageleave_factory where applicable
* Use the correct generated query-api.d.ts
* npm format
* Switch upgrade to enterprise plan view to teams schema behind FF
* Switch change plan preview action to teams schema behind FF
* Switch remaining billing controller actions to teams schema behind FF
* Switch on teams schema in choose plan view behind FF
* Proxy via Read adapter where applicable for billing context
* Proxy remaining plan-related functions
* Switch enterprise_configured?/1 tests to use the adapter
* Format
* Update SiteLocker tests
* Actually use `has_active_subscription?/1` billing adapter
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Populate `current_team` to site's team and make site and subscription preloads consistent
* Accept only full `User` struct in `Users.get_for_user(!)`
* Make all uses of `Sites.get_for_user(!)` switch on team schema
* Remove redundant preloads for funnel/props settings
* Use adapter transitions in subscription settings
* Use team's schema subscription when listing invoices
* Fix typespec
* Turn owned site IDs into a specific query
* Add clauses for when FF is on but no team has been created
* Fix formatting
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Clamp dates on both directions
We could still get warnings about negative date ranges when the date
being queried is in the future.
This could happen if the users local time is in the future for some
reason or they manually edit the url.
* Update cond
* Update acquisition channel UDF to prioritize display over paid search
* Remove migration
Will run this manually together with a backfill, self-hosted will get this for free.
* Add test
---------
Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
* Expose site limit, usage, ensure_can_add_new_site via Adapter
* Print to stdout if TEST_READ_TEAM_SCHEMAS is enabled
* Add factory wrappers for remaining subscription types
* Ensure consistent ordering when fetching latest subscription
* Switch creating new site to read team schemas
* Dedup code based on read team schemas switching
* Switch to transitional factory where necessary
* Update yet another test requiring transitional factory
* Modify test utils to use teams test factories
* Implement alternative routes for updating and removing membership
* Implement teams read adapter for listing site members and invitees
* Use new teams read adapter for Settings > People view
* Add `invitation_id` column to `guest_invitations` schema
* Add `invitation_id` to `GuestInvitation` schema and populate it
* Sync guest invitation's invitation ID instead of team invitation
* Expose guest invitation's invitation ID in sites list
* Sync guest invitation invitation ID instead of team invitation in backfill
* Update team consistency check to account for guest invitation IDs
* Remove workaround for no invitation ID on guest invitation in `list_people`
* Test listing pending invitations
* Test listing memberships
* Format
* Test membership changes via new routes
* Remove old membership altering routes
* Clean up
* Revert "Modify test utils to use teams test factories"
This reverts commit 5eb8754782.
* Ensure test setup provisions teams for people listing
* See if we can avoid exposing user id
* Revert "See if we can avoid exposing user id"
This reverts commit 672429b9d1.
* Fix faulty member label in people list
* Fix sites listing for a case of pending invite with existing pin
---------
Co-authored-by: hq1 <hq@mtod.org>
* Solve noisy warnings about `a negative range was inferred for Date.range/2`
query.now is in utc but the date range was in the querys timezone
This is visible in the pattern on the graph:
* Add tests
* Implement user owning existing and pending sites check for teams
* Add predicate checking whether user has any existin or pending sites via teams
* Check need to upgrade for sites list via teams when FF is up
* Backfill teams for users on trial without a team
* Create team for users who register with trial started
* Replicate trial start logic on user create in team factories
* Make `ensure_can_take_ownership` in sites LV work via teams too
* Dispatch feature access check from `/sites` to Teams-schema reads
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Remove query.v2 flag
This was originally used for making sure queries use the right table as
migrating to APIv2. This is no longer needed
* Remove experimental_reduced_joins flag
* Expose a few data migration functions, add quiet option to do_run
* Create functions and test acquisition channel logic in clickhouse
Tests were lifted from test/plausible_web/controllers/api/external_controller_test.exs
* Clean up test code a bit
* Property test for acquisition channels
* Handle empty strings properly in reference implementation
* Fix spelling, minor issues
* Revert "Property test for acquisition channels"
This reverts commit 3fa0e0e4eb.
* Only test clickhouse functions
* Solve minor code issue
* update channels logic
* Revert "Only test clickhouse functions"
This reverts commit e12784031a.
* Add more tests
* Add small result assertion
* Make query options explicit in data migrations
* Move multi-query running logic to within datamigration lib
* Unbreak numeric ids migration
* Named params directly to Clickhouse
* Update reference test implementation
---------
Co-authored-by: Uku Taht <uku.taht@gmail.com>
* Run all tests with `read_team_schemas` flag up and down on CI
Replacing the existing setup for `experimental_reduced_joins` which
wasn't actively used and will be removed some time in the future.
This way we don't make CI run longer than necessary.
* Add one more test case for listing invitations on /sites LV
* Intentionally reintroduce regression to test CI
* Revert "Intentionally reintroduce regression to test CI"
This reverts commit cdb83c1494.
* Run tests for `experimental_reduced_joins` together with `read_team_schemas`
