Commit Graph

3170 Commits

Author SHA1 Message Date
Adrian Gruntkowski
729a32e610
Teams writes switch (#4883)
* Comment out legacy fields and relationships

* WIP

* WIP 2

* WIP 3

* wip

* Remove teams backfill and consistency check scripts

* WIP 3

* Fix CheckUsage tests

* Update billing/subscription tests

* WIP 4

* Make site transfer fail if some invitation already exists

* Fixup: do symmetric invitation/site transfer check

* Update UI bugs: make listing sites/inviting admins work like before

* Fix Sites test

* Fix external sites controller test

* Fix live sites tests

* Fix props availability lookup

* Fix site controller tests

* Fix billing controller tests

* WIP - accept invitation tests

* Another round of test fixes + invitations logic bugs

* users_test -> teams_test

* Update registration via invitation

Here, we still rely on "polymorphic" invitation structures,
hence the "unified by id" helper.

For now, it'll remain local unless we discover it's
needed in the broader `Teams.Invitations` context.

cc @zoldar

* Yet another round of test and bugfixes along the way

* Include team in site setup success e-mail

* Fix send_site_setup_emails worker

* Fixed almost all tests except CRM ones

* Update enterprise plan admin test

* Fix CRM + remaining tests

* Address credo warnings (modulo one FIXME)

* Remove last FIXME and rephrase the invitation test case description

* Set Team fields via User CRM transparently

* Map user reference in Enterprise Plan CRM via team owner

* Fix resource actions in user CRM

* Get rid of warning when opening create form in API keys CRM

* Stop emitting warnings when editing Enterprise Plans via CRM

* Tests: Bump await_clickhouse_count interval

* Remove XXX marker

* Fix register from invitation link in email sent for ownership transfer

* Simplify fetching all pending site ownership site IDs

* Remove commented out schema fields

* Remove unused functions

* Address flakiness in ingest counter tests

* Remove unused `Teams.Sites.create`

* Don't restart trial on team with subscription when creating site

* Account for cases of legacy teams with empty trial expiry date

* Revert "Address flakiness in ingest counter tests"

This reverts commit 60dc1e4115.

* Fix flaky ingest counters tests under load

* Attempt 2

* Pre-emptively hardcode site ids in sampling cache test

to avoid supplying the same IDs alongside with counters test,
that inserts through another repo (async).
what we're observing is, clickhouse not summing mergetree columns fast
enough, even though we wait quite a bit.

* Fix ingest counter tests by accounting for delayed summation

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2024-12-16 11:11:14 +00:00
RobertJoonas
d21d48558a
Average Scroll Depth Metric: handle missing data better (#4889)
* ingest missing pageleave as 255 for pageleave events

* return scroll depth as nil when no valid pageleave data in range

* also set empty comparison value as nil instead of 0

* add data migration
2024-12-10 12:29:13 +00:00
RobertJoonas
c847d16a44
Average Scroll Depth Metric: dashboard sorting (#4887)
* allow scroll_depth sort

* test order by scroll_depth in dashboard controller action too
2024-12-09 11:08:37 +00:00
Karl-Aksel Puulmann
5b7b5434c5
Remove LegacyDashboardFilterParser (#4882)
* Remove LegacyDashboardFilterParser usage in stats_controller

* Update tests to avoid legacy dashboard parsing

* Update top stats tests

Note a few behavioral test changes since dashboard doesn't do wildcards anymore

* Update test/plausible_web/controllers/stats_controller_test.exs

* Update sources_test

Removed test was dead functionality

* Update countries_test.exs

* Update test/plausible_web/controllers/api/stats_controller/browsers_test.exs

* Update test/plausible_web/controllers/api/stats_controller/custom_prop_breakdown_test.exs

* Update test/plausible_web/controllers/api/stats_controller/suggestions_test.exs

* Remove dead tests

* Update pages_test

* Update conversions_test.exs

* Update funnels_test.exs

* Update test/plausible_web/controllers/api/stats_controller/main_graph_test.exs

* Update test/plausible_web/controllers/api/stats_controller/suggestions_test.exs

* Update test/plausible/stats/query_test.exs

* Remove legacy parsing code

* Changelog

* Inline utils
2024-12-09 09:00:07 +00:00
Karl-Aksel Puulmann
0c8fed9570
Fix: Add missing SAMPLE BY to query (#4880)
* Fix: Add missing SAMPLE BY to query

* Update lib/plausible/stats/sql/query_builder.ex
2024-12-09 08:54:39 +00:00
ruslandoga
87b65aefd7
Add Plausible EE plugs in CE (#4694)
* add Plausible EE plugs in CE

* use bottom padding instead of spacer

* change wording, move it up in report template
2024-12-09 08:44:30 +00:00
Adrian Gruntkowski
76cf87427d
Split current_team into site_team and my_team (#4881)
* Change TODO into NOTE

* Split `current_team` into `site_team` and `my_team`
2024-12-05 11:14:38 +00:00
Karl-Aksel Puulmann
bec14ee77c
Improve report performance with high-cardinality import joins (#4848)
* Improve report performance in cases where site has a lot of unique pathnames

Ref: https://3.basecamp.com/5308029/buckets/39750953/card_tables/cards/8052057081

JOINs in ClickHouse are slow. In one degenerate case I found a user had
over 20 million unique paths in an import, which resulted in extremely slow
JOINs. This introduces a sort-of hacky solution to it by limiting the
amount of data analyzed.

Query timing without this change:
```
9 rows in set. Elapsed: 11.383 sec. Processed 49.16 million rows, 5.75 GB (4.32 million rows/s., 505.29 MB/s.)
Peak memory usage: 14.75 GiB.
```

After:
```
9 rows in set. Elapsed: 0.572 sec. Processed 49.18 million rows, 5.75 GB (86.03 million rows/s., 10.06 GB/s.)
Peak memory usage: 9.01 GiB.
```

* Splitting should no longer remove pagination. Handle special cases in special_metrics.ex

* select_merge_as in imports

This sets up selected_as aliases which will be used in a subsequent commit

* Add explicit ORDER BY to import

* Rewrite comment

* quoting

* merge conflict

* Split test

* Merge conflict fail fix
2024-12-05 10:05:57 +00:00
Adrian Gruntkowski
3afec60d98
Remove team adapters (#4877)
* wip

* wip

* 🍌

* WIP

* Draw the rest of the owl (well, almost)

* Remove obsolete unlimited trial logic

* Remove `allow_next_upgrade_override?` adapter

* Remove `Teams` adapter

* Remove /sites adapters

* Remove `Sites` adapter

* Remove `change_plan` adapter

* Fix up CE test

* Remove adapter for Billing.latest_enterprise_plan_with_price(s)

* Remove adapter for `Billing.has_active_subscription?`

* Remove adapter for `Billing.active_subscription_for`

* Remove remaining billing adapter

* Remove all_pending_transfers

* Remove `get_owner` adapter

* Remove `has_sites?` and `owns_sites?` adapters

* Remove `Ownership` adapter

* Remove `check_invitation_permissions` adapter

* Remove `check_team_member_limit` adapter

* Remove `ensure_transfer_valid` adapter

* Remove Invitations adapter

* Remove sole teams adapter

* Make dialyzer happy

* Consolidate `Billing.features_usage` definition

* Remove unused `Quota.Usage`

* Remove remains of `read_team_schema` FF and reduce number of CI passes

* Bang up the condition

* Include pending invitations when querying `has_sites?`

* Fix and improve conditional expression in `plan_box` component

* Update comments

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2024-12-05 09:02:09 +00:00
Adrian Gruntkowski
90e541be38
Send correct invitation ID in email for new users (#4873) 2024-12-03 12:31:04 +00:00
hq1
a47a9adfbc
Don't crash ingesting >1M bytes payloads (#4872)
* Don't crash ingesting >1M bytes payloads

* Format
2024-12-03 11:50:47 +00:00
Karl-Aksel Puulmann
133423f7e6
Limit workflows being executed for elixir (#4856)
We're not expecting a lot of incompatibilities with pg 15 so it makes sense to combine these with reading team schemas
2024-12-03 11:23:34 +00:00
Karl-Aksel Puulmann
a38eacfed5
APIv2: Case insensitive search (#4863)
* WIP: Optional modifiers to queries

* WIP: Modifiers v2

* Use preloaded_goals when determining whether imports can be included

This was previously broken with conversion_rate totals metrics since it removed event:goal
filters but did not update preloaded_goals

* Preload goals according to modifiers

* Make case_sensitive: false work for is/contains operators

* Make modals send { case_sensitive: false } to backend for search

* CHANGELOG.md

* Typegen

* Prettier

* Refactor: more DRY where_builder for case sensitivity

* Support case_sensitive modifier for is_not/contains_not

* Cleanup

* credo

* remove defaults

* negating a previously set filter
2024-12-03 10:32:16 +00:00
hq1
0134c4ed32
Return team member usage of 0 if no team present (#4871) 2024-12-03 08:42:49 +00:00
Adrian Gruntkowski
8b8040edc5
Provide site-related team for current team in goal settings LV (#4870)
* Provide site-related team for current team in goal settings LV

* Make billing notice components accept nil current_team gracefully
2024-12-02 15:30:59 +00:00
Adrian Gruntkowski
b86c2e715f
Switch reads to Teams schemas across the rest of the app (#4860)
* Pre-emptively introduce `site.team_owner` relation

* Drop null constraint on user_id from subscriptions and enterprise_plans

* Temporarily remove populating old schemas in Teams.Test

* Point to site.owner via new schema

* Switch more reads to teams schema WIP

* Fix AuhtorizeSiteAccess test

There's no need to translate `admin`<->`editor` here,
the redundancy is inlined wherever the plug is initialized.

* Fix regions test

* Fix main graph test

* Fix authorization test

* Try to rely on team for subscription/plans where applicable

* Test fixes

* Fix plans test

* Prep for CheckUsage changes

* Skip remaining CheckUsage tests for now

* Fix user deletion to account for team relations

* Fix HelpScout tests

* 💀 Modify ingestion to read team schemas

* Made all tests green except skipped ones

* Mute warnings about transferring site with no order

By making artificial site membership struct,
when reading data off team membership schema.

* Fix site removal test case

* Re-enable locked site tests, that don't have to rely on `SiteLocker`

* Format

* Revert "Mute warnings about transferring site with no order"

This reverts commit 0e45f8c9d9.

* Re-enable old models and fix remaining tests

* Use new factories in a long running minio test

* FIXME->TODO

* Fix remaining tests in legacy mode (no FF raised)

* oof

cc @zoldar

* Add missing definitions of editor role in FE code

* Remove no longer relevant comment about roles

* Fix JS formatting

* Always prioritize site transfers over memberships in sites list

* Fix misaligned "Reject" invitation button

* Fix site pinning when user is guest in multiple sites in team

* Fix subscription settings controller tests

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2024-12-02 12:20:16 +00:00
hq1
badc477048
Drop null constraint on user_id from subscriptions and enterprise_plans (#4868)
* Drop null constraint on user_id from subscriptions and enterprise_plans

* Format

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-12-02 11:14:16 +00:00
Artur Pata
a155365b63
Register segments schema (#4805) 2024-12-02 07:59:43 +00:00
ruslandoga
24888cff3c
Drop Logger backends (#4862)
* drop console logger backend

* ah, switch Sentry too
2024-11-28 12:18:35 +00:00
Karl-Aksel Puulmann
7f4c2b6fb7
Fix debug controller domain filtering (#4858) 2024-11-26 15:03:01 +00:00
Karl-Aksel Puulmann
2112feee88
CLICKHOUSE_DEFAULT_STORAGE_POLICY env (#4851)
In production we use `storage_policy = 'tiered'` by default but this is
not reflected in any migrations.

This change fixes that by introducing a new environment variable and
plumbing to be used in new (and old) migrations

Tested via setting env, doing `mix ecto.drop; mix ecto.create; mix ecto.migrate` and
checking resulting table schemas.
2024-11-26 12:54:17 +02:00
Adrian Gruntkowski
95471c0085
Switch AcceptInvitation ops to read from team schemas behind FF (#4847)
* Move `bulk_transfer_ownership_direct` under `AcceptInvitation`

* [WIP] Switch ownership transfer operations to read from team schemas behind FF

* Fix usage test regression

* Semantics - current user; ownership is not necessarily involved

* Perform remaining read via adapter; remove obsolete test

* Properly list site with pending site transfer while being guest on a team

* Account for pending site transfers in Settings > People list

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2024-11-26 08:47:12 +00:00
Marko Saric
8dad965c29
adding a note about being able to switch to a lower plan at any time (#4853)
* adding a note about being able to switch to a lower plan at any time

* Update dashboard_locked.html.heex

* mix format

---------

Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
2024-11-25 10:34:10 +00:00
Adrian Gruntkowski
405c33f7cd
Fix a regression in accepting site transfers with pending invitations (#4852) 2024-11-25 08:49:48 +00:00
hq1
333238f085
Add GTM installation option, when no GTM detected (#4850) 2024-11-25 07:24:28 +00:00
Karl-Aksel Puulmann
969f094e61
fractional_sample_rate: Vary sample rate according to traffic estimation (#4839)
* fractional_sample_rate: Vary sample rate according to traffic in the past 30 days

Our old sampling mechanism used SAMPLE 20000000 syntax. This was
wonderful since it allowed essentially dynamic sampling based on the
data being queried. However this ran into many issues relating to JOINs
and sample rate being different for different tables.

Instead, we now start to dynamically vary sample rates fractionally.

At query time we check the time window being queried and estimate how
many rows this query might reach. For large queries, we then dynamically
decide the sample rate.

For getting the traffic estimate for a site, we have a new SampingCache class which queries `ingest_counters`.

The query being cached is slightly expensive and can be sped up with a
ClickHouse projection.

* ce behavior

* Fix test

* Update test/plausible/stats/sampling_cache_test.exs

Co-authored-by: hq1 <hq@mtod.org>

* Update tests

* Update tests

---------

Co-authored-by: hq1 <hq@mtod.org>
2024-11-25 07:02:19 +00:00
RobertJoonas
2a7d02b6f0
Average Scroll Depth Metric: extracted tracker changes (#4826)
* (cherry-pick) implement scroll depth tracking under pageleave variant

* drop unnecessary vars

* remove unused require

* add scroll depth tests

* improve error messages in test util

* reevaluate currentDocumentHeight on page load

* account for dynamically loaded content when initializing documnent height

* remove all semicolons from tracker specs

* allow window and document globals in tracker eslint

* tweak global tracker dir eslint rules

* update comment

* reevaluate document height on scroll

* add test

* remove unneccessary timeout
2024-11-21 14:29:52 +00:00
hq1
4d9ea15e9e
Switch reads in invitation creation logic to teams behind FF + loosely related fixes 🍌 (#4845)
* Reduce "choose plan" reliance on the old schema

* Fix team members usage computation on teams schema

* Switch CreateInvitation to reading from team schemas behind FF (WIP)

* Allow test-inviting one guest into multiple sites

* Convert another test case where team members count is wrong

cc @zoldar

* WIP: support site transfer notification e-mails

* Even more strict SiteTransfer fetching

* Make skipping permissions work

* Make CreateInvitation read from team schemas behind FF fully

* Fix passing options to `check_invitation_permissions`

* Fix allowance check for pageview usage for active or recently ended trial case

* Fix `check_invitation_permissions`

* Remove no longer relevant invite implementations for Teams

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-11-21 11:12:51 +00:00
Marko Saric
5072613c4f
Change the copy when switching a plan (#4796)
* Update plan_box.ex

* Fix test assertions

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2024-11-21 05:28:28 +00:00
RobertJoonas
6822b29016
Average Scroll Depth Metric: put scroll depth on the dashboard under a feature flag (#4832)
* migration: add scroll_depth to events_v2

* (cherry-pick) ingest scroll depth

* replace convoluted test with more concise ones

* QueryParser: parse internal scroll_depth metric + validation

* turn QueryComparisonsTest into QueryInternalTest

* rename file

* (cherry pick) query scroll depth 15b14d3

...and move the tests into `internal_query_test.exs`

* review feedback

* Get rid of unnecessary separation between aggregate and group scroll depth
* Drop irrelevant other metrics in tests

* add test ensuring scroll depth unavailable in Stats API v1

* Put scroll depth on the dashboard

* Top Stats
* Main Graph
* Top Pages > Details

* feature flag for dashboard scroll depth access

* ignore credo warning

* enable scroll_depth flag in tests

* remove duplication

* write timestamps explicitly in a test

* revert moving tests around

* Add query_comparisons_test back
* Move scroll_depth tests into query_test
* Delete query_internal_test

* rename setup util (got updated on master)

* use pageleave_factory where applicable

* Use the correct generated query-api.d.ts

* npm format
2024-11-20 13:13:04 +00:00
Adrian Gruntkowski
a29eb3d3ca
Switch billing controller action to teams schema behind FF (#4843)
* Switch upgrade to enterprise plan view to teams schema behind FF

* Switch change plan preview action to teams schema behind FF

* Switch remaining billing controller actions to teams schema behind FF
2024-11-20 10:10:54 +00:00
Adrian Gruntkowski
380dc00d1a
Switch on teams schema in choose plan view behind FF (#4838)
* Switch on teams schema in choose plan view behind FF

* Proxy via Read adapter where applicable for billing context

* Proxy remaining plan-related functions

* Switch enterprise_configured?/1 tests to use the adapter

* Format

* Update SiteLocker tests

* Actually use `has_active_subscription?/1` billing adapter

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2024-11-20 08:35:18 +00:00
hq1
7b49dd33c7
Remove DebugReplayInfo (#4842)
We have now better tools to debug queries 😈
2024-11-20 08:20:35 +00:00
RobertJoonas
7f783920f3
Average Scroll Depth Metric: extracted ingestion changes (#4828)
* migration: add scroll_depth to events_v2

* (cherry-pick) ingest scroll depth

* replace convoluted test with more concise ones
2024-11-19 11:38:34 +00:00
RobertJoonas
e93c97de1e
migration: add scroll_depth to events_v2 (#4827) 2024-11-19 09:59:23 +00:00
Adrian Gruntkowski
4ff2a66548
Switch on team schema in site settings controller actions and LVs (#4834)
* Populate `current_team` to site's team and make site and subscription preloads consistent

* Accept only full `User` struct in `Users.get_for_user(!)`

* Make all uses of `Sites.get_for_user(!)` switch on team schema

* Remove redundant preloads for funnel/props settings

* Use adapter transitions in subscription settings

* Use team's schema subscription when listing invoices

* Fix typespec

* Turn owned site IDs into a specific query

* Add clauses for when FF is on but no team has been created

* Fix formatting

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2024-11-19 09:49:37 +00:00
Karl-Aksel Puulmann
9af498833e
Channels: backfill utm_medium based on click_param_id (#4833)
* Backfill utm_medium

Follow-up to https://github.com/plausible/analytics/pull/4817

* Update backfill
2024-11-19 08:12:39 +00:00
Uku Taht
73166774e0
Infer medium from click id if not present (#4817) 2024-11-18 11:19:11 +00:00
RobertJoonas
916c2bb4c5
Stop suggesting pageleave as a custom event goal name (#4829)
* stop suggestions pageleave as a custom event goal name

* add missing test
2024-11-18 10:16:33 +00:00
Adrian Gruntkowski
c2a95a142d
Make feature toggle work with teams schema and adjust affected tests (#4825) 2024-11-18 08:52:03 +00:00
Karl-Aksel Puulmann
07a3436fa2
Follow-up to Solve noisy warnings about a negative range was inferred for Date.range/2 (#4816)
* Clamp dates on both directions

We could still get warnings about negative date ranges when the date
being queried is in the future.

This could happen if the users local time is in the future for some
reason or they manually edit the url.

* Update cond
2024-11-18 08:00:18 +00:00
Artur Pata
d187e59049
Fix routes not opening that take dynamic route params when the param contains forward slash (#4824) 2024-11-14 22:20:06 +00:00
Uku Taht
b0933e1730
Import acquisiton channel from GA4 (#4814)
* Import acquisiton channel from GA4

* Remove unnecessary step

* Fix spelling

* Remove migration

* Show empty channel in imported data as (not set)

* Revert "Remove migration"

This reverts commit da0b9403e4.

* Fix channel suggestions with imported data

* Merge group field entries

* Add note about channel mappings

* Revert "Revert "Remove migration""

This reverts commit 7958a46c5c.
2024-11-14 18:22:14 +00:00
Uku Taht
738db2df98
Channels UI polish (#4820)
* Make top sources UI hierarchical

* Display report title as card title in top sources

* Manage useEffect deps properly
2024-11-14 17:13:05 +00:00
Uku Taht
0bbdbc9f42
Imported channel migration (#4815) 2024-11-14 17:12:18 +00:00
Uku Taht
daa42cbc9d
Update acquisition channel UDF to prioritize display over paid search (#4818)
* Update acquisition channel UDF to prioritize display over paid search

* Remove migration

Will run this manually together with a backfill, self-hosted will get this for free.

* Add test

---------

Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
2024-11-14 16:01:34 +00:00
RobertJoonas
b42ab94a14
Make BreakdownModal dimension column use up all remaining width (#4809) 2024-11-14 14:16:33 +00:00
Uku Taht
cf4ba664ed
Tiny source data update (#4821)
* Merge teams.microsoft.com -> Microsoft Teams

* Display favicon for Linkedin
2024-11-14 13:28:45 +00:00
hq1
0d6bec1bbe
Switch sites creation to read teams schemas (#4823)
* Expose site limit, usage, ensure_can_add_new_site via Adapter

* Print to stdout if TEST_READ_TEAM_SCHEMAS is enabled

* Add factory wrappers for remaining subscription types

* Ensure consistent ordering when fetching latest subscription

* Switch creating new site to read team schemas

* Dedup code based on read team schemas switching

* Switch to transitional factory where necessary

* Update yet another test requiring transitional factory
2024-11-14 11:03:10 +00:00
Adrian Gruntkowski
9b6961ce9b
Try to make ingest lock timeout tests less flaky (#4628) 2024-11-13 12:49:08 +00:00