Commit Graph

3077 Commits

Author SHA1 Message Date
Uku Taht
a1b1b84963
Remap sources migration (#4751) 2024-10-31 08:07:50 +00:00
Uku Taht
c3a06caa97
Channel and source data updates (#4599)
* Channel and source data updates

* Update source mappings for migration

* Fix codespell

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* Update lib/plausible/ingestion/acquisition.ex

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* Standardize access to utm params

* Add wikipedia as "known" source

* Move custom sources to json file

* Add some advertising utm_sources

* Move source mapping logic to refinspector file

* Rename PlausibleWeb.RefInspector -> Plausible.Ingestion.Source

* Move mapping overrides to custom_sources.json

* More robust detection of paid sources

* Add missing utm_sources to migration

* Codespell

* Add moduledoc for Plausible.Ingestion.Source

* Fix dialyzer

* Remove migration

* Add more custom favicons

* Re-generate referrer favicons file

* Add doctest for sources

---------

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
2024-10-30 13:41:51 +00:00
Karl-Aksel Puulmann
62fb285b71
Comparisons: Fix exports issue (#4756)
* Increase max_parser_depth setting

This is currently causing issues in production with exports w/ comparisons, but
would blow up in other cases with sufficient data.

The reason this is now blowing up is that we create a large AND/OR condition
for breakdown comparisons. ecto-ch nests each conditional in its separate parentheses
causing us to reach the default 1000 max_parser_depth limit

We can improve this in a couple of ways:
- Not add unneccessary parentheses in ecto-ch
- Adding a new operator to make the AND/OR nesting require less conditionals, using
arrays and tuples instead

* Don't query comparisons for exports

csv exports don't actually output any comparison data, so querying for
it just causes issues.
2024-10-30 13:40:26 +00:00
Adrian Gruntkowski
dd79ad3c22
Fix further team sync discrepancies (#4755)
* Sync removing and rejecting invitations and site transfers

* Sync invitation/transfer ID with each new invite

* Accept only subset of guest invitations on sync

* Pass test

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2024-10-30 11:20:33 +00:00
Karl-Aksel Puulmann
b5d48a7347
Breakdown comparisons: Tooltip UX, arrows (#4719)
* Revert "Revert "Remove no change icon, better alignment""

This reverts commit a69d0a14a7fe038fe110907571240ad99a2ba6a4.

* different arrows

* Render graph tooltips in react

* Use ChangeArrow in graph tooltips

* Only display a single date if viewing a single date

* Tests for breakdown responses with labels

* Simplify code a bit

* strokeClass

* refactor ChangeArrow

* Play with stroke widths

* jest tests

* Silence logging in jest tests

* not variable stroke width

* prettier

* Conversion rate widths

* Update a test

* Trim trailing dates for label when comparing running month

* Reformat file

* Update expect.toHaveTextContent for multiline

* Update date range labelling to be more consistent with the frontend

* reformat
2024-10-30 10:19:47 +00:00
Adrian Gruntkowski
dc7e019d58
Adjust GA4 import timeout and page size to reduce risk of 502s and timeouts (#4753) 2024-10-29 11:24:01 +00:00
RobertJoonas
ad8f49b38d
fix dashboard crash (#4754) 2024-10-29 10:59:58 +00:00
RobertJoonas
37116a2b12
Add tests for the pageleave script (#4744)
* move util function to util module

* move playwright config file to tracker dir root

* update Playwright + add gitignores

* Playwright: enable reuseExistingServer (non-CI env)

* store tracker src copies to avoid re-compilation in dev env

* test pageleave on simple navigation

* fix test util fn

* rename local_test npm script

* make test util able to expect multiple requests

* test pageleaves in SPAs

* test pageleave with manual URL

* test pageleave not sent in manual when pageview not triggered

* extend util fn to refute event requests

* test pageleaves not sent in excluded hash pages

* store hashes instead of file copies to detect /tracker/src changes

* drop async

* speed up test suite
2024-10-28 15:30:03 +00:00
RobertJoonas
1a6fd1dc15
Remove unused controller action (#4748)
* remove unused controller action

* remove unused alias
2024-10-28 14:33:53 +00:00
Adrian Gruntkowski
a3ab3c9061
Ensure upserted invitations and memberships are up to date on conflict (#4749) 2024-10-28 13:54:57 +00:00
Adrian Gruntkowski
27ac3b6b5b
Address team syncing discrepancies (#4739)
* Clean site transfers after 48 hours

* Sync accepting site transfers and invitations within transaction

* Add dry run mode to teams backfill and make it a default

* Extend invitation clean worker tests
2024-10-28 10:11:47 +00:00
RobertJoonas
b49fc82c2b
add size reporter GH action (#4745) 2024-10-28 09:59:16 +00:00
hq1
8bd32b68c9
Bring back premium feature notices in tiles (#4742)
* Lost tapes: bring back premium feature notices in tiles

* Ensure that e-mail CTA is only presented to business/enterprise customers

* Format

* Fix typespecs

* Add t() type to EnterprisePlan

* Reduce Plan.t() definition

Found no reason for it to be there.
2024-10-28 09:07:18 +00:00
Adrian Gruntkowski
0404522b5a
Sync team consistently on site creation (#4736) 2024-10-24 18:44:28 +00:00
Adrian Gruntkowski
2ec9e32557
Extend team consistency checks and improve syncing team and memberships (#4735)
* Extend team consistency checks

* Sync team against user right after creating the site

* Prune orphaned team guest memberships and invitations on site removal
2024-10-24 13:38:58 +00:00
hq1
c1a6501ec1
Delete teams along with users on DELETE /me (#4734)
* Delete teams along with users on DELETE /me

* Delete orphaned teams during backfill

* Format
2024-10-24 10:24:12 +00:00
Adrian Gruntkowski
5dbe1870e0
Implement teams consistency check and improve teams backfill (#4733)
* Implement team consistency check script

* Improve backfill teams sync check

* Fix team grace period sync
2024-10-24 10:03:22 +00:00
Karl-Aksel Puulmann
35851e9084
Comparisons: Only render comparisons in top stats when both query and data supports it (#4732)
* Comparisons: Only render comparisons in top stats when both query and data support it

This fixes a bug from navigation:
1. Comparison disabled, choose a non-realtime period (e.g. press T for "Last 30 days")
2. Enable comparison mode (press X)
3. Choose the realtime period (press R)
4. Choose a non-realtime period again (press T)

Kudos to robert for discovering the repro case.

* Fix conditional

Broken when fixing another bug d727ba5ed5 (diff-f2361637bc87773faced883d7560491e4612b7581f4748f03241821f4ff8f6feL166)
2024-10-24 08:27:55 +00:00
Karl-Aksel Puulmann
a59f12733d
Fix filter transformations (#4727)
Previously the code would use the incorrect operator and lead to nesting
2024-10-23 15:33:34 +00:00
hq1
1234cbf2c1
Enable teams sync (#4726) 2024-10-23 12:25:46 +00:00
RobertJoonas
0f93328b6f
Fix custom property value filter suggestions (#4720)
* new endpoint for custom prop value filter suggestions

* add test coverage for searching prop suggestions

* plug the new endpoint into FE

* update changelog

* format JS

* Update CHANGELOG.md

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

---------

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
2024-10-23 12:24:23 +00:00
Adrian Gruntkowski
6695f2d7a0
Fix backfilling teams for users without sites but with subscriptions (#4725) 2024-10-23 11:59:33 +00:00
Adrian Gruntkowski
7d6f10f0cb
Ensure team is present before use in sync logic (#4709)
* Ensure team is present before use in sync logic

* Ensure teams backfill works against partially assigned sites

* Associate site with team on creation

* Associate site with team on sync

* Reuse alias

* Add tests for invitation creation sync

* Move team assertions to a helper module

* Format

* Test team creation on site creation via Sites context module

* Add tests for teams sync on subscription changes

* Tag tests

* Test grace period start syncing up with teams

* Test grace period manual lock sycning w/ teams

* Test grace period end sycing up w/ teams

* Test clearing grace period sync with teams

* Update moduledoc

* Fix missing preloads and wrong result pattern matching in sync logic

* Test sync on accepting invites and site transfers

* Test sync on membership role update and member removal

* transfer async fix WIP

* Stop privisioning team in site factory

* Remove unused relationship from Site schema

* Ensure consistent parsing of `passthrough` from Paddle webhook

* Update team passthrough notification tests & logic

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2024-10-23 11:29:15 +00:00
Karl-Aksel Puulmann
7794ff1417
Fix bounce_rate metric causing 500s with comparisons (#4723) 2024-10-23 10:18:23 +00:00
Karl-Aksel Puulmann
296b3e53d9
Make time-on-page ungraphable (#4721)
This was marked as graphable in FE but not in BE previously. After removing the BE check,
this started throwing errors:

https://sentry.plausible.io/organizations/plausible/issues/626/?project=2&query=is%3Aunresolved+issue.priority%3A%5Bhigh%2C+medium%5D&referrer=issue-stream&statsPeriod=14d&stream_index=0
2024-10-23 09:56:47 +00:00
Karl-Aksel Puulmann
d648505a9b
Fix money values on graphs and conversions graph tooltips (#4717)
* Fix conversions graph tooltips

* Show revenue value instead of `undefined` in graph

This broke due to the unification of formatters. Not sure if this is the best fix
2024-10-22 12:43:49 +00:00
Karl-Aksel Puulmann
d727ba5ed5
Restore average/total revenue to top stats (#4716)
* Restore average/total revenue to top stats

These werent being graphed due to incorrect conditional

* prefer ternary
2024-10-22 11:47:29 +00:00
Adrian Gruntkowski
be83b7829d
Improve GA4 import error handling (#4714)
* Log warnings on HTTP GA4 import errors and add Sentry context on socket errors

* Treat socket and 5xx errors as recoverable (up for retry in ~1hr) in GA4 imports
2024-10-22 10:57:29 +00:00
Karl-Aksel Puulmann
472f4f181c
Comparisons pagination fixes (#4697)
* Add filter clauses for each main result filter

This handles the case where main query has a limit and results change. Doesnt handle metrics like percentage.

* Fix percentage calculations by ignoring breakdown-related filters in totals queries

* Refactor comparisons test suite

* Move comparisons logic to comparisons module

* New route for internal query tests

Only to be used in testing

* Support comparison queries with imports/breakdowns

* time dimension predicate extraction

* Clean up a test

* Update docstring

* Update route test

* fix a typo
2024-10-22 10:23:40 +00:00
Karl-Aksel Puulmann
aec0318c3b
Dashboard: show comparison for breakdowns (#4692)
* Comparisons: Move code to LegacyQueryBuilder

* WIP: Return comparison results to frontend

* refactor: remove useless param

* Different result format

* Pass object to metric.renderValue

* remove dead code

* Fixup response format

* Comparison in a tooltip

Not perfect at all, but a good start. Problems arise with money etc.

* Simple change arrow

* Extract metric entry to ts

* popper attempt WIP

* Slightly nicer content

* Solve warning

* Unified changeArrow in app

* Remove needless spanning

* Always set `graph_metric` in top stats.

FE already has business logic around whether a given metric is graphable

* Remove dead code

* Move Money module under dashboard utils, keep in build

* change <Metric /> definition to take in a `formatter` and store default formatters in another (typed) const

* Use standard system for formatting numbers

* Arrows only in table

* remove dead import

* Inline renderValue

* Render metric name in tooltip

* numberFormatter -> numberShortFormatter

* numberShortFormatter update

* Separate long/short formatters

* Use long vs short formatters

* Put column name into tooltip

* Slightly improved label handling for percentages, conversion rate

* Improved boundary handling in tooltip.js

* Iterate tooltips, no tooltip for - revenue

* Update top stats tests after graph_metric change

* Change revenue metrics stats API return structure

Conversion now happens earlier in query pipeline, we return float for comparison purposes

* useQueryContext in a component

* graph_metric for current visitors to fix realtime view

* No tooltips if fully - row

* renderValue as a proper function

* Simplify MetricEntry

* Use common const

* tooltip to typescript

* More explicit return structure

* metric-entry -> metric-value

* Restore some files

* ChangeArrow

* Restore MoreLink

* Fix typing in MoreLink

* <MetricValue />

* Tests for MetricValue and ChangeArrow

* details modal fixups

* re-add space between arrow and percentage

* Solve stylelint issues

* Update test

* Format

* Add flag `breakdown_comparisons_ui`

* reformat

* Remove no change icon, better alignment

* Revert "Remove no change icon, better alignment"

This reverts commit a8d62b6383.

* number-formatter.ts

* numberLongFormatter refactor

* useMemo dependency

* Handle nulls/undefined in top stats

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
2024-10-22 09:02:13 +00:00
ruslandoga
0e4e614d4c
Remove eex artifact (#4712)
fixes #4710
2024-10-22 07:20:39 +00:00
Adrian Gruntkowski
3023d322d4
Limit resource usage in teams backfill data migration (#4708) 2024-10-21 08:52:07 +00:00
hq1
4bb8b44cad
Fix docs URL slug (#4707) 2024-10-21 08:27:34 +00:00
Adrian Gruntkowski
076d7509e6
Fix with_teams (#4706) 2024-10-21 08:11:49 +00:00
Adrian Gruntkowski
17b12ddaeb
Implement basics of Teams (#4658)
* Extend schemas with new fields and relationships for teams

* Implement listing sites and sites with invitations with teams

* Implement creating invitations with teams

* Implement accepting invites with teams

* Add `Teams.SiteTransfer` schema

* Implement creating ownership transfers

* Implement accepting site transfer between teams

* Make results shapes from `Teams.Memberships` role functions more consistent

* Remove :team relation from ApiKey schema

* Pass and provision team on subscription creation

* Pass and provision team on enterprise plan creation

* Implement creating site for a team

* Keep team in sync during legacy ownership transfer and invitations

* Resolve conflict in `Teams.get_or_create` without transaction

* Abstract `GracePeriod` manipulation behind `Plausible.Users`

* Put `User.start_trial` behind `Plausible.Users` API

* Sync team fields on user update, if team exists

* Sync cleaning invitations, updating and removing members

* Transfer invitations too

* Implement backfill script

* Allow separate pg repo for backfill script

* Rollback purposefully at the end

* Update backfill script with parallel processing

* Use `IS DISTINCT FROM` when comparing nullable fields

* Handle no teams to backfill case gracefully when reporting

* Parallelize guest memberships backfill

* Remove transaction wrapping and query timeouts

* Make team sync check more granular and fix formatting

* Wrap single team backfill in a transatction for consistent restarts

* Make invitation and site transfer backfills preserve invitation ID

* Update migration repo config for easier dev access

* Backfill teams for users with subscriptions without sites

* Log timestamps

* Put teams sync behind a compile-time flag

* Keep timestamps in sync and fix subscriptions backfill

* Fix formatting

* Make credo happy

* Don't `use Plausible.Migration` to avoid dialyzer complaining

None of the tooling from there is used anywhere and `@repo` can
be defined directly in the migration script.

* Drop SSL workarounds in the backfill script

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2024-10-21 07:35:23 +00:00
Adrian Gruntkowski
1e38bd8771
Add fields and tables for teams (#4696)
* Add migration adding team related tables and fields

* Add `team_site_transfers` table to the teams migration

* Remove team_id FK from api_keys table

* Change new FK constraints on existing tables to `nilify_all` on delete

* Ensure unique indexes on invitation_id and transfer_id fields
2024-10-17 11:28:56 +00:00
ruslandoga
7a91ff1f5a
Use anchor tags in email templates (#4690)
* replace styled_link and unstyled_link components with plan anchor tag in email templates

* format

* change formatting for better whitespace

* use phx-no-format on bad anchor
2024-10-17 06:48:26 +00:00
hq1
7458d729ed
Allow me deletions with enterprise plans test (#4685)
* Add test for user deletion with enterprise plan attached

* Format
2024-10-16 09:01:38 +00:00
hq1
4188b79217
Fix form field name disabling 2FA (#4688) 2024-10-16 08:56:35 +00:00
hq1
67e35fa1d2
Migration: cascade delete enterprise plans on user removal (#4684) 2024-10-16 08:43:16 +00:00
ruslandoga
69949e9a16
Remove bracket from welcome email (#4686)
* remove bracket from welcome email

* remove another bracket from welcome email
2024-10-16 07:53:44 +00:00
ruslandoga
ff3ee1ca3b
Improve boolean environment variables reading (#4632)
* improve bool env var reading

* add yes/no and tests
2024-10-16 07:11:53 +00:00
ruslandoga
509dde7edf
rm Timex.before? and Timex.after? (#4552) 2024-10-16 07:11:10 +00:00
Adrian Gruntkowski
f9477c7ff4
Properly interpolate site name in site setup success email (#4681) 2024-10-15 13:45:01 +00:00
Karl-Aksel Puulmann
82b60365a1
fix(revenue): Fix selecting and graphing revenue goals with comparisons (#4680)
* Support graphing revenue goals in dashboard again

This has been broken since the introduction of display names

* Support graphing revenue metrics timeseries with comparisons

This was previously broken with an ArithmeticError and didn't have any coverage.

Related sentry report: https://sentry.plausible.io/organizations/plausible/issues/560/?project=2&query=is%3Aunresolved+issue.priority%3A%5Bhigh%2C+medium%5D&referrer=issue-stream&statsPeriod=14d&stream_index=1

* Simplify

* NPM test update

* Update goals test
2024-10-15 08:33:38 +00:00
Artur Pata
80fa21a865
Remove duplication related to nav menu (#4666) 2024-10-15 08:13:07 +00:00
hq1
023e2041fb
Set root domain (none) for ingests without hostname (#4678)
* Set root domain `(none)` for ingests without hostname

* No `nil` hostname should be allowed to enter ingestion

* Invoke `sanitize_hostname` only when applicable
2024-10-15 07:06:30 +00:00
hq1
0099164cdc
Add explicit order by to a flaky test (#4679)
otherwise it falls back to `visitors` descending,
which AFAIU is 2 for both metrics leading to random
order effectively.
2024-10-15 05:42:00 +00:00
hq1
70997abfc7
Reduce Phoenix.HTML usage (#4675)
* Turn .eex templates into .heex

* Add new compile-time presets to `PlausibleWeb`

* Fix remaining templates

* Update static components

* Update live components

* Update live views

* Update rest of the owl

* Update mjml template

* Format

* Format

* Revert MJML stuff, it's coupled with EEx

* yawn at test

* Get rid of `FormHelpers` module

* Ensure YOU label shows up first on IP rules list

* Update lib/plausible_web/templates/email/welcome_email.html.heex

Co-authored-by: Artur Pata <artur.pata@gmail.com>

* Fix create site email link

* Fix server error markup (and turn thanks into heex)

* Format

---------

Co-authored-by: Artur Pata <artur.pata@gmail.com>
2024-10-15 03:55:14 +00:00
Cenk Kücük
9de6d3d5fc
add monitoring for websocket endpoint (#4676) 2024-10-14 16:57:43 +01:00