Commit Graph

1064 Commits

Author SHA1 Message Date
Adrian Gruntkowski
17b12ddaeb
Implement basics of Teams (#4658)
* Extend schemas with new fields and relationships for teams

* Implement listing sites and sites with invitations with teams

* Implement creating invitations with teams

* Implement accepting invites with teams

* Add `Teams.SiteTransfer` schema

* Implement creating ownership transfers

* Implement accepting site transfer between teams

* Make results shapes from `Teams.Memberships` role functions more consistent

* Remove :team relation from ApiKey schema

* Pass and provision team on subscription creation

* Pass and provision team on enterprise plan creation

* Implement creating site for a team

* Keep team in sync during legacy ownership transfer and invitations

* Resolve conflict in `Teams.get_or_create` without transaction

* Abstract `GracePeriod` manipulation behind `Plausible.Users`

* Put `User.start_trial` behind `Plausible.Users` API

* Sync team fields on user update, if team exists

* Sync cleaning invitations, updating and removing members

* Transfer invitations too

* Implement backfill script

* Allow separate pg repo for backfill script

* Rollback purposefully at the end

* Update backfill script with parallel processing

* Use `IS DISTINCT FROM` when comparing nullable fields

* Handle no teams to backfill case gracefully when reporting

* Parallelize guest memberships backfill

* Remove transaction wrapping and query timeouts

* Make team sync check more granular and fix formatting

* Wrap single team backfill in a transatction for consistent restarts

* Make invitation and site transfer backfills preserve invitation ID

* Update migration repo config for easier dev access

* Backfill teams for users with subscriptions without sites

* Log timestamps

* Put teams sync behind a compile-time flag

* Keep timestamps in sync and fix subscriptions backfill

* Fix formatting

* Make credo happy

* Don't `use Plausible.Migration` to avoid dialyzer complaining

None of the tooling from there is used anywhere and `@repo` can
be defined directly in the migration script.

* Drop SSL workarounds in the backfill script

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2024-10-21 07:35:23 +00:00
hq1
7458d729ed
Allow me deletions with enterprise plans test (#4685)
* Add test for user deletion with enterprise plan attached

* Format
2024-10-16 09:01:38 +00:00
ruslandoga
ff3ee1ca3b
Improve boolean environment variables reading (#4632)
* improve bool env var reading

* add yes/no and tests
2024-10-16 07:11:53 +00:00
Karl-Aksel Puulmann
82b60365a1
fix(revenue): Fix selecting and graphing revenue goals with comparisons (#4680)
* Support graphing revenue goals in dashboard again

This has been broken since the introduction of display names

* Support graphing revenue metrics timeseries with comparisons

This was previously broken with an ArithmeticError and didn't have any coverage.

Related sentry report: https://sentry.plausible.io/organizations/plausible/issues/560/?project=2&query=is%3Aunresolved+issue.priority%3A%5Bhigh%2C+medium%5D&referrer=issue-stream&statsPeriod=14d&stream_index=1

* Simplify

* NPM test update

* Update goals test
2024-10-15 08:33:38 +00:00
hq1
023e2041fb
Set root domain (none) for ingests without hostname (#4678)
* Set root domain `(none)` for ingests without hostname

* No `nil` hostname should be allowed to enter ingestion

* Invoke `sanitize_hostname` only when applicable
2024-10-15 07:06:30 +00:00
hq1
0099164cdc
Add explicit order by to a flaky test (#4679)
otherwise it falls back to `visitors` descending,
which AFAIU is 2 for both metrics leading to random
order effectively.
2024-10-15 05:42:00 +00:00
hq1
70997abfc7
Reduce Phoenix.HTML usage (#4675)
* Turn .eex templates into .heex

* Add new compile-time presets to `PlausibleWeb`

* Fix remaining templates

* Update static components

* Update live components

* Update live views

* Update rest of the owl

* Update mjml template

* Format

* Format

* Revert MJML stuff, it's coupled with EEx

* yawn at test

* Get rid of `FormHelpers` module

* Ensure YOU label shows up first on IP rules list

* Update lib/plausible_web/templates/email/welcome_email.html.heex

Co-authored-by: Artur Pata <artur.pata@gmail.com>

* Fix create site email link

* Fix server error markup (and turn thanks into heex)

* Format

---------

Co-authored-by: Artur Pata <artur.pata@gmail.com>
2024-10-15 03:55:14 +00:00
Cenk Kücük
9de6d3d5fc
add monitoring for websocket endpoint (#4676) 2024-10-14 16:57:43 +01:00
Cenk Kücük
0c40042d8e
Update Clickhouse version in CI (#4617)
* Update Clickhouse version in CI

* update order

* Update clickhouse image

* Update Makefile
2024-10-11 00:13:59 +01:00
Karl-Aksel Puulmann
141eea88ff
APIv2: Revenue metrics (#4659)
* WIP: Start refactoring revenue metrics

* Hacks to make things work

* Remove old revenue code, remove revenue metrics if needed

* Update query_optimizer docs

* Minor fixes

* Add tests around average/total revenue when non-revenue goal filtering going on

* Optimize, calculate filters as expected (OR-ing clauses)

* Revenue: Handle cases where revenue metrics should not be returned or nil

* Expose revenue metrics in internal schema, add tests

* Docstring

* Remove TODO

* Typegen

* Solve warnings

* Remove nesting

* ce_test fix

* Tag tests as ee_only

* Fix: When filtering by revenue goal and no conversions, return 0.0 instead of nil

* More straight-forward preloading logic
2024-10-09 10:18:48 +00:00
Karl-Aksel Puulmann
5ad743c8d3
APIv2: Comparisons for breakdowns, timeseries, time_on_page (#4647)
* Refactor comparisons to a new options format

Prerequisite for APIv2 comparison work

* Experiment with default include deduplication

* WIP

Oops, breaks `include.total_rows`

* WIP

* Refactor breakdown.ex

* Pagination fix: dont paginate split subqueries

* Timeseries tests pass

* Aggregate tests use QueryExecutor

* Simplify QueryExecutor

* Handle legacy time-on-page metric in query_executor.ex

No behavioral changes

* Remove keep_requested_metrics

* Clean up imports

* Refactor aggregate.ex to be more straight-forward in output format building

* top stats: compute comparison via apiv2

* Minor cleanups

* WIP: Pipelines

* WIP: refactor for code cleanliness

* QueryExecutor to QueryRunner

* Make compilable

* Comparisons for timeseries works

Except for comparisons where comparison window is bigger than source query window

* Add special case for timeseries

* JSON schema tests for comparisons

* Test comparisons with the new API

* comparison date range parsing improvement

* Make comparisons api internal-only

* typegen

* credo

* Different schemata

* get_comparison_query

* Add comment on timeseries result format

* comparisons typegen

* Percent change for revenue metrics fix

* Use defstruct for query_runner over map

* Remove preloading atoms
2024-10-08 10:13:04 +00:00
hq1
2359cb920c
Account settings w sidebar (#4654)
* Outline /settings/v2 fundamentals

* Add setting tiles stubs

* Bootstrap name change

* Bootstrap theme change

* Bootstrap security settings

* Use table component for listing sessions

* Disable current e-mail field

* Implement Danger Zone

* Deal with compilation warnings

* Implement "Subscription" section

* Implement invoices list

* Fix invoices empty state & add API keys

* Fix headings in Subscription section

* Fix API keys mobile view

* Fix subscription boxes width

* Fix formatting

* Move tests for settings WIP

* Adjust remaining tests and router placement

Include docs links in tiles, where applicable.

* Fix remaining routes and remove dead code

* Fix route in a live view where no @conn is available

* Update mobile view settings picker

* Format

* Fix subscription section headings

* Fix account e-mail on dark mode

* Delete unused template

* Fix mobile setting section picker

* Optimize Login Management tile for mobile

* Update invoices section with docs link

* Update copy

* Remove trailing dots from (sub)titles

* Fix CSV export padding for "exporting" state

* Align subscription status to the right

* Fix failing test

* Fix subscription status alignment once again

* Improve subscription mobile view a little

* Fixup test compilation 🙈

* Add extra margin to subscription status box

* Make cancel button in 2FA modals expand in mobile view

* Stats API only

* Capitalize "Current session" indicator

* Show "Show More" invoices button only when there's >12

* tiny change

* Update changelog

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
Co-authored-by: Marko Saric <34340819+metmarkosaric@users.noreply.github.com>
2024-10-08 08:30:01 +00:00
Karl-Aksel Puulmann
4224f9d187
Fix revenue metrics for custom named goals (#4653)
Since https://github.com/plausible/analytics/pull/4415, revenue metrics
have not been correctly displayed on the dashboard if the goal name has
been renamed.
2024-10-07 08:18:46 +00:00
hq1
6940281d66
Settings password reset (#4649)
* Enable exceptions when revoking all user sessions

* Add `User` changeset for changing password

* Make button in `2fa_input` component optional

* Implement password change from User Settings

* Add tests

* Fix 2FA modal cancel button formatting

* Update changelog

* Don't pass redundant params to `render_settings` and clean up code a bit

* Render one error per field in password reset form

* Refactor inline form 2FA validation

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-10-03 06:39:32 +00:00
hq1
ec2a560016
Rework settings UI (#4626)
* Update generic components library

* Import generic components via `PlausibleWeb`

* Update Settings/Danger Zone

* Update new shared link template and convert to heex

* Update site settings layout

* Update site settings sidebar tab layout

* Update Settings/Email Reports

* Update Funnels

* Update ComboBox

* Extend/update form components

* Update Modal live component

* Update Settings/Goals

* Update Shields

* Update Settings/Props

* Update Settings/Import & Export

* Update flow progress

* Import Routes in settings

* Update Billing components

* Update Billing notice component

* Update feature toggle component

* Update 2fa component

* Update verification markup

* Update installation

* Update Settings/Integrations/Plugins

* Update domain change markup

* Update Settings/General

* Update Settings/Integrations

* Update Settings/People

* Update Settings/Integrations/GSC

* Update Settings/Visiblity

* ukuwip

* ukuwip

* Tables & paddings

* Imports exports

* Brighten disabled input text color for dark mode

* Tune down table border/divider in dark mode

* Format

* Fix goal list on mobile

* Fix IP Shields table on mobile

* Fix country shields list on mobile

* Fix country shield list on mobile

* Fix page shields list on mobile

* Fix import/export settings on mobile

* Fix combobox dropdown background in dark mode

* Fix filter bar search input on mobile

* Revert @ukutaht's changes to goal list

* Maybe maybe maybe

* Revert the current prod goal list + fix mobile issues

* Format

* Revert tests change

cc @ukutaht

* Fix markup expectation in a test

* Set autocomplete="off" again

* Bring back `text-sm` where previously removed

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
2024-10-02 09:05:21 +00:00
hq1
de04f222fd
Update con_cache (#4642) 2024-10-01 09:11:46 +00:00
RobertJoonas
5c72de0155
Experimental pageleave events (#4624)
* add experimental pageleave script variant

* also send pageleave events on SPA navigation

* disallow goals with 'pageleave' event name

* do not count pageleaves towards the event metric

* remove duplication in test file

* do not update sessions on pageleave events

* ignore pageleaves in the current time_on_page implementation

* make pageleave events not billable

* rename function

* Prevent multiple pageleaves being sent at the same time
2024-09-30 12:52:40 +00:00
Adrian Gruntkowski
714c036757
Implement listing user sessions in user settings (#4588)
* Implement listing user sessions in user settings

* Make copy adjustments (h/t @metmarkosaric)

* Make warning button text color more consistent across user settings

* Add tests for `UserAuth.revoke_user_session/2`

* Test and improve `Auth.UserSessions`

* Test and improve controller actions

* Update CHANGELOG.md
2024-09-30 11:03:08 +00:00
Artur Pata
cac4ad20c9
Allow configuring AuthorizeSiteAccess plug site param (#4597)
* Stop typescript command clearing previous server start output in dev

* Allow auth site access plug to specify that domain is found in request body at some key

* Fix init order

* Make domain retrieval part of the role pipeline

* Add tests

* Refactor how is_binary is applied in get_domain/2

* Make plug tests rely on dedicated test routes for more stability

* Consistently treat empty `allowed_roles` list as permitting all roles

* Fix async test param, add extra case for init

* Make `DocsQueryTest` async again

* Improve a bit and document plug configuration

* Make docs more legible when viewed from source directly

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-26 09:52:45 +00:00
Adrian Gruntkowski
2d7943aec8
Use :domain uniformly for all routes instead of :website (#4598)
* Replace :website with :domain in routes uniformly

* Remove support for :website parameter from `AuthorizeSiteAccess` plug
2024-09-26 08:20:35 +00:00
Adrian Gruntkowski
9d997a74d6
Implement user notes in CRM and HelpScout integration (#4611)
* Implement user notes in CRM and HelpScout integration

* Update HS tests

* Make slight markup and styling changes
2024-09-25 14:56:33 +00:00
Adrian Gruntkowski
6981972617
Make invitations cleanup worker rely on UTC time (#4610) 2024-09-25 12:44:55 +00:00
hq1
3251b5bb33
Check usage for users with active/cancelled/past due subscriptions (#4607)
* Check usage for users with active/cancelled/past due subscriptions

So that over limit e-mails and grace period is handled
regardless - they're still subscribers.

* Ensure current subscription is fetched, for an ongoing plan

* Add extra test

* Revert "Ensure current subscription is fetched, for an ongoing plan"

This reverts commit 9ddc7bdccb.

* Fixup test

* Reapply "Ensure current subscription is fetched, for an ongoing plan"

This reverts commit 7ab537954b.

* Do a subscription preload and assert ID match

* Use inner lateral join to fetch subscriptions
2024-09-25 11:27:12 +00:00
ruslandoga
09c74ee0b3
Mua: support 465 port connections (#4606)
* mua: support 465 port connections

* allow port=465 but SMTP_HOST_SSL_ENABLED=false
2024-09-24 07:57:15 +00:00
ruslandoga
71fa38709f
skip verification in ce (#4604) 2024-09-23 12:17:32 +00:00
Artur Pata
59c7ce2ef1
Fix auth issue with POST /api/docs/query (#4593) 2024-09-18 15:46:05 +00:00
Artur Pata
82a15884ad
Automatically generate Typescript types for v2 API query schema (#4574)
* Generate types from query schema

* Flip the query schema so private is static

* Ensure private schema stays private

* Refactor comment, json schema utils
2024-09-18 11:01:20 +00:00
Uku Taht
7a77ebf9bf
Add feature-flagged channels UI (#4585)
* Add feature-flagged channels UI

* Implement channels modal

* Channel -> Channels tab
2024-09-18 08:34:12 +00:00
Adrian Gruntkowski
6769ebdc24
Refactor AuthorizeSiteAccess (#4583)
* Refactor and fix `AuthorizeSiteAccess` plug shared link case handling

* Make plug module name more consistent

* Add moduledoc

* Add regression test for shared link auth case

* Adjust failing case test to account for different response

* Further restrict accepted input and configuration

* Extend plug-specific tests

* Tag EE-only test properly

* Remove one DB roundtrip from AuthorizeSiteAccess

* Improve naming slightly

* Filter shared link by site ID on query level already

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
2024-09-17 12:52:20 +00:00
RobertJoonas
45c0d538da
return city codes as integers in suggestions (#4589) 2024-09-17 12:37:13 +00:00
hq1
f645a9ba1d
Warm up ConCache before tests (#4587)
ref: https://github.com/sasa1977/con_cache/pull/79
2024-09-17 11:34:46 +00:00
Adrian Gruntkowski
f7bbbf8c33
Remove support for legacy user sessions (#4540)
* Remove support for legacy user sessions

* Implement revoking all sessions for a given user

* Revoke all user sessions on password reset

* Add tests for revoking all user sessions on password reset

* Reload page when dashboard API request fails with 404

* Revert "Reload page when dashboard API request fails with 404"

This reverts commit 77d1a1035658915f9afe538afc5fb9a3da0ec905.
2024-09-17 07:31:41 +00:00
Adrian Gruntkowski
ae0c5a173d
Drop event explicitly on session lock timeout (#4582)
* Drop event explicitly on session lock timeout

* Make session `user_id` more random in tests to avoid excess locking

* Improve testability to event ingestion

* Test lock timeout in ingestion
2024-09-16 10:36:53 +00:00
Karl-Aksel Puulmann
ef57502854
APIv2: Implement pagination and include.total_rows (#4575)
Offset-based pagination is used to make sure Looker integration
is able to work as efficiently as possible. To know how many
requests users need to do `include.total_rows` option was added.
2024-09-12 15:51:18 +03:00
Artur Pata
9fcb76d8e6
Allow sorting breakdown lists by some metrics (#4513)
* On the dashboard, allow Metrics (aka column configurations) to
  * specify width they need
  * specify if they are sortable
* Default sort order for breakdown endpoints is hardcoded on the dashboard (needed to show what column the report is sorted by)
* Unifies Google Keywords modal with other breakdowns
2024-09-12 14:49:43 +03:00
Karl-Aksel Puulmann
dd1021eff3
Limit docs query API, return 401 for API routes (#4570)
* Limit docs query API, return 401 for API routes

* Changelog entry

* Use 404 with changed error message

* Update changelog
2024-09-12 14:11:22 +03:00
Karl-Aksel Puulmann
bd11b4cf67
APIv2: Standard iso8601 timestamps, operate on UTC (#4563)
* query.date_range is now in UTC instead of user timezone

This simplifies things down the line and fixes several bugs where
query.date_range is cast to naivedatetime for ecto purposes

Many places still remain broken:
- comparison queries
- `to_date_range` calls

* Make default_for_date_range not care about time zones

* Make timezone parameter mandatory for to_date_range

* Simplify utc_date_range, update legacy query builder

* Fix more cases where query date range is needed

* query.date_range -> query.utc_time_range

* Query.date_range/1 function

* ensure_include_imported update

* Clean up send_email_report
2024-09-11 09:21:59 +03:00
Artur Pata
52b94842c0
Assert filters are tuples, simplify schema (#4541) 2024-09-10 18:01:42 +03:00
Adrian Gruntkowski
8ba5f7d32f
Safeguard queries relying on sign from faulty old session entries (#4555)
* Safeguard session queries relying on `sign` from faulty old session entries

* Comment updated metric

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* Apply safeguards to `bounce_rate` metric only

* Add note to bounce rate definition in SQL fragments as well

* Add test for graceful bounce rate handling in breakdown

* Make user_id more unique

* Add a note to the test

* Move regression test to APIv2 tests

---------

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
2024-09-10 13:22:49 +02:00
Adrian Gruntkowski
c105ebceaf
Fix test failing due to timestamp setup (#4562) 2024-09-10 12:30:49 +02:00
ruslandoga
c536af0df0
Don't attempt to create a database if it already exists (#4498)
* don't attempt to create a database if already created

* add tests

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-09 11:55:39 +02:00
ruslandoga
d17ac82058
Remove Timex.now (#4546)
* rm Timex.now

* fix test

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-09 11:40:15 +02:00
ruslandoga
8ee4827fea
Run migrations in order across repos (#4466)
* order migrations

* migrate in 'streaks'

* cleanup

* docs

* tests

* add comment

* changelog

* continue

* run migrations tests in ci

* add pending_streams cmd

* fix ci

* add docs

* fix test

* simplify test

* only test v2+

* fix test

* return async: true

* cleanup

* add 'no pending migration'

* drop migrate/0 and pending_migrations/0

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-09 11:27:11 +02:00
hq1
0031d1487d
Verification: Add super-admin report displaying diagnostics (#4554)
* Verification: Add super-admin report displaying diagnostics

* Revert LSP not catching up with formatter rules on time

* Remove redundant condition
2024-09-09 10:49:29 +02:00
Karl-Aksel Puulmann
604dde99fd
APIv2: Regex operations, consistent operators (#4488)
* Rename matches/does_not_match filters internally

These have never been exposed to the frontend/user directly, only via
APIv1 filtering syntax. As such we are free to rename these without
breaking things

* Rename function arguments for consistency, simplify

* Add support for `match`/`not_match` operators for query apiv2

These match the string against a regular expression, as defined in
https://github.com/google/re2/wiki/Syntax

* not_match -> match_not

* does_not_contain -> contains_not

Note that for backwards compatibility:
- Browser handles does_not_contain in URL
- Backend will handle does_not_contain in queries for a day where we will remove it for better autocompletion

* not_matches_wildcard -> matches_wildcard_not

* prettier

* match -> matches

* Fix and test fix for matches_wildcard against prop when prop is missing

* Custom properties support for matches/matches_not

* Restore contains_not

* Test contains and contains_not behavior for custom properties
2024-09-09 10:05:24 +03:00
ruslandoga
67d7c6522c
Change SMTP defaults (#4538)
* default smtp config updates

* tests

* improve SMTP_MIDDLEBOX_COMP_MODE decoding

* changelog

* format

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-06 10:31:23 +02:00
hq1
b9493f687c
Replace native tooltips with simplified, generic component (#4528)
* Replace native tooltips with simplified, generic component

* Make tooltip a button?

* Revert "Make tooltip a button?"

This reverts commit adacc560f9.
2024-09-05 12:18:16 +02:00
ruslandoga
654558dbf9
Remove Timex from Plausible.Timezones (#3899)
* use stdlib for timezones

* Remove unused `Timezones.to_utc_datetime/2`

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-05 11:48:07 +02:00
Uku Taht
d56d6998df
Acquisition channel (#4489)
* WIP

* Add acquisition channel

* Add detection for gclid and msclkid

* Add GA4 source categories file as external resource
2024-09-05 12:02:15 +03:00
Karl-Aksel Puulmann
8fa3a83129
APIv2: and/or/not support (#4480)
* First approximation of AND/OR/NOT support

Broken by this:
- Goal filtering
- Table deciding
- Imports

* TableDecider handle nesting

* Query.remove_top_level_filters

* Plausible.Stats.Imported.SQL.Expression

* Handle AND/OR/NOT with imported data, create Plausible.Stats.Imported.SQL.WhereBuilder

* Add parser validations for event:goal, event:hostname and event:props:x filters top level constraints

* Move module around

* Query.get_filter -> Filters.filtering_on_dimension? in some callsites

* Filters.get_toplevel_filter

* TableDecider.sessions_join_events?, remove old method

* Transforming filters in query_optimizer

* Query API tests for and/or/not

* Reorder parser steps

* Post-merge test fixups

* Solve merge issue

* Simplify filtering_on_dimension?

* Update transformer code

* dimensions_used_in_filters min_depth option, simplify parser validations

* rename_dimensions_used_in_filter

* fix rename_dimensions_used_in_filter

* Rename a test
2024-09-04 15:44:03 +03:00