Commit Graph

1953 Commits

Author SHA1 Message Date
Adrian Gruntkowski
6695f2d7a0
Fix backfilling teams for users without sites but with subscriptions (#4725) 2024-10-23 11:59:33 +00:00
Adrian Gruntkowski
7d6f10f0cb
Ensure team is present before use in sync logic (#4709)
* Ensure team is present before use in sync logic

* Ensure teams backfill works against partially assigned sites

* Associate site with team on creation

* Associate site with team on sync

* Reuse alias

* Add tests for invitation creation sync

* Move team assertions to a helper module

* Format

* Test team creation on site creation via Sites context module

* Add tests for teams sync on subscription changes

* Tag tests

* Test grace period start syncing up with teams

* Test grace period manual lock sycning w/ teams

* Test grace period end sycing up w/ teams

* Test clearing grace period sync with teams

* Update moduledoc

* Fix missing preloads and wrong result pattern matching in sync logic

* Test sync on accepting invites and site transfers

* Test sync on membership role update and member removal

* transfer async fix WIP

* Stop privisioning team in site factory

* Remove unused relationship from Site schema

* Ensure consistent parsing of `passthrough` from Paddle webhook

* Update team passthrough notification tests & logic

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2024-10-23 11:29:15 +00:00
Karl-Aksel Puulmann
7794ff1417
Fix bounce_rate metric causing 500s with comparisons (#4723) 2024-10-23 10:18:23 +00:00
Adrian Gruntkowski
be83b7829d
Improve GA4 import error handling (#4714)
* Log warnings on HTTP GA4 import errors and add Sentry context on socket errors

* Treat socket and 5xx errors as recoverable (up for retry in ~1hr) in GA4 imports
2024-10-22 10:57:29 +00:00
Karl-Aksel Puulmann
472f4f181c
Comparisons pagination fixes (#4697)
* Add filter clauses for each main result filter

This handles the case where main query has a limit and results change. Doesnt handle metrics like percentage.

* Fix percentage calculations by ignoring breakdown-related filters in totals queries

* Refactor comparisons test suite

* Move comparisons logic to comparisons module

* New route for internal query tests

Only to be used in testing

* Support comparison queries with imports/breakdowns

* time dimension predicate extraction

* Clean up a test

* Update docstring

* Update route test

* fix a typo
2024-10-22 10:23:40 +00:00
Karl-Aksel Puulmann
aec0318c3b
Dashboard: show comparison for breakdowns (#4692)
* Comparisons: Move code to LegacyQueryBuilder

* WIP: Return comparison results to frontend

* refactor: remove useless param

* Different result format

* Pass object to metric.renderValue

* remove dead code

* Fixup response format

* Comparison in a tooltip

Not perfect at all, but a good start. Problems arise with money etc.

* Simple change arrow

* Extract metric entry to ts

* popper attempt WIP

* Slightly nicer content

* Solve warning

* Unified changeArrow in app

* Remove needless spanning

* Always set `graph_metric` in top stats.

FE already has business logic around whether a given metric is graphable

* Remove dead code

* Move Money module under dashboard utils, keep in build

* change <Metric /> definition to take in a `formatter` and store default formatters in another (typed) const

* Use standard system for formatting numbers

* Arrows only in table

* remove dead import

* Inline renderValue

* Render metric name in tooltip

* numberFormatter -> numberShortFormatter

* numberShortFormatter update

* Separate long/short formatters

* Use long vs short formatters

* Put column name into tooltip

* Slightly improved label handling for percentages, conversion rate

* Improved boundary handling in tooltip.js

* Iterate tooltips, no tooltip for - revenue

* Update top stats tests after graph_metric change

* Change revenue metrics stats API return structure

Conversion now happens earlier in query pipeline, we return float for comparison purposes

* useQueryContext in a component

* graph_metric for current visitors to fix realtime view

* No tooltips if fully - row

* renderValue as a proper function

* Simplify MetricEntry

* Use common const

* tooltip to typescript

* More explicit return structure

* metric-entry -> metric-value

* Restore some files

* ChangeArrow

* Restore MoreLink

* Fix typing in MoreLink

* <MetricValue />

* Tests for MetricValue and ChangeArrow

* details modal fixups

* re-add space between arrow and percentage

* Solve stylelint issues

* Update test

* Format

* Add flag `breakdown_comparisons_ui`

* reformat

* Remove no change icon, better alignment

* Revert "Remove no change icon, better alignment"

This reverts commit a8d62b6383.

* number-formatter.ts

* numberLongFormatter refactor

* useMemo dependency

* Handle nulls/undefined in top stats

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
2024-10-22 09:02:13 +00:00
ruslandoga
0e4e614d4c
Remove eex artifact (#4712)
fixes #4710
2024-10-22 07:20:39 +00:00
Adrian Gruntkowski
3023d322d4
Limit resource usage in teams backfill data migration (#4708) 2024-10-21 08:52:07 +00:00
hq1
4bb8b44cad
Fix docs URL slug (#4707) 2024-10-21 08:27:34 +00:00
Adrian Gruntkowski
076d7509e6
Fix with_teams (#4706) 2024-10-21 08:11:49 +00:00
Adrian Gruntkowski
17b12ddaeb
Implement basics of Teams (#4658)
* Extend schemas with new fields and relationships for teams

* Implement listing sites and sites with invitations with teams

* Implement creating invitations with teams

* Implement accepting invites with teams

* Add `Teams.SiteTransfer` schema

* Implement creating ownership transfers

* Implement accepting site transfer between teams

* Make results shapes from `Teams.Memberships` role functions more consistent

* Remove :team relation from ApiKey schema

* Pass and provision team on subscription creation

* Pass and provision team on enterprise plan creation

* Implement creating site for a team

* Keep team in sync during legacy ownership transfer and invitations

* Resolve conflict in `Teams.get_or_create` without transaction

* Abstract `GracePeriod` manipulation behind `Plausible.Users`

* Put `User.start_trial` behind `Plausible.Users` API

* Sync team fields on user update, if team exists

* Sync cleaning invitations, updating and removing members

* Transfer invitations too

* Implement backfill script

* Allow separate pg repo for backfill script

* Rollback purposefully at the end

* Update backfill script with parallel processing

* Use `IS DISTINCT FROM` when comparing nullable fields

* Handle no teams to backfill case gracefully when reporting

* Parallelize guest memberships backfill

* Remove transaction wrapping and query timeouts

* Make team sync check more granular and fix formatting

* Wrap single team backfill in a transatction for consistent restarts

* Make invitation and site transfer backfills preserve invitation ID

* Update migration repo config for easier dev access

* Backfill teams for users with subscriptions without sites

* Log timestamps

* Put teams sync behind a compile-time flag

* Keep timestamps in sync and fix subscriptions backfill

* Fix formatting

* Make credo happy

* Don't `use Plausible.Migration` to avoid dialyzer complaining

None of the tooling from there is used anywhere and `@repo` can
be defined directly in the migration script.

* Drop SSL workarounds in the backfill script

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2024-10-21 07:35:23 +00:00
ruslandoga
7a91ff1f5a
Use anchor tags in email templates (#4690)
* replace styled_link and unstyled_link components with plan anchor tag in email templates

* format

* change formatting for better whitespace

* use phx-no-format on bad anchor
2024-10-17 06:48:26 +00:00
hq1
4188b79217
Fix form field name disabling 2FA (#4688) 2024-10-16 08:56:35 +00:00
ruslandoga
69949e9a16
Remove bracket from welcome email (#4686)
* remove bracket from welcome email

* remove another bracket from welcome email
2024-10-16 07:53:44 +00:00
ruslandoga
ff3ee1ca3b
Improve boolean environment variables reading (#4632)
* improve bool env var reading

* add yes/no and tests
2024-10-16 07:11:53 +00:00
ruslandoga
509dde7edf
rm Timex.before? and Timex.after? (#4552) 2024-10-16 07:11:10 +00:00
Adrian Gruntkowski
f9477c7ff4
Properly interpolate site name in site setup success email (#4681) 2024-10-15 13:45:01 +00:00
Karl-Aksel Puulmann
82b60365a1
fix(revenue): Fix selecting and graphing revenue goals with comparisons (#4680)
* Support graphing revenue goals in dashboard again

This has been broken since the introduction of display names

* Support graphing revenue metrics timeseries with comparisons

This was previously broken with an ArithmeticError and didn't have any coverage.

Related sentry report: https://sentry.plausible.io/organizations/plausible/issues/560/?project=2&query=is%3Aunresolved+issue.priority%3A%5Bhigh%2C+medium%5D&referrer=issue-stream&statsPeriod=14d&stream_index=1

* Simplify

* NPM test update

* Update goals test
2024-10-15 08:33:38 +00:00
hq1
023e2041fb
Set root domain (none) for ingests without hostname (#4678)
* Set root domain `(none)` for ingests without hostname

* No `nil` hostname should be allowed to enter ingestion

* Invoke `sanitize_hostname` only when applicable
2024-10-15 07:06:30 +00:00
hq1
70997abfc7
Reduce Phoenix.HTML usage (#4675)
* Turn .eex templates into .heex

* Add new compile-time presets to `PlausibleWeb`

* Fix remaining templates

* Update static components

* Update live components

* Update live views

* Update rest of the owl

* Update mjml template

* Format

* Format

* Revert MJML stuff, it's coupled with EEx

* yawn at test

* Get rid of `FormHelpers` module

* Ensure YOU label shows up first on IP rules list

* Update lib/plausible_web/templates/email/welcome_email.html.heex

Co-authored-by: Artur Pata <artur.pata@gmail.com>

* Fix create site email link

* Fix server error markup (and turn thanks into heex)

* Format

---------

Co-authored-by: Artur Pata <artur.pata@gmail.com>
2024-10-15 03:55:14 +00:00
Karl-Aksel Puulmann
141eea88ff
APIv2: Revenue metrics (#4659)
* WIP: Start refactoring revenue metrics

* Hacks to make things work

* Remove old revenue code, remove revenue metrics if needed

* Update query_optimizer docs

* Minor fixes

* Add tests around average/total revenue when non-revenue goal filtering going on

* Optimize, calculate filters as expected (OR-ing clauses)

* Revenue: Handle cases where revenue metrics should not be returned or nil

* Expose revenue metrics in internal schema, add tests

* Docstring

* Remove TODO

* Typegen

* Solve warnings

* Remove nesting

* ce_test fix

* Tag tests as ee_only

* Fix: When filtering by revenue goal and no conversions, return 0.0 instead of nil

* More straight-forward preloading logic
2024-10-09 10:18:48 +00:00
ruslandoga
f28a1b8086
Log Oban errors (and still report them to Sentry) (#4657)
* log oban errors in CE

* update changelog

* not just ce

* add comment
2024-10-08 10:26:35 +00:00
Karl-Aksel Puulmann
5ad743c8d3
APIv2: Comparisons for breakdowns, timeseries, time_on_page (#4647)
* Refactor comparisons to a new options format

Prerequisite for APIv2 comparison work

* Experiment with default include deduplication

* WIP

Oops, breaks `include.total_rows`

* WIP

* Refactor breakdown.ex

* Pagination fix: dont paginate split subqueries

* Timeseries tests pass

* Aggregate tests use QueryExecutor

* Simplify QueryExecutor

* Handle legacy time-on-page metric in query_executor.ex

No behavioral changes

* Remove keep_requested_metrics

* Clean up imports

* Refactor aggregate.ex to be more straight-forward in output format building

* top stats: compute comparison via apiv2

* Minor cleanups

* WIP: Pipelines

* WIP: refactor for code cleanliness

* QueryExecutor to QueryRunner

* Make compilable

* Comparisons for timeseries works

Except for comparisons where comparison window is bigger than source query window

* Add special case for timeseries

* JSON schema tests for comparisons

* Test comparisons with the new API

* comparison date range parsing improvement

* Make comparisons api internal-only

* typegen

* credo

* Different schemata

* get_comparison_query

* Add comment on timeseries result format

* comparisons typegen

* Percent change for revenue metrics fix

* Use defstruct for query_runner over map

* Remove preloading atoms
2024-10-08 10:13:04 +00:00
Marko Saric
e7332d95b2
Change the "outgrown" copy (#4652)
* Update notice.ex

* Format

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2024-10-08 09:27:50 +00:00
RobertJoonas
799edec81c
Trigger pageviews on pages loaded from bfcache (#4656)
* trigger pageview on pageshow with event.persisted

* update _tracking.html to do the same
2024-10-08 09:11:03 +00:00
hq1
2359cb920c
Account settings w sidebar (#4654)
* Outline /settings/v2 fundamentals

* Add setting tiles stubs

* Bootstrap name change

* Bootstrap theme change

* Bootstrap security settings

* Use table component for listing sessions

* Disable current e-mail field

* Implement Danger Zone

* Deal with compilation warnings

* Implement "Subscription" section

* Implement invoices list

* Fix invoices empty state & add API keys

* Fix headings in Subscription section

* Fix API keys mobile view

* Fix subscription boxes width

* Fix formatting

* Move tests for settings WIP

* Adjust remaining tests and router placement

Include docs links in tiles, where applicable.

* Fix remaining routes and remove dead code

* Fix route in a live view where no @conn is available

* Update mobile view settings picker

* Format

* Fix subscription section headings

* Fix account e-mail on dark mode

* Delete unused template

* Fix mobile setting section picker

* Optimize Login Management tile for mobile

* Update invoices section with docs link

* Update copy

* Remove trailing dots from (sub)titles

* Fix CSV export padding for "exporting" state

* Align subscription status to the right

* Fix failing test

* Fix subscription status alignment once again

* Improve subscription mobile view a little

* Fixup test compilation 🙈

* Add extra margin to subscription status box

* Make cancel button in 2FA modals expand in mobile view

* Stats API only

* Capitalize "Current session" indicator

* Show "Show More" invoices button only when there's >12

* tiny change

* Update changelog

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
Co-authored-by: Marko Saric <34340819+metmarkosaric@users.noreply.github.com>
2024-10-08 08:30:01 +00:00
hq1
6940281d66
Settings password reset (#4649)
* Enable exceptions when revoking all user sessions

* Add `User` changeset for changing password

* Make button in `2fa_input` component optional

* Implement password change from User Settings

* Add tests

* Fix 2FA modal cancel button formatting

* Update changelog

* Don't pass redundant params to `render_settings` and clean up code a bit

* Render one error per field in password reset form

* Refactor inline form 2FA validation

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-10-03 06:39:32 +00:00
Adrian Gruntkowski
35c8010078
Improve CheckUsage query perf (#4650) 2024-10-02 15:02:35 +00:00
hq1
ec2a560016
Rework settings UI (#4626)
* Update generic components library

* Import generic components via `PlausibleWeb`

* Update Settings/Danger Zone

* Update new shared link template and convert to heex

* Update site settings layout

* Update site settings sidebar tab layout

* Update Settings/Email Reports

* Update Funnels

* Update ComboBox

* Extend/update form components

* Update Modal live component

* Update Settings/Goals

* Update Shields

* Update Settings/Props

* Update Settings/Import & Export

* Update flow progress

* Import Routes in settings

* Update Billing components

* Update Billing notice component

* Update feature toggle component

* Update 2fa component

* Update verification markup

* Update installation

* Update Settings/Integrations/Plugins

* Update domain change markup

* Update Settings/General

* Update Settings/Integrations

* Update Settings/People

* Update Settings/Integrations/GSC

* Update Settings/Visiblity

* ukuwip

* ukuwip

* Tables & paddings

* Imports exports

* Brighten disabled input text color for dark mode

* Tune down table border/divider in dark mode

* Format

* Fix goal list on mobile

* Fix IP Shields table on mobile

* Fix country shields list on mobile

* Fix country shield list on mobile

* Fix page shields list on mobile

* Fix import/export settings on mobile

* Fix combobox dropdown background in dark mode

* Fix filter bar search input on mobile

* Revert @ukutaht's changes to goal list

* Maybe maybe maybe

* Revert the current prod goal list + fix mobile issues

* Format

* Revert tests change

cc @ukutaht

* Fix markup expectation in a test

* Set autocomplete="off" again

* Bring back `text-sm` where previously removed

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
2024-10-02 09:05:21 +00:00
ruslandoga
52437e8df0
Handle cross-device file move (#4640)
* handle cross-device file move

* don't change tests

* rm all the time

* no need to rm if rename succeeds

* don't raise in after
2024-09-30 16:16:36 +00:00
RobertJoonas
5c72de0155
Experimental pageleave events (#4624)
* add experimental pageleave script variant

* also send pageleave events on SPA navigation

* disallow goals with 'pageleave' event name

* do not count pageleaves towards the event metric

* remove duplication in test file

* do not update sessions on pageleave events

* ignore pageleaves in the current time_on_page implementation

* make pageleave events not billable

* rename function

* Prevent multiple pageleaves being sent at the same time
2024-09-30 12:52:40 +00:00
Adrian Gruntkowski
714c036757
Implement listing user sessions in user settings (#4588)
* Implement listing user sessions in user settings

* Make copy adjustments (h/t @metmarkosaric)

* Make warning button text color more consistent across user settings

* Add tests for `UserAuth.revoke_user_session/2`

* Test and improve `Auth.UserSessions`

* Test and improve controller actions

* Update CHANGELOG.md
2024-09-30 11:03:08 +00:00
ruslandoga
88144ecdf6
start totp vault early (#4631) 2024-09-27 08:15:53 +00:00
Artur Pata
cac4ad20c9
Allow configuring AuthorizeSiteAccess plug site param (#4597)
* Stop typescript command clearing previous server start output in dev

* Allow auth site access plug to specify that domain is found in request body at some key

* Fix init order

* Make domain retrieval part of the role pipeline

* Add tests

* Refactor how is_binary is applied in get_domain/2

* Make plug tests rely on dedicated test routes for more stability

* Consistently treat empty `allowed_roles` list as permitting all roles

* Fix async test param, add extra case for init

* Make `DocsQueryTest` async again

* Improve a bit and document plug configuration

* Make docs more legible when viewed from source directly

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-26 09:52:45 +00:00
Adrian Gruntkowski
2d7943aec8
Use :domain uniformly for all routes instead of :website (#4598)
* Replace :website with :domain in routes uniformly

* Remove support for :website parameter from `AuthorizeSiteAccess` plug
2024-09-26 08:20:35 +00:00
ruslandoga
356f50e9c8
Change cookie key in CE (#4621)
* use default sessions options in ce

* there is no RuntimeSessionAdapter anymore
2024-09-26 07:43:47 +00:00
Adrian Gruntkowski
9d997a74d6
Implement user notes in CRM and HelpScout integration (#4611)
* Implement user notes in CRM and HelpScout integration

* Update HS tests

* Make slight markup and styling changes
2024-09-25 14:56:33 +00:00
Adrian Gruntkowski
6981972617
Make invitations cleanup worker rely on UTC time (#4610) 2024-09-25 12:44:55 +00:00
hq1
3251b5bb33
Check usage for users with active/cancelled/past due subscriptions (#4607)
* Check usage for users with active/cancelled/past due subscriptions

So that over limit e-mails and grace period is handled
regardless - they're still subscribers.

* Ensure current subscription is fetched, for an ongoing plan

* Add extra test

* Revert "Ensure current subscription is fetched, for an ongoing plan"

This reverts commit 9ddc7bdccb.

* Fixup test

* Reapply "Ensure current subscription is fetched, for an ongoing plan"

This reverts commit 7ab537954b.

* Do a subscription preload and assert ID match

* Use inner lateral join to fetch subscriptions
2024-09-25 11:27:12 +00:00
ruslandoga
0094e8a20f
rm unnecessary reject (#4615) 2024-09-24 18:50:53 +00:00
ruslandoga
71fa38709f
skip verification in ce (#4604) 2024-09-23 12:17:32 +00:00
ruslandoga
1b912240e3
fix link (#4605) 2024-09-23 09:39:06 +00:00
ruslandoga
5766478887
don't render empty list element in ce (#4596) 2024-09-19 07:11:02 +00:00
Cenk Kücük
234d63e549
enrich with information about app node (#4592) 2024-09-18 17:08:24 +00:00
Artur Pata
59c7ce2ef1
Fix auth issue with POST /api/docs/query (#4593) 2024-09-18 15:46:05 +00:00
Artur Pata
82a15884ad
Automatically generate Typescript types for v2 API query schema (#4574)
* Generate types from query schema

* Flip the query schema so private is static

* Ensure private schema stays private

* Refactor comment, json schema utils
2024-09-18 11:01:20 +00:00
Uku Taht
7a77ebf9bf
Add feature-flagged channels UI (#4585)
* Add feature-flagged channels UI

* Implement channels modal

* Channel -> Channels tab
2024-09-18 08:34:12 +00:00
Adrian Gruntkowski
6769ebdc24
Refactor AuthorizeSiteAccess (#4583)
* Refactor and fix `AuthorizeSiteAccess` plug shared link case handling

* Make plug module name more consistent

* Add moduledoc

* Add regression test for shared link auth case

* Adjust failing case test to account for different response

* Further restrict accepted input and configuration

* Extend plug-specific tests

* Tag EE-only test properly

* Remove one DB roundtrip from AuthorizeSiteAccess

* Improve naming slightly

* Filter shared link by site ID on query level already

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
2024-09-17 12:52:20 +00:00
RobertJoonas
45c0d538da
return city codes as integers in suggestions (#4589) 2024-09-17 12:37:13 +00:00
Adrian Gruntkowski
f7bbbf8c33
Remove support for legacy user sessions (#4540)
* Remove support for legacy user sessions

* Implement revoking all sessions for a given user

* Revoke all user sessions on password reset

* Add tests for revoking all user sessions on password reset

* Reload page when dashboard API request fails with 404

* Revert "Reload page when dashboard API request fails with 404"

This reverts commit 77d1a1035658915f9afe538afc5fb9a3da0ec905.
2024-09-17 07:31:41 +00:00
Marko Saric
68e03c2112
Update to the copy (#4558)
* Update to the copy

Update to the copy to include a note about billable pageviews. hopefully it doesn't make the copy too long...

* Don't show tooltip on small/mobile screens

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-16 13:50:36 +00:00
Adrian Gruntkowski
ae0c5a173d
Drop event explicitly on session lock timeout (#4582)
* Drop event explicitly on session lock timeout

* Make session `user_id` more random in tests to avoid excess locking

* Improve testability to event ingestion

* Test lock timeout in ingestion
2024-09-16 10:36:53 +00:00
Karl-Aksel Puulmann
748e0d902f
APIv2: Make imports_skip_reason flag not appear when imports not requested (#4565)
* APIv2: Make imports_skip_reason flag not appear when imports not requested

* Update lib/plausible/stats/query_result.ex

Co-authored-by: hq1 <hq@mtod.org>

---------

Co-authored-by: hq1 <hq@mtod.org>
2024-09-16 06:13:32 +00:00
hq1
f979ca938a
Fix race condition merging session data when cache returns (#4579)
`persist_session/1` is used in `populate_native_stats/1` test helper
to merge session data. It sometimes gets `:ok` instead of the actual
session data so that tests randomly fail with:

```
** (BadMapError) expected a map, got: :ok
code: populate_stats(site, [build(:pageview)])
stacktrace:
(elixir 1.17.1) lib/map.ex:486: Map.take/2
(plausible 0.0.1) lib/plausible/clickhouse_event_v2.ex:92: Plausible.ClickhouseEventV2.merge_session/2
(plausible 0.0.1) test/support/test_utils.ex:188: anonymous fn/2 in Plausible.TestUtils.populate_native_stats/1
(elixir 1.17.1) lib/enum.ex:2531: Enum."-reduce/3-lists^foldl/2-0-"/3
(plausible 0.0.1) test/support/test_utils.ex:184: Plausible.TestUtils.populate_native_stats/1
(plausible 0.0.1) test/support/test_utils.ex:179: Plausible.TestUtils.populate_stats/1
test/workers/send_site_setup_emails_test.exs:46: (test)
```
2024-09-16 06:12:18 +00:00
ruslandoga
e9ced986ce
prepare for wiki (#4512) 2024-09-13 11:32:39 +00:00
Karl-Aksel Puulmann
ef57502854
APIv2: Implement pagination and include.total_rows (#4575)
Offset-based pagination is used to make sure Looker integration
is able to work as efficiently as possible. To know how many
requests users need to do `include.total_rows` option was added.
2024-09-12 15:51:18 +03:00
Artur Pata
9fcb76d8e6
Allow sorting breakdown lists by some metrics (#4513)
* On the dashboard, allow Metrics (aka column configurations) to
  * specify width they need
  * specify if they are sortable
* Default sort order for breakdown endpoints is hardcoded on the dashboard (needed to show what column the report is sorted by)
* Unifies Google Keywords modal with other breakdowns
2024-09-12 14:49:43 +03:00
Karl-Aksel Puulmann
dd1021eff3
Limit docs query API, return 401 for API routes (#4570)
* Limit docs query API, return 401 for API routes

* Changelog entry

* Use 404 with changed error message

* Update changelog
2024-09-12 14:11:22 +03:00
Karl-Aksel Puulmann
bd11b4cf67
APIv2: Standard iso8601 timestamps, operate on UTC (#4563)
* query.date_range is now in UTC instead of user timezone

This simplifies things down the line and fixes several bugs where
query.date_range is cast to naivedatetime for ecto purposes

Many places still remain broken:
- comparison queries
- `to_date_range` calls

* Make default_for_date_range not care about time zones

* Make timezone parameter mandatory for to_date_range

* Simplify utc_date_range, update legacy query builder

* Fix more cases where query date range is needed

* query.date_range -> query.utc_time_range

* Query.date_range/1 function

* ensure_include_imported update

* Clean up send_email_report
2024-09-11 09:21:59 +03:00
Artur Pata
52b94842c0
Assert filters are tuples, simplify schema (#4541) 2024-09-10 18:01:42 +03:00
Karl-Aksel Puulmann
e8d544c841
Remove does_not_contain support (#4564)
It only needed to be live until users have reload. This has been live
for >24h.
2024-09-10 15:38:04 +03:00
ruslandoga
2180ab4764
Auto HTTPS in CE (#4491)
* auto https

* changelog

* enable auto-tls only when http_port is 80

* make erlang ssl less verbose

* simplify config

* simplify config

* fix ce_dev typo

* fix another typo

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-10 14:15:43 +02:00
Adrian Gruntkowski
8ba5f7d32f
Safeguard queries relying on sign from faulty old session entries (#4555)
* Safeguard session queries relying on `sign` from faulty old session entries

* Comment updated metric

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* Apply safeguards to `bounce_rate` metric only

* Add note to bounce rate definition in SQL fragments as well

* Add test for graceful bounce rate handling in breakdown

* Make user_id more unique

* Add a note to the test

* Move regression test to APIv2 tests

---------

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
2024-09-10 13:22:49 +02:00
ruslandoga
c536af0df0
Don't attempt to create a database if it already exists (#4498)
* don't attempt to create a database if already created

* add tests

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-09 11:55:39 +02:00
ruslandoga
d17ac82058
Remove Timex.now (#4546)
* rm Timex.now

* fix test

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-09 11:40:15 +02:00
ruslandoga
8ee4827fea
Run migrations in order across repos (#4466)
* order migrations

* migrate in 'streaks'

* cleanup

* docs

* tests

* add comment

* changelog

* continue

* run migrations tests in ci

* add pending_streams cmd

* fix ci

* add docs

* fix test

* simplify test

* only test v2+

* fix test

* return async: true

* cleanup

* add 'no pending migration'

* drop migrate/0 and pending_migrations/0

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-09 11:27:11 +02:00
ruslandoga
9ba9e2abc5
Make ClickhouseRepo task timeouts configurable in CE (#4494)
* remove ch timeouts from ce

* just infinity

* make timeout configurable

* quadruple

* add test

* improve test

* fix test filtering in ci

* just don't compile the test in ee

* fix test in ee

* maybe this would work

* rm test

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-09 11:20:06 +02:00
hq1
0031d1487d
Verification: Add super-admin report displaying diagnostics (#4554)
* Verification: Add super-admin report displaying diagnostics

* Revert LSP not catching up with formatter rules on time

* Remove redundant condition
2024-09-09 10:49:29 +02:00
Karl-Aksel Puulmann
604dde99fd
APIv2: Regex operations, consistent operators (#4488)
* Rename matches/does_not_match filters internally

These have never been exposed to the frontend/user directly, only via
APIv1 filtering syntax. As such we are free to rename these without
breaking things

* Rename function arguments for consistency, simplify

* Add support for `match`/`not_match` operators for query apiv2

These match the string against a regular expression, as defined in
https://github.com/google/re2/wiki/Syntax

* not_match -> match_not

* does_not_contain -> contains_not

Note that for backwards compatibility:
- Browser handles does_not_contain in URL
- Backend will handle does_not_contain in queries for a day where we will remove it for better autocompletion

* not_matches_wildcard -> matches_wildcard_not

* prettier

* match -> matches

* Fix and test fix for matches_wildcard against prop when prop is missing

* Custom properties support for matches/matches_not

* Restore contains_not

* Test contains and contains_not behavior for custom properties
2024-09-09 10:05:24 +03:00
hq1
b9493f687c
Replace native tooltips with simplified, generic component (#4528)
* Replace native tooltips with simplified, generic component

* Make tooltip a button?

* Revert "Make tooltip a button?"

This reverts commit adacc560f9.
2024-09-05 12:18:16 +02:00
ruslandoga
e7ac60c067
Return domain-less cookies in CE (#4482) 2024-09-05 12:05:40 +02:00
ruslandoga
654558dbf9
Remove Timex from Plausible.Timezones (#3899)
* use stdlib for timezones

* Remove unused `Timezones.to_utc_datetime/2`

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-05 11:48:07 +02:00
Uku Taht
d56d6998df
Acquisition channel (#4489)
* WIP

* Add acquisition channel

* Add detection for gclid and msclkid

* Add GA4 source categories file as external resource
2024-09-05 12:02:15 +03:00
Karl-Aksel Puulmann
8fa3a83129
APIv2: and/or/not support (#4480)
* First approximation of AND/OR/NOT support

Broken by this:
- Goal filtering
- Table deciding
- Imports

* TableDecider handle nesting

* Query.remove_top_level_filters

* Plausible.Stats.Imported.SQL.Expression

* Handle AND/OR/NOT with imported data, create Plausible.Stats.Imported.SQL.WhereBuilder

* Add parser validations for event:goal, event:hostname and event:props:x filters top level constraints

* Move module around

* Query.get_filter -> Filters.filtering_on_dimension? in some callsites

* Filters.get_toplevel_filter

* TableDecider.sessions_join_events?, remove old method

* Transforming filters in query_optimizer

* Query API tests for and/or/not

* Reorder parser steps

* Post-merge test fixups

* Solve merge issue

* Simplify filtering_on_dimension?

* Update transformer code

* dimensions_used_in_filters min_depth option, simplify parser validations

* rename_dimensions_used_in_filter

* fix rename_dimensions_used_in_filter

* Rename a test
2024-09-04 15:44:03 +03:00
hq1
09999f8b0c
Disguise dogfooding LV URLs (#4529) 2024-09-04 11:36:00 +02:00
Karl-Aksel Puulmann
059b5e0cdd
Dont include imports for time:hour and time:minute dimensions (#4504)
* Dont include imports for time:hour and time:minute dimensions

Also include more information about import warnings in query results

* Update lib/plausible/stats/query_result.ex

Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>

* Revert patch

* Imported disabled graph notice (#4522)

* add explicit skip_imported_reason for unsupported interval

* stop returning information about imports from main_graph

* return warning about interval in Stats API Timeseries

* display warning bubble about interval too short for imported data

* update changelog

* improve styling of the exclamation circle icon

* return tuple from timeseries instead of map

* rename variable

* Update CHANGELOG.md

---------

Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
2024-09-04 10:53:00 +03:00
Adrian Gruntkowski
d0619aaea0
Redirect to login on expired legacy session (#4523)
After refactor, it turned out that when the _legacy_ session times out,
the conn is halted but no body is sent. This results in 500 error (`Plug.Conn.NotSentError`).

This PR fixes it by redirecting to login page.
2024-09-03 21:48:05 +02:00
ruslandoga
530d290678
Adapt site_setup_success email for CE (#4496)
* adapt site_setup_success email for ce

* add tests

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-03 14:50:08 +02:00
hq1
acefd792fb
Installation: don't crash when socket reconnects and finds no changes (#4520) 2024-09-03 13:59:29 +02:00
ruslandoga
6c7f123d2c
Add better assets compression to CE (#4476)
* add better assets compression to ce

* changelog

* fix conditional compilation

* improve changelog message

* don't need zstd
2024-09-03 13:54:11 +02:00
ruslandoga
4a36148f9b
Better Google error messages in CE (#4485) 2024-09-03 13:44:34 +02:00
ruslandoga
2634ff7673
Fix HTML-in-JSON in ErrorView (#4500)
* fix html-in-json errors

* add tests

---------

Co-authored-by: hq1 <hq@mtod.org>
2024-09-03 13:35:07 +02:00
hq1
53d94b5b1b
Remove custom background from "Choose Plan" page (#4516)
* Remove custom background on "Choose Plan" page

* Avoid `UserAuth` test failing due to bad timing

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-03 13:17:26 +02:00
Adrian Gruntkowski
373d4dd665
Implement token-based sessions (#4463)
* Turn `Plausible.Auth.UserSession` into full schema

* Implement token based sessions and use them as default

* Ignore expired user sessions during retrieval from DB

* Implement plug bumping user session last used and timeout timestamps

* Implement Oban worker removing expired user sessions with grace period

* Implement legacy session conversion on touch, when applicable

* Update `UserAuth` moduledoc

* Extend `UserAuth` tests to account for db-backed session tokens

* Update CHANGELOG

* Add tests for `UserSessionTouch` plug

* Add test for `CleanUserSessions` worker

* Add logging of legacy session retrievals

* Use single update permitting stale records  when touching user session

* Don't fetch session and user for external API endpoints (/api/event too)

* Refactor `Users.with_subscription/1` and expose helper query

* Skip fetching session in legacy `SessionTimeoutPlug`

* Rely on user session assign from `AuthContext` in `SentryContext`

* Silence legacy session warnings in `UserSessionTouchTest`

* Rely on session assign from `AuthPlug` in `SuperAdminOnlyPlug`

* Change `UserAuth` to get session, user and last subscription in one go

* Avoid refetching user session in `AuthorizeSiteAccess` plug

* Fix code formatting

* Refactor `UserAuth.get_user_token/1` (h/t @aerosol)

* Remove bogus empty opts from `scope` declarations in router

* Only touch session once an hour and keep `user.last_seen` in sync

* Bring back logging of legacy token use
2024-09-03 11:34:37 +02:00
Adrian Gruntkowski
8a1c6e0913
Enforce sequential processing of session events (#4493)
* Create a regression demonstration test for race condition

* Use `ConCache.isolated/1` to force sequential processing of session events

* Revise comment in regression test

* Put lock call behind cache adapter API

* Add more explicit handling of failing lock

NOTE: Apparent double execution of lock function needs to be investigated.

* Improve slow lock cases tests

* Reduce number of session cache locks and instrument them w/ telemetry

* Format

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2024-09-03 09:29:32 +02:00
RobertJoonas
93c9061ab2
Proper fix for realtime timeseries queries (#4509)
* enforce same timezones in date_range param

* use query.date_range.first.timezone as query.timezone
2024-09-03 10:12:50 +03:00
hq1
caef1e80f4
Allow installation lookups by superadmins (#4510) 2024-09-03 07:56:32 +02:00
Joris Drenth
49ff00252e
Fix documentation link for Google integration (#4472)
* Fix documentation link for Google integration

* Fix documentation link for Google integration (test update)

* Update lib/plausible_web/templates/site/settings_search_console.html.heex

---------

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-02 21:19:43 +02:00
hq1
979764d010
Onboarding improvements improvements (#4507)
* Make it clear(er) that info links are clickable

* Don't pollute browser history with checkbox states
2024-09-02 20:06:27 +02:00
RobertJoonas
5e26e2c97d
Fix realtime periods returning wrong date range for non-UTC timezone sites (#4508)
* fix realtime

* mix format
2024-09-02 19:21:17 +03:00
Artur Pata
9b59e95c33
Support order_by in URL params of breakdown endpoints (#4502)
* Support order_by in URL params of breakdown endpoints
2024-09-02 15:00:18 +03:00
hq1
07d3582746
Fix password reset link (#4505) 2024-09-02 13:06:50 +02:00
hq1
e3af1a317d
Onboarding improvements (#4459)
* Migration: add installation meta

* Update site schema with installation meta

* Remove VERIFICATION_ENABLED env var

* Add context API to create/remove special goals

* Add context api to update installation meta

* Remove verification enabled check

* Update new progress flow definitions

* Update generic components

* Remove internal /status API

* Implement installation live view

* Update traffic change notifier link

* Update verification, no more modal

* Update routes

* Remove focus.html - will unify everything under app layout

* Fix broken link

* Update templates with focus_box mostly

* Update controller tests

* Update controllers and stop using the focus layout

* copy changes

* Update verification.ex

* Remove dead template

* Update settings_general.html.heex

* Update copy in tests

* Update installation.ex

* Remove dangling dot

* Fix link

* Update installation.ex

* Update installation.ex

* Better tooltips?

* Simpler labels

* Revert "Simpler labels"

This reverts commit 797560ef82f2067458b03b884be5aecc8fdc72bc.

* Add copy to clipboard link and fix snippet's dark mode

* Offer installation detection skip only if ws connected

* Put COPY link at the bottom with background

* Make tooltips link to docs

* Fix cherry-pick gone wrong

* Hide tooltips on mobile screens

* WIP: 404 tracking wizard

* Revert "WIP: 404 tracking wizard"

This reverts commit a9c9c79bbd.

* Update lib/plausible_web/live/components/verification.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update lib/plausible_web/live/installation.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Use current_user from socket.assigns

* Update lib/plausible_web/live/installation.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Use current_user from socket.assigns

* Use conn.private to steer verification tests

* Drop non-sticky tooltip in favour of component parametrization

Co-authored-by: Artur Pata <artur.pata@gmail.com>

* Reapply "WIP: 404 tracking wizard"

This reverts commit 3ba81671d7.

* Fix installation tests including 404 tracking

* Fixup the tooltip component

* Format

* Update installation.ex

* Put flash whenever installation option changes

* Use last known installation type on domain change

* Extract user flow definition to provide compile-time checks

* See if this helps running CE migrations successfully

* Use `styled_link` on registration/login views

* Don't crash when there's no conn.private carried over

* Format

* Push "Determining installation type" message a bit lower

* Use links and footer lists uniformly

This commit introduces a `<.focus_list/>` component
for rendering focus box footer links with colored
discs. It also equips generic link components
with the ability of sending non-GET requests
along with CSRF token, so we can apply uniform
styling and stop using legacy Phoenix link tags.

cc @zoldar @apata

* ws 👾

* Render more descriptive flashes on script config change

---------

Co-authored-by: Marko Saric <34340819+metmarkosaric@users.noreply.github.com>
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
Co-authored-by: Artur Pata <artur.pata@gmail.com>
2024-09-02 12:49:54 +02:00
RobertJoonas
f04c47f881
Support realtime periods in API v2 (#4469)
* add realtime date_ranges into the private API schema

This commit starts parsing date ranges into a new NaiveDateTimeRange
struct, rather than a simple Date.Range.

* transform realtime labels into negative integers + test

* move schema type argument to last position in helper functions

* allow passing a date param + tests

* Update test/plausible/stats/query_parser_test.exs

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* Update test/plausible/stats/query_parser_test.exs

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* Update test/plausible/stats/query_parser_test.exs

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* Update test/plausible/stats/query_parser_test.exs

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* keep test file structure consistent

* Turn NaiveDateTimeRange into DateTimeRange

* change 'now' field from NaiveDateTime to DateTime in v2 query

* fix minute interval labels + add missing tests

* return query_result.date_range as iso8601 timestamps with timezone

* allow timestamps with tz as date_range arguments in API v2

* delete Plausible.Timezones.to_utc_datetime

* simplify returning comparison periods

* add comment about realtime not supported in comparisons

* pass only now instead of test_opts

* drop redundant else branch

* separate tests

* stick to a single check_date_range function in tests

* fix credo error

---------

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
2024-09-02 12:56:58 +03:00
Adrian Gruntkowski
e9dd895d6c
Do not try querying imported data for unsupported props (#4467)
* Do not try querying imported data for unsupported props

* Remove unnecessary `unquote`s

Co-authored-by: hq1 <hq@mtod.org>

* Add respective regression test to APIv2 tests

---------

Co-authored-by: hq1 <hq@mtod.org>
2024-08-30 10:55:31 +02:00
ruslandoga
b73bdf932f
add /var/lib/plausible (#4275)
Co-authored-by: Cenk Kücük <cenk@plausible.io>
2024-08-28 13:44:53 +02:00
Uku Taht
2e89a50188
Map lowercase tagged sources to capitalized form during ingestion (#4417)
* Map lowercase tagged sources to capitalized form during ingestion

* @moduledoc

* Add changlog entry
2024-08-27 14:03:15 +03:00
Karl-Aksel Puulmann
9c71161eab
APIv2: JSON schema validation, separate internal and public API validation (#4464)
* Restore `date` internal parameter, validate via json schema

* Improved error formatting from json schema, get most tests passing

* Handle internal overrides to JSON schema

* Parsing tests all pass

* Remove some repeated code, enforce length/uniqueness in schema

* Explicit separation between internal and public API validation

* Mark file as external_resource

* map_join

* Update query tests

* Update query tests

* Serve schema under an /api/docs/query/schema.json endpoint

* dotify errors
2024-08-26 14:01:27 +03:00
Adrian Gruntkowski
bd93cf3b46
Refactor and consolidate user session logic (#4452)
* Extract session management from AuthController

* Don't explicitly pass `current_user_id` to `live_render`'s session

* Add ability to retrieve session and user from token via `UserAuth`

* Always fetch current user (or just id) via `UserAuth` API

* Introduce `UserSession` as an embedded schema for now

* Make `UserAuth.get_user/1` accept `UserSession` as an input

* Introduce LV auth context populating user data from session on mount

* Refactor `AuthPlug` and make it populate `current_user_session` as well

* Rely on authenticated user data provided by auth plug or LV context

* Make `Sites.get_for_user(!)` accept `User` struct as well

* Set `logged_in` cookie explicitly when it's out of sync with session

* Expand modules documentation a bit

* Improve and extend tests slightly
2024-08-23 10:53:33 +02:00
RobertJoonas
039790b82d
Prepare for graph v2 (#4451)
* add the ability to pass date param into Query.build

* stop returning interval from main_graph controller action

* globally rename 'date' interval to 'day'

* Allow parsing query date range from a 'period' param

The 'period' param will not be exposed in the public API, but makes it
possible to construct a "realtime" query.

* Revert "Allow parsing query date range from a 'period' param"

This reverts commit c5630eaef9.

* call beginning_of_time for first_datetime instead of last

* allow 'realtime' and '30m' date_range shortcuts

* evaluate date_range to 'realtime' or '30m' instead of Date.range(today, today)

* Revert "evaluate date_range to 'realtime' or '30m' instead of Date.range(today, today)"

This reverts commit a887569ec5.

* Revert "allow 'realtime' and '30m' date_range shortcuts"

This reverts commit 91ae0fa5e6.

* fix graph tooltips
2024-08-22 14:33:05 +03:00