Commit Graph

1913 Commits

Author SHA1 Message Date
ruslandoga
71fa38709f
skip verification in ce (#4604) 2024-09-23 12:17:32 +00:00
ruslandoga
1b912240e3
fix link (#4605) 2024-09-23 09:39:06 +00:00
ruslandoga
5766478887
don't render empty list element in ce (#4596) 2024-09-19 07:11:02 +00:00
Cenk Kücük
234d63e549
enrich with information about app node (#4592) 2024-09-18 17:08:24 +00:00
Artur Pata
59c7ce2ef1
Fix auth issue with POST /api/docs/query (#4593) 2024-09-18 15:46:05 +00:00
Artur Pata
82a15884ad
Automatically generate Typescript types for v2 API query schema (#4574)
* Generate types from query schema

* Flip the query schema so private is static

* Ensure private schema stays private

* Refactor comment, json schema utils
2024-09-18 11:01:20 +00:00
Uku Taht
7a77ebf9bf
Add feature-flagged channels UI (#4585)
* Add feature-flagged channels UI

* Implement channels modal

* Channel -> Channels tab
2024-09-18 08:34:12 +00:00
Adrian Gruntkowski
6769ebdc24
Refactor AuthorizeSiteAccess (#4583)
* Refactor and fix `AuthorizeSiteAccess` plug shared link case handling

* Make plug module name more consistent

* Add moduledoc

* Add regression test for shared link auth case

* Adjust failing case test to account for different response

* Further restrict accepted input and configuration

* Extend plug-specific tests

* Tag EE-only test properly

* Remove one DB roundtrip from AuthorizeSiteAccess

* Improve naming slightly

* Filter shared link by site ID on query level already

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
2024-09-17 12:52:20 +00:00
RobertJoonas
45c0d538da
return city codes as integers in suggestions (#4589) 2024-09-17 12:37:13 +00:00
Adrian Gruntkowski
f7bbbf8c33
Remove support for legacy user sessions (#4540)
* Remove support for legacy user sessions

* Implement revoking all sessions for a given user

* Revoke all user sessions on password reset

* Add tests for revoking all user sessions on password reset

* Reload page when dashboard API request fails with 404

* Revert "Reload page when dashboard API request fails with 404"

This reverts commit 77d1a1035658915f9afe538afc5fb9a3da0ec905.
2024-09-17 07:31:41 +00:00
Marko Saric
68e03c2112
Update to the copy (#4558)
* Update to the copy

Update to the copy to include a note about billable pageviews. hopefully it doesn't make the copy too long...

* Don't show tooltip on small/mobile screens

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-16 13:50:36 +00:00
Adrian Gruntkowski
ae0c5a173d
Drop event explicitly on session lock timeout (#4582)
* Drop event explicitly on session lock timeout

* Make session `user_id` more random in tests to avoid excess locking

* Improve testability to event ingestion

* Test lock timeout in ingestion
2024-09-16 10:36:53 +00:00
Karl-Aksel Puulmann
748e0d902f
APIv2: Make imports_skip_reason flag not appear when imports not requested (#4565)
* APIv2: Make imports_skip_reason flag not appear when imports not requested

* Update lib/plausible/stats/query_result.ex

Co-authored-by: hq1 <hq@mtod.org>

---------

Co-authored-by: hq1 <hq@mtod.org>
2024-09-16 06:13:32 +00:00
hq1
f979ca938a
Fix race condition merging session data when cache returns (#4579)
`persist_session/1` is used in `populate_native_stats/1` test helper
to merge session data. It sometimes gets `:ok` instead of the actual
session data so that tests randomly fail with:

```
** (BadMapError) expected a map, got: :ok
code: populate_stats(site, [build(:pageview)])
stacktrace:
(elixir 1.17.1) lib/map.ex:486: Map.take/2
(plausible 0.0.1) lib/plausible/clickhouse_event_v2.ex:92: Plausible.ClickhouseEventV2.merge_session/2
(plausible 0.0.1) test/support/test_utils.ex:188: anonymous fn/2 in Plausible.TestUtils.populate_native_stats/1
(elixir 1.17.1) lib/enum.ex:2531: Enum."-reduce/3-lists^foldl/2-0-"/3
(plausible 0.0.1) test/support/test_utils.ex:184: Plausible.TestUtils.populate_native_stats/1
(plausible 0.0.1) test/support/test_utils.ex:179: Plausible.TestUtils.populate_stats/1
test/workers/send_site_setup_emails_test.exs:46: (test)
```
2024-09-16 06:12:18 +00:00
ruslandoga
e9ced986ce
prepare for wiki (#4512) 2024-09-13 11:32:39 +00:00
Karl-Aksel Puulmann
ef57502854
APIv2: Implement pagination and include.total_rows (#4575)
Offset-based pagination is used to make sure Looker integration
is able to work as efficiently as possible. To know how many
requests users need to do `include.total_rows` option was added.
2024-09-12 15:51:18 +03:00
Artur Pata
9fcb76d8e6
Allow sorting breakdown lists by some metrics (#4513)
* On the dashboard, allow Metrics (aka column configurations) to
  * specify width they need
  * specify if they are sortable
* Default sort order for breakdown endpoints is hardcoded on the dashboard (needed to show what column the report is sorted by)
* Unifies Google Keywords modal with other breakdowns
2024-09-12 14:49:43 +03:00
Karl-Aksel Puulmann
dd1021eff3
Limit docs query API, return 401 for API routes (#4570)
* Limit docs query API, return 401 for API routes

* Changelog entry

* Use 404 with changed error message

* Update changelog
2024-09-12 14:11:22 +03:00
Karl-Aksel Puulmann
bd11b4cf67
APIv2: Standard iso8601 timestamps, operate on UTC (#4563)
* query.date_range is now in UTC instead of user timezone

This simplifies things down the line and fixes several bugs where
query.date_range is cast to naivedatetime for ecto purposes

Many places still remain broken:
- comparison queries
- `to_date_range` calls

* Make default_for_date_range not care about time zones

* Make timezone parameter mandatory for to_date_range

* Simplify utc_date_range, update legacy query builder

* Fix more cases where query date range is needed

* query.date_range -> query.utc_time_range

* Query.date_range/1 function

* ensure_include_imported update

* Clean up send_email_report
2024-09-11 09:21:59 +03:00
Artur Pata
52b94842c0
Assert filters are tuples, simplify schema (#4541) 2024-09-10 18:01:42 +03:00
Karl-Aksel Puulmann
e8d544c841
Remove does_not_contain support (#4564)
It only needed to be live until users have reload. This has been live
for >24h.
2024-09-10 15:38:04 +03:00
ruslandoga
2180ab4764
Auto HTTPS in CE (#4491)
* auto https

* changelog

* enable auto-tls only when http_port is 80

* make erlang ssl less verbose

* simplify config

* simplify config

* fix ce_dev typo

* fix another typo

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-10 14:15:43 +02:00
Adrian Gruntkowski
8ba5f7d32f
Safeguard queries relying on sign from faulty old session entries (#4555)
* Safeguard session queries relying on `sign` from faulty old session entries

* Comment updated metric

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* Apply safeguards to `bounce_rate` metric only

* Add note to bounce rate definition in SQL fragments as well

* Add test for graceful bounce rate handling in breakdown

* Make user_id more unique

* Add a note to the test

* Move regression test to APIv2 tests

---------

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
2024-09-10 13:22:49 +02:00
ruslandoga
c536af0df0
Don't attempt to create a database if it already exists (#4498)
* don't attempt to create a database if already created

* add tests

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-09 11:55:39 +02:00
ruslandoga
d17ac82058
Remove Timex.now (#4546)
* rm Timex.now

* fix test

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-09 11:40:15 +02:00
ruslandoga
8ee4827fea
Run migrations in order across repos (#4466)
* order migrations

* migrate in 'streaks'

* cleanup

* docs

* tests

* add comment

* changelog

* continue

* run migrations tests in ci

* add pending_streams cmd

* fix ci

* add docs

* fix test

* simplify test

* only test v2+

* fix test

* return async: true

* cleanup

* add 'no pending migration'

* drop migrate/0 and pending_migrations/0

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-09 11:27:11 +02:00
ruslandoga
9ba9e2abc5
Make ClickhouseRepo task timeouts configurable in CE (#4494)
* remove ch timeouts from ce

* just infinity

* make timeout configurable

* quadruple

* add test

* improve test

* fix test filtering in ci

* just don't compile the test in ee

* fix test in ee

* maybe this would work

* rm test

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-09 11:20:06 +02:00
hq1
0031d1487d
Verification: Add super-admin report displaying diagnostics (#4554)
* Verification: Add super-admin report displaying diagnostics

* Revert LSP not catching up with formatter rules on time

* Remove redundant condition
2024-09-09 10:49:29 +02:00
Karl-Aksel Puulmann
604dde99fd
APIv2: Regex operations, consistent operators (#4488)
* Rename matches/does_not_match filters internally

These have never been exposed to the frontend/user directly, only via
APIv1 filtering syntax. As such we are free to rename these without
breaking things

* Rename function arguments for consistency, simplify

* Add support for `match`/`not_match` operators for query apiv2

These match the string against a regular expression, as defined in
https://github.com/google/re2/wiki/Syntax

* not_match -> match_not

* does_not_contain -> contains_not

Note that for backwards compatibility:
- Browser handles does_not_contain in URL
- Backend will handle does_not_contain in queries for a day where we will remove it for better autocompletion

* not_matches_wildcard -> matches_wildcard_not

* prettier

* match -> matches

* Fix and test fix for matches_wildcard against prop when prop is missing

* Custom properties support for matches/matches_not

* Restore contains_not

* Test contains and contains_not behavior for custom properties
2024-09-09 10:05:24 +03:00
hq1
b9493f687c
Replace native tooltips with simplified, generic component (#4528)
* Replace native tooltips with simplified, generic component

* Make tooltip a button?

* Revert "Make tooltip a button?"

This reverts commit adacc560f9.
2024-09-05 12:18:16 +02:00
ruslandoga
e7ac60c067
Return domain-less cookies in CE (#4482) 2024-09-05 12:05:40 +02:00
ruslandoga
654558dbf9
Remove Timex from Plausible.Timezones (#3899)
* use stdlib for timezones

* Remove unused `Timezones.to_utc_datetime/2`

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-05 11:48:07 +02:00
Uku Taht
d56d6998df
Acquisition channel (#4489)
* WIP

* Add acquisition channel

* Add detection for gclid and msclkid

* Add GA4 source categories file as external resource
2024-09-05 12:02:15 +03:00
Karl-Aksel Puulmann
8fa3a83129
APIv2: and/or/not support (#4480)
* First approximation of AND/OR/NOT support

Broken by this:
- Goal filtering
- Table deciding
- Imports

* TableDecider handle nesting

* Query.remove_top_level_filters

* Plausible.Stats.Imported.SQL.Expression

* Handle AND/OR/NOT with imported data, create Plausible.Stats.Imported.SQL.WhereBuilder

* Add parser validations for event:goal, event:hostname and event:props:x filters top level constraints

* Move module around

* Query.get_filter -> Filters.filtering_on_dimension? in some callsites

* Filters.get_toplevel_filter

* TableDecider.sessions_join_events?, remove old method

* Transforming filters in query_optimizer

* Query API tests for and/or/not

* Reorder parser steps

* Post-merge test fixups

* Solve merge issue

* Simplify filtering_on_dimension?

* Update transformer code

* dimensions_used_in_filters min_depth option, simplify parser validations

* rename_dimensions_used_in_filter

* fix rename_dimensions_used_in_filter

* Rename a test
2024-09-04 15:44:03 +03:00
hq1
09999f8b0c
Disguise dogfooding LV URLs (#4529) 2024-09-04 11:36:00 +02:00
Karl-Aksel Puulmann
059b5e0cdd
Dont include imports for time:hour and time:minute dimensions (#4504)
* Dont include imports for time:hour and time:minute dimensions

Also include more information about import warnings in query results

* Update lib/plausible/stats/query_result.ex

Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>

* Revert patch

* Imported disabled graph notice (#4522)

* add explicit skip_imported_reason for unsupported interval

* stop returning information about imports from main_graph

* return warning about interval in Stats API Timeseries

* display warning bubble about interval too short for imported data

* update changelog

* improve styling of the exclamation circle icon

* return tuple from timeseries instead of map

* rename variable

* Update CHANGELOG.md

---------

Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
2024-09-04 10:53:00 +03:00
Adrian Gruntkowski
d0619aaea0
Redirect to login on expired legacy session (#4523)
After refactor, it turned out that when the _legacy_ session times out,
the conn is halted but no body is sent. This results in 500 error (`Plug.Conn.NotSentError`).

This PR fixes it by redirecting to login page.
2024-09-03 21:48:05 +02:00
ruslandoga
530d290678
Adapt site_setup_success email for CE (#4496)
* adapt site_setup_success email for ce

* add tests

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-03 14:50:08 +02:00
hq1
acefd792fb
Installation: don't crash when socket reconnects and finds no changes (#4520) 2024-09-03 13:59:29 +02:00
ruslandoga
6c7f123d2c
Add better assets compression to CE (#4476)
* add better assets compression to ce

* changelog

* fix conditional compilation

* improve changelog message

* don't need zstd
2024-09-03 13:54:11 +02:00
ruslandoga
4a36148f9b
Better Google error messages in CE (#4485) 2024-09-03 13:44:34 +02:00
ruslandoga
2634ff7673
Fix HTML-in-JSON in ErrorView (#4500)
* fix html-in-json errors

* add tests

---------

Co-authored-by: hq1 <hq@mtod.org>
2024-09-03 13:35:07 +02:00
hq1
53d94b5b1b
Remove custom background from "Choose Plan" page (#4516)
* Remove custom background on "Choose Plan" page

* Avoid `UserAuth` test failing due to bad timing

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-03 13:17:26 +02:00
Adrian Gruntkowski
373d4dd665
Implement token-based sessions (#4463)
* Turn `Plausible.Auth.UserSession` into full schema

* Implement token based sessions and use them as default

* Ignore expired user sessions during retrieval from DB

* Implement plug bumping user session last used and timeout timestamps

* Implement Oban worker removing expired user sessions with grace period

* Implement legacy session conversion on touch, when applicable

* Update `UserAuth` moduledoc

* Extend `UserAuth` tests to account for db-backed session tokens

* Update CHANGELOG

* Add tests for `UserSessionTouch` plug

* Add test for `CleanUserSessions` worker

* Add logging of legacy session retrievals

* Use single update permitting stale records  when touching user session

* Don't fetch session and user for external API endpoints (/api/event too)

* Refactor `Users.with_subscription/1` and expose helper query

* Skip fetching session in legacy `SessionTimeoutPlug`

* Rely on user session assign from `AuthContext` in `SentryContext`

* Silence legacy session warnings in `UserSessionTouchTest`

* Rely on session assign from `AuthPlug` in `SuperAdminOnlyPlug`

* Change `UserAuth` to get session, user and last subscription in one go

* Avoid refetching user session in `AuthorizeSiteAccess` plug

* Fix code formatting

* Refactor `UserAuth.get_user_token/1` (h/t @aerosol)

* Remove bogus empty opts from `scope` declarations in router

* Only touch session once an hour and keep `user.last_seen` in sync

* Bring back logging of legacy token use
2024-09-03 11:34:37 +02:00
Adrian Gruntkowski
8a1c6e0913
Enforce sequential processing of session events (#4493)
* Create a regression demonstration test for race condition

* Use `ConCache.isolated/1` to force sequential processing of session events

* Revise comment in regression test

* Put lock call behind cache adapter API

* Add more explicit handling of failing lock

NOTE: Apparent double execution of lock function needs to be investigated.

* Improve slow lock cases tests

* Reduce number of session cache locks and instrument them w/ telemetry

* Format

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2024-09-03 09:29:32 +02:00
RobertJoonas
93c9061ab2
Proper fix for realtime timeseries queries (#4509)
* enforce same timezones in date_range param

* use query.date_range.first.timezone as query.timezone
2024-09-03 10:12:50 +03:00
hq1
caef1e80f4
Allow installation lookups by superadmins (#4510) 2024-09-03 07:56:32 +02:00
Joris Drenth
49ff00252e
Fix documentation link for Google integration (#4472)
* Fix documentation link for Google integration

* Fix documentation link for Google integration (test update)

* Update lib/plausible_web/templates/site/settings_search_console.html.heex

---------

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-02 21:19:43 +02:00
hq1
979764d010
Onboarding improvements improvements (#4507)
* Make it clear(er) that info links are clickable

* Don't pollute browser history with checkbox states
2024-09-02 20:06:27 +02:00
RobertJoonas
5e26e2c97d
Fix realtime periods returning wrong date range for non-UTC timezone sites (#4508)
* fix realtime

* mix format
2024-09-02 19:21:17 +03:00
Artur Pata
9b59e95c33
Support order_by in URL params of breakdown endpoints (#4502)
* Support order_by in URL params of breakdown endpoints
2024-09-02 15:00:18 +03:00
hq1
07d3582746
Fix password reset link (#4505) 2024-09-02 13:06:50 +02:00
hq1
e3af1a317d
Onboarding improvements (#4459)
* Migration: add installation meta

* Update site schema with installation meta

* Remove VERIFICATION_ENABLED env var

* Add context API to create/remove special goals

* Add context api to update installation meta

* Remove verification enabled check

* Update new progress flow definitions

* Update generic components

* Remove internal /status API

* Implement installation live view

* Update traffic change notifier link

* Update verification, no more modal

* Update routes

* Remove focus.html - will unify everything under app layout

* Fix broken link

* Update templates with focus_box mostly

* Update controller tests

* Update controllers and stop using the focus layout

* copy changes

* Update verification.ex

* Remove dead template

* Update settings_general.html.heex

* Update copy in tests

* Update installation.ex

* Remove dangling dot

* Fix link

* Update installation.ex

* Update installation.ex

* Better tooltips?

* Simpler labels

* Revert "Simpler labels"

This reverts commit 797560ef82f2067458b03b884be5aecc8fdc72bc.

* Add copy to clipboard link and fix snippet's dark mode

* Offer installation detection skip only if ws connected

* Put COPY link at the bottom with background

* Make tooltips link to docs

* Fix cherry-pick gone wrong

* Hide tooltips on mobile screens

* WIP: 404 tracking wizard

* Revert "WIP: 404 tracking wizard"

This reverts commit a9c9c79bbd.

* Update lib/plausible_web/live/components/verification.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update lib/plausible_web/live/installation.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Use current_user from socket.assigns

* Update lib/plausible_web/live/installation.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Use current_user from socket.assigns

* Use conn.private to steer verification tests

* Drop non-sticky tooltip in favour of component parametrization

Co-authored-by: Artur Pata <artur.pata@gmail.com>

* Reapply "WIP: 404 tracking wizard"

This reverts commit 3ba81671d7.

* Fix installation tests including 404 tracking

* Fixup the tooltip component

* Format

* Update installation.ex

* Put flash whenever installation option changes

* Use last known installation type on domain change

* Extract user flow definition to provide compile-time checks

* See if this helps running CE migrations successfully

* Use `styled_link` on registration/login views

* Don't crash when there's no conn.private carried over

* Format

* Push "Determining installation type" message a bit lower

* Use links and footer lists uniformly

This commit introduces a `<.focus_list/>` component
for rendering focus box footer links with colored
discs. It also equips generic link components
with the ability of sending non-GET requests
along with CSRF token, so we can apply uniform
styling and stop using legacy Phoenix link tags.

cc @zoldar @apata

* ws 👾

* Render more descriptive flashes on script config change

---------

Co-authored-by: Marko Saric <34340819+metmarkosaric@users.noreply.github.com>
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
Co-authored-by: Artur Pata <artur.pata@gmail.com>
2024-09-02 12:49:54 +02:00
RobertJoonas
f04c47f881
Support realtime periods in API v2 (#4469)
* add realtime date_ranges into the private API schema

This commit starts parsing date ranges into a new NaiveDateTimeRange
struct, rather than a simple Date.Range.

* transform realtime labels into negative integers + test

* move schema type argument to last position in helper functions

* allow passing a date param + tests

* Update test/plausible/stats/query_parser_test.exs

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* Update test/plausible/stats/query_parser_test.exs

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* Update test/plausible/stats/query_parser_test.exs

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* Update test/plausible/stats/query_parser_test.exs

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* keep test file structure consistent

* Turn NaiveDateTimeRange into DateTimeRange

* change 'now' field from NaiveDateTime to DateTime in v2 query

* fix minute interval labels + add missing tests

* return query_result.date_range as iso8601 timestamps with timezone

* allow timestamps with tz as date_range arguments in API v2

* delete Plausible.Timezones.to_utc_datetime

* simplify returning comparison periods

* add comment about realtime not supported in comparisons

* pass only now instead of test_opts

* drop redundant else branch

* separate tests

* stick to a single check_date_range function in tests

* fix credo error

---------

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
2024-09-02 12:56:58 +03:00
Adrian Gruntkowski
e9dd895d6c
Do not try querying imported data for unsupported props (#4467)
* Do not try querying imported data for unsupported props

* Remove unnecessary `unquote`s

Co-authored-by: hq1 <hq@mtod.org>

* Add respective regression test to APIv2 tests

---------

Co-authored-by: hq1 <hq@mtod.org>
2024-08-30 10:55:31 +02:00
ruslandoga
b73bdf932f
add /var/lib/plausible (#4275)
Co-authored-by: Cenk Kücük <cenk@plausible.io>
2024-08-28 13:44:53 +02:00
Uku Taht
2e89a50188
Map lowercase tagged sources to capitalized form during ingestion (#4417)
* Map lowercase tagged sources to capitalized form during ingestion

* @moduledoc

* Add changlog entry
2024-08-27 14:03:15 +03:00
Karl-Aksel Puulmann
9c71161eab
APIv2: JSON schema validation, separate internal and public API validation (#4464)
* Restore `date` internal parameter, validate via json schema

* Improved error formatting from json schema, get most tests passing

* Handle internal overrides to JSON schema

* Parsing tests all pass

* Remove some repeated code, enforce length/uniqueness in schema

* Explicit separation between internal and public API validation

* Mark file as external_resource

* map_join

* Update query tests

* Update query tests

* Serve schema under an /api/docs/query/schema.json endpoint

* dotify errors
2024-08-26 14:01:27 +03:00
Adrian Gruntkowski
bd93cf3b46
Refactor and consolidate user session logic (#4452)
* Extract session management from AuthController

* Don't explicitly pass `current_user_id` to `live_render`'s session

* Add ability to retrieve session and user from token via `UserAuth`

* Always fetch current user (or just id) via `UserAuth` API

* Introduce `UserSession` as an embedded schema for now

* Make `UserAuth.get_user/1` accept `UserSession` as an input

* Introduce LV auth context populating user data from session on mount

* Refactor `AuthPlug` and make it populate `current_user_session` as well

* Rely on authenticated user data provided by auth plug or LV context

* Make `Sites.get_for_user(!)` accept `User` struct as well

* Set `logged_in` cookie explicitly when it's out of sync with session

* Expand modules documentation a bit

* Improve and extend tests slightly
2024-08-23 10:53:33 +02:00
RobertJoonas
039790b82d
Prepare for graph v2 (#4451)
* add the ability to pass date param into Query.build

* stop returning interval from main_graph controller action

* globally rename 'date' interval to 'day'

* Allow parsing query date range from a 'period' param

The 'period' param will not be exposed in the public API, but makes it
possible to construct a "realtime" query.

* Revert "Allow parsing query date range from a 'period' param"

This reverts commit c5630eaef9.

* call beginning_of_time for first_datetime instead of last

* allow 'realtime' and '30m' date_range shortcuts

* evaluate date_range to 'realtime' or '30m' instead of Date.range(today, today)

* Revert "evaluate date_range to 'realtime' or '30m' instead of Date.range(today, today)"

This reverts commit a887569ec5.

* Revert "allow 'realtime' and '30m' date_range shortcuts"

This reverts commit 91ae0fa5e6.

* fix graph tooltips
2024-08-22 14:33:05 +03:00
Karl-Aksel Puulmann
11acadfde9
APIv2: docs-related changes (#4453)
* Order QueryResult in API response

This improves experience in docs when querying interactively

* More utm in seeds

* More improved seeds

* Proper QueryResult.query structure

* Allow docs to query /api/v2/query and sites

The new endpoints use cookie authentication. The docs site uses
these endpoints to provide an interactive docs editor.

* query_result ordering test

* Refresh router

* Test module name
2024-08-22 10:44:41 +03:00
ruslandoga
48dfce7a8b
update ecto_ch and use ecto hints for array joins (#4456) 2024-08-21 06:28:25 +02:00
hq1
572f8abac5
First pass: unify onboarding UI (#4445)
* First pass: unify onboarding UI

* Dark mode fixes

* Format

* Update self-hosted index

* Remove unnecessary child box in Enter Your 2FA screen

* Fixup Enter Your 2FA screen
2024-08-19 09:10:54 +02:00
Adrian Gruntkowski
a20cb39652
Unify and refactor login regardless of trigger source (explicit/register) (#4434)
* Unify and refactor login regardless of trigger source (explicit or registration)

* Fix code formatting
2024-08-16 10:59:31 +02:00
Adrian Gruntkowski
77d841221b
Cut off 24h sites stats at respective native stats start dates (#4439) 2024-08-16 10:28:59 +02:00
hq1
0440d36577
Implement new onboarding/provisioning flow progress layout (#4441)
* Implement new onboarding/provisioning flow progress layout

* Delete dead template

* Fixup & test FlowProgress component
2024-08-16 10:07:48 +02:00
hq1
4d4c9a8a21
Fix dark mode in clickhouse debug view (#4442) 2024-08-14 20:51:07 +02:00
Karl-Aksel Puulmann
4967960278
Populate log_comment with debug information, /debug/clickhouse route (#4435)
* Set log_comment with request information

* CRMAuthPlug -> SuperAdminOnlyPlug

* Super basic debug view

* Handle clustered setups

* Changelog entry

* Cleanup

* fragment trick to use ecto querying, filtering

* Move clustered_table? function to IngestRepo module

* Format

* More resilient user_id getting in helper
2024-08-14 12:33:36 +03:00
Karl-Aksel Puulmann
ee3d1e770e
APIv2: visit:country_name, visit:region_name, visit:city_name dimensions (#4328)
* Add data migration for creating and syncing location_data table and dictionary

* Migration to populate location data

* Daily cron to refresh location dataset if changed

* Add support for visit:country_name, visit:region_name and visit:city_name dimensions

Under the hood this relies on a `location_data` table in clickhouse being regularly synced with
plausible/location repo and dictionary lookups used in ALIAS columns

* Update queue name

* Update documentation

* Explicit structs

* Improve docs further

* Migration comment

* Add queues

* Add error when already loaded

* Test for filtering by new dimensions

* Update deps

* dimension -> select_dimension

* Update a test
2024-08-13 09:44:58 +03:00
ruslandoga
5e2a9563f7
Remove Timex.format (#4402) 2024-08-12 10:01:13 +02:00
Adrian Gruntkowski
4c5ce0f1fe
Fix flaky auth rate limit tests and refactor auth rate limiting (#4401)
* Add safeguard against flaky auth rate limit tests

* Fix typo in a log message

* Extract and abstract rate limiting from `AuthController`

* Fix flaky rate limit test tag

* Don't leak prefix from auth rate limit

* Use more compact map syntax

* Remove special tag in favor of `eventually` test util function
2024-08-09 14:58:34 +02:00
hq1
cc769dfb3d
Edit goals with display names (#4415)
* Update Goal schema

* Equip ComboBox with the ability of JS selection callbacks

* Update factory so display_name is always present

* Extend Goals context interface

* Update seeds

Also farming unsuspecting BEAM programmers for better
sample page paths :)

* Update ComboBox test

* Unify error message color class with helpers seen elsewhere

* Use goal.display_name where applicable

* Implement LiveView extensions for editing goals

* Sprinkle display name in external stats controller tests

* Format

* Fix goal list mobile view

* Update lib/plausible_web/live/goal_settings/list.ex

Co-authored-by: Artur Pata <artur.pata@gmail.com>

* Update lib/plausible_web/live/goal_settings/form.ex

Co-authored-by: Artur Pata <artur.pata@gmail.com>

* Update the APIs: plugins and external

* Update test so the intent is clearer

* Format

* Update CHANGELOG

* Simplify form tabs tests

* Revert "Format"

This reverts commit c1647b5307.

* Fixup format commit that went too far

* ComboBox: select the input contents on first focus

* Update lib/plausible/goal/schema.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update lib/plausible/goals/goals.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update lib/plausible_web/live/goal_settings/form.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Pass form goal instead of just ID

* Make tab component dumber

* Extract separate render functions for edit and create forms

* Update test to account for extracted forms

* Inline goal get query

* Extract revenue goal settings to a component and avoid computing assigns in flight

* Make LV modal preload optional

* Disable preload for goal settings form modal

* Get rid of phash component ID hack

* For another render after render_submit when testing goal updates

* Fix LV preload option

* Enable preload back for goals modal for now

* Make formatter happy

* Implement support for preopening of LV modal

* Preopen goals modal to avoid feedback gap on loading edited goal

* Remove `console.log` call from modal JS

* Clean up display name input IDs

* Make revenue settings functional on first edit again

* Display names: 2nd stage migration

* Update migration with data backfill

---------

Co-authored-by: Artur Pata <artur.pata@gmail.com>
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-08-09 11:12:00 +02:00
RobertJoonas
1f8662438d
Add Details modals for all Devices reports (#4419)
* return concatenated browser names and versions directly from the API

* return concatenated os names and versions directly from the API

* add detailed views for all device reports

* extract put_combined_name_with_version function

* return only version under the name key when detailed=true

* update changelog

* add more metrics into detailed views of Devices reports

* split up different devices modals into separate files
2024-08-02 15:50:22 +03:00
Uku Taht
b868042581
Refactor: remove transform_filters function from imported base (#4409) 2024-07-31 15:36:25 +03:00
Marko Saric
cad7ae7f53
Change the example in the hostname block (#4403)
* Update hostname_rules.ex

* Update hostname_rules.ex
2024-07-31 15:18:38 +03:00
Uku Taht
a1ee58426e
Implement goal contains filter + search in Conversions > Details (#4404)
* Implement goal contains

* add search into ConversionsModal

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
2024-07-31 10:21:30 +03:00
Uku Taht
05b70e37a8
Generate goal filter SQL from postgres goals (#4391)
* Use goals from postgres to build goal filter

* Remove unnecessary goal preloading

* Add contains filter for goals

* Make sure the correct imported tables are used when goals are in filters

* Remove 'contains' filter type for now

* Remove TODO comment

* Fix QueryParser test

* move goals.ex into a goals subfolder

* extract goal filtering logic into a separate module

* remove duplication from Imported.Base

* Extract `get_filter_goals` function

* Credo suggestions

* Credo ignore nesting

* apply suggestion - remove Sql.Util

* apply suggestion - pass imported? via opts

* remove duplicate function

* Uncomment tests

* Fix 500 error with goal suggestions

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
2024-07-30 12:07:23 +03:00
Adrian Gruntkowski
556c896936
Add ability to lookup by email and site domain in HelpScout integration (#4377)
* Add ability to lookup by email and site domain in HelpScout integration

* Handle callback with invalid parameters gracefully

* Add routine adjusting iframe height to the content of HS integration

* Add requestStorageAccess routine to support iframe cookies in Safari

* Fill first email pick when on failed user lookup

* Fix a typo in a comment
2024-07-26 16:26:51 +02:00
RobertJoonas
37014c86a8
Tracking improvements (#4392)
* track browser_language, theme, and logged_in for 'Filter Menu: Open'

* track props for 'Signup' and 'Signup via invitation'

* do not track theme as 'system' for non-logged-in users

* fix tests
2024-07-26 15:24:55 +02:00
Adrian Gruntkowski
e2d1714f90
Make time range filter for sessions not break sample size estimation (#4394) 2024-07-26 16:19:11 +03:00
Adrian Gruntkowski
590785794f
Add search_query to internal props (#4390) 2024-07-26 09:40:23 +02:00
Uku Taht
49bb57f601
Fix email report sentry errors (#4381)
* Fix missing email param on email unsubscribe

* Guard against email report being deleted
2024-07-23 12:05:42 +03:00
RobertJoonas
2f87832532
Implement search and pagination in Google Keywords modal (#4378)
* Extend the GSC API with search functionality

* Fix typo in error tuple atom

* log GSC errors

* rename confusing variable name

* Fix the API response format and error handling

* Read results under the `results` key to be consistent with other
  endpoints

* Return errors with a non-200 status code, and with an error payload
  that will be well constructed into ApiError

* rebuild Google Keywords modal with useAPIClient

* Add pagination support in Search Terms API

* delete unused fixture file

* rename fixture files

* fix tests
2024-07-23 11:53:52 +03:00
ruslandoga
1f2ebe4d2c
Remove Timex.today (#4357)
* remove Timex.today

* format
2024-07-23 09:02:14 +02:00
Adrian Gruntkowski
0310cecef8
Add listing sites, goals and custom props to Sites API (#4302)
* Add listing sites, goals and custom props to Sites API

* Rename exmaple props in tests

* Rename `allowed_custom_props` -> `custom_properties`

* Expose goal name in GET endpoints for goals in Sites API

* Bump default pagination limit to 100 and max to 1000

* Introduce Goal.name/display_name and use the first one for name in API

* Extend goal list response but hide currency

* Settle on `display_name` instead of `name`

* Allow viewer members to get site details and list site goals

* Don't include currency in goal's display name
2024-07-19 10:05:54 +02:00
hq1
feea0628c4
Add margin to CRM link (#4373) 2024-07-18 15:30:16 +02:00
hq1
1488683d26
Implement UI to edit existing funnels (#4369)
* Update tailwind config

Turns out `extra` wasn't scanned properly with
previous wildcards. I don't care any more, it's not slow.

* Add interface for updating funnels

* Extend ComboBox with the ability to preselect an initial value

* Implement editing funnels UI

* Update CHANGELOG

* s/add_funnel/setup_funnel

* modal width 2/5 => 2/3

* Let's not make the list disappear on modal pop-up

* Fix the damn modal width again

* Watch extra dir

* Format

* Remove commented code

The test is implemented elsewhere

* Track funnel modified to drop default selection

* Fix screen size adoption and format large numbers

* Preserve currency so that string casting includes it

* Format

* Fix ComboBox attribute for preselected option
2024-07-17 22:04:36 +02:00
Uku Taht
28787198b2
Fix present_index for daily, weekly and monthly intervals (#4358)
* Fix present_index for daily, weekly and monthly intervals

* Add test for present_index
2024-07-16 12:12:38 +03:00
ruslandoga
ba77f0513e
use nimble_csv everywhere (#4183) 2024-07-16 12:12:29 +03:00
Adrian Gruntkowski
7015075e12
Show number of owned sites in HS integration with a link to sites list in CRM (#4355) 2024-07-16 09:18:34 +02:00
ruslandoga
f09db60207
Remove Timex.shift in app (#4338)
* rm Timex.shift in app

* update current_visitors_12h/1 as well

* type and fix
2024-07-15 13:31:58 +02:00
ruslandoga
14d5a8028f
Don't require TOTP vault key in CE (#4317) 2024-07-15 12:02:31 +02:00
ruslandoga
922b9ab4d6
Allow to configure Finch pools (#4270) 2024-07-15 12:01:36 +02:00
Adrian Gruntkowski
1c5c4a25aa
HelpScout integration (#4327)
* Implement basic HelpScout integration

* Set 127.0.0.1 as a default customer IP in `Plans.with_prices/2`

* Use `is_nil/1` instead of `... == nil` (h/t @aerosol)

* Use `Path.join/1,2` to build API URLs a bit more safely (h/t @aerosol)

* Check for locked sites entirely within query logic

* Move conditional start of HelpScout vault to compile-time

* Include customer_id in error sent to Sentry

* Use `Plug.Crypto.secure_compare/2` for constant-time signature comparison

* Refactor token request function

* Use `Path.join/1` in one more spot

* Use route helper to build CRM URL
2024-07-12 12:01:59 +02:00
hq1
d565175c13
Remove verification related feature flags (#4345)
- `:verification` flag is no longer needed,
   however we'll keep the old "awaiting first pageview" flow
   for Community Edition where verification is not available
   by default
 - `:traffic_drop_notifications` have been tested and should
   be enabled for everyone
2024-07-12 09:43:54 +02:00
hq1
a2497f6f8c
Open verification modal when following drop notification link (#4344)
* Open verification modal when following drop notification link

* Add test

* Fixup

* Move `open-modal` event to Components.Modal API
2024-07-12 06:26:40 +02:00
hq1
d56bb2b4d5
Implement traffic drop notifications (#4300)
* Expose current visitors 12h aggregate

* Remove unused site association

* Distinct drop/spike notification factories

* Rename modules accordingly + implement drop handling

* Rename periodic oban service

* Implement drop email

* Rest of the owl

* Update changelog

* Update moduledoc

* Update moduledoc

* Min threshold to 1

* Threshold 1

* Remove merge artifact

* Put panel behind a feature flag

* Format
2024-07-11 14:55:18 +02:00
Karl-Aksel Puulmann
3ab47e6401
Remove SAMPLE BY from an events query that joins sessions (#4341)
Some users are reporting wrong results for stats querying from events table
but with session filters.

This seems to be caused by sampling: Namely both tables end up sampled and the
incorrect sample rate seems to get used.

This is sadly impossible to test for in our test suite, but reveals itself in
production with the following query:

```sql
SELECT
    toUInt64 (round(uniq (se0.user_id) * any(_sample_factor))) AS visitors
FROM
    events_v2 AS se0 SAMPLE 20000000
    INNER JOIN (
        SELECT
            sss0.session_id AS session_id,
            any(_sample_factor) AS _sample_factor
        FROM
            sessions_v2 AS sss0 SAMPLE 20000000
        WHERE
            (sss0.site_id = _CAST (8195000, 'Int64'))
            AND (sss0.timestamp >= _CAST (1720497600, 'DateTime'))
            AND (sss0.start < _CAST (1720584000, 'DateTime'))
            AND (sss0.entry_page IN ('/'))
            AND (sss0.sign = 1)
        GROUP BY
            sss0.session_id
    ) AS ss1 ON se0.session_id = ss1.session_id
WHERE
    (se0.site_id = _CAST (8195000, 'Int64'))
    AND (se0.timestamp >= _CAST (1720497600, 'DateTime'))
    AND (se0.timestamp < _CAST (1720584000, 'DateTime'))
```

By removing the inner query SAMPLE clause the result changes >50%

Related old PR: https://github.com/plausible/analytics/pull/2962

This will likely require more work in the future so looking forward to that -
after my vacation.
2024-07-11 15:08:20 +03:00
hq1
76da6b49a1
Migration: add traffic change type (#4325)
* Migration: add traffic change type

* Update schema
2024-07-11 11:01:11 +02:00
Karl-Aksel Puulmann
0114689779
fix(TimeSeries): Work around weird time zones (#4337)
* Refactor Expression.dimension to accept q

* Handle quarter- and half-hour timezones

Previously APIv2 output didn't start at a full hour for these time zones
and main graph was blank

The core reasoning is that ClickHouse `timeSlots` is not time-zone
aware and works off of unix epoch - meaning that in time zones which
have an offset of 5:45 the "hours" reported would start at minute :45.

The fix is kind of silly - we now divide each hour into 4 and handle
things that way.

Related basecamp issue: https://3.basecamp.com/5308029/buckets/36789884/card_tables/cards/7590936581

* Fix test typo
2024-07-11 11:48:16 +03:00