Commit Graph

610 Commits

Author SHA1 Message Date
Adrian Gruntkowski
8ba5f7d32f
Safeguard queries relying on sign from faulty old session entries (#4555)
* Safeguard session queries relying on `sign` from faulty old session entries

* Comment updated metric

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* Apply safeguards to `bounce_rate` metric only

* Add note to bounce rate definition in SQL fragments as well

* Add test for graceful bounce rate handling in breakdown

* Make user_id more unique

* Add a note to the test

* Move regression test to APIv2 tests

---------

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
2024-09-10 13:22:49 +02:00
Adrian Gruntkowski
c105ebceaf
Fix test failing due to timestamp setup (#4562) 2024-09-10 12:30:49 +02:00
Karl-Aksel Puulmann
604dde99fd
APIv2: Regex operations, consistent operators (#4488)
* Rename matches/does_not_match filters internally

These have never been exposed to the frontend/user directly, only via
APIv1 filtering syntax. As such we are free to rename these without
breaking things

* Rename function arguments for consistency, simplify

* Add support for `match`/`not_match` operators for query apiv2

These match the string against a regular expression, as defined in
https://github.com/google/re2/wiki/Syntax

* not_match -> match_not

* does_not_contain -> contains_not

Note that for backwards compatibility:
- Browser handles does_not_contain in URL
- Backend will handle does_not_contain in queries for a day where we will remove it for better autocompletion

* not_matches_wildcard -> matches_wildcard_not

* prettier

* match -> matches

* Fix and test fix for matches_wildcard against prop when prop is missing

* Custom properties support for matches/matches_not

* Restore contains_not

* Test contains and contains_not behavior for custom properties
2024-09-09 10:05:24 +03:00
Uku Taht
d56d6998df
Acquisition channel (#4489)
* WIP

* Add acquisition channel

* Add detection for gclid and msclkid

* Add GA4 source categories file as external resource
2024-09-05 12:02:15 +03:00
Karl-Aksel Puulmann
8fa3a83129
APIv2: and/or/not support (#4480)
* First approximation of AND/OR/NOT support

Broken by this:
- Goal filtering
- Table deciding
- Imports

* TableDecider handle nesting

* Query.remove_top_level_filters

* Plausible.Stats.Imported.SQL.Expression

* Handle AND/OR/NOT with imported data, create Plausible.Stats.Imported.SQL.WhereBuilder

* Add parser validations for event:goal, event:hostname and event:props:x filters top level constraints

* Move module around

* Query.get_filter -> Filters.filtering_on_dimension? in some callsites

* Filters.get_toplevel_filter

* TableDecider.sessions_join_events?, remove old method

* Transforming filters in query_optimizer

* Query API tests for and/or/not

* Reorder parser steps

* Post-merge test fixups

* Solve merge issue

* Simplify filtering_on_dimension?

* Update transformer code

* dimensions_used_in_filters min_depth option, simplify parser validations

* rename_dimensions_used_in_filter

* fix rename_dimensions_used_in_filter

* Rename a test
2024-09-04 15:44:03 +03:00
Karl-Aksel Puulmann
059b5e0cdd
Dont include imports for time:hour and time:minute dimensions (#4504)
* Dont include imports for time:hour and time:minute dimensions

Also include more information about import warnings in query results

* Update lib/plausible/stats/query_result.ex

Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>

* Revert patch

* Imported disabled graph notice (#4522)

* add explicit skip_imported_reason for unsupported interval

* stop returning information about imports from main_graph

* return warning about interval in Stats API Timeseries

* display warning bubble about interval too short for imported data

* update changelog

* improve styling of the exclamation circle icon

* return tuple from timeseries instead of map

* rename variable

* Update CHANGELOG.md

---------

Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
2024-09-04 10:53:00 +03:00
ruslandoga
4a36148f9b
Better Google error messages in CE (#4485) 2024-09-03 13:44:34 +02:00
Adrian Gruntkowski
373d4dd665
Implement token-based sessions (#4463)
* Turn `Plausible.Auth.UserSession` into full schema

* Implement token based sessions and use them as default

* Ignore expired user sessions during retrieval from DB

* Implement plug bumping user session last used and timeout timestamps

* Implement Oban worker removing expired user sessions with grace period

* Implement legacy session conversion on touch, when applicable

* Update `UserAuth` moduledoc

* Extend `UserAuth` tests to account for db-backed session tokens

* Update CHANGELOG

* Add tests for `UserSessionTouch` plug

* Add test for `CleanUserSessions` worker

* Add logging of legacy session retrievals

* Use single update permitting stale records  when touching user session

* Don't fetch session and user for external API endpoints (/api/event too)

* Refactor `Users.with_subscription/1` and expose helper query

* Skip fetching session in legacy `SessionTimeoutPlug`

* Rely on user session assign from `AuthContext` in `SentryContext`

* Silence legacy session warnings in `UserSessionTouchTest`

* Rely on session assign from `AuthPlug` in `SuperAdminOnlyPlug`

* Change `UserAuth` to get session, user and last subscription in one go

* Avoid refetching user session in `AuthorizeSiteAccess` plug

* Fix code formatting

* Refactor `UserAuth.get_user_token/1` (h/t @aerosol)

* Remove bogus empty opts from `scope` declarations in router

* Only touch session once an hour and keep `user.last_seen` in sync

* Bring back logging of legacy token use
2024-09-03 11:34:37 +02:00
Joris Drenth
49ff00252e
Fix documentation link for Google integration (#4472)
* Fix documentation link for Google integration

* Fix documentation link for Google integration (test update)

* Update lib/plausible_web/templates/site/settings_search_console.html.heex

---------

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-02 21:19:43 +02:00
RobertJoonas
5e26e2c97d
Fix realtime periods returning wrong date range for non-UTC timezone sites (#4508)
* fix realtime

* mix format
2024-09-02 19:21:17 +03:00
Artur Pata
9b59e95c33
Support order_by in URL params of breakdown endpoints (#4502)
* Support order_by in URL params of breakdown endpoints
2024-09-02 15:00:18 +03:00
hq1
e3af1a317d
Onboarding improvements (#4459)
* Migration: add installation meta

* Update site schema with installation meta

* Remove VERIFICATION_ENABLED env var

* Add context API to create/remove special goals

* Add context api to update installation meta

* Remove verification enabled check

* Update new progress flow definitions

* Update generic components

* Remove internal /status API

* Implement installation live view

* Update traffic change notifier link

* Update verification, no more modal

* Update routes

* Remove focus.html - will unify everything under app layout

* Fix broken link

* Update templates with focus_box mostly

* Update controller tests

* Update controllers and stop using the focus layout

* copy changes

* Update verification.ex

* Remove dead template

* Update settings_general.html.heex

* Update copy in tests

* Update installation.ex

* Remove dangling dot

* Fix link

* Update installation.ex

* Update installation.ex

* Better tooltips?

* Simpler labels

* Revert "Simpler labels"

This reverts commit 797560ef82.

* Add copy to clipboard link and fix snippet's dark mode

* Offer installation detection skip only if ws connected

* Put COPY link at the bottom with background

* Make tooltips link to docs

* Fix cherry-pick gone wrong

* Hide tooltips on mobile screens

* WIP: 404 tracking wizard

* Revert "WIP: 404 tracking wizard"

This reverts commit a9c9c79bbd.

* Update lib/plausible_web/live/components/verification.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update lib/plausible_web/live/installation.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Use current_user from socket.assigns

* Update lib/plausible_web/live/installation.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Use current_user from socket.assigns

* Use conn.private to steer verification tests

* Drop non-sticky tooltip in favour of component parametrization

Co-authored-by: Artur Pata <artur.pata@gmail.com>

* Reapply "WIP: 404 tracking wizard"

This reverts commit 3ba81671d7.

* Fix installation tests including 404 tracking

* Fixup the tooltip component

* Format

* Update installation.ex

* Put flash whenever installation option changes

* Use last known installation type on domain change

* Extract user flow definition to provide compile-time checks

* See if this helps running CE migrations successfully

* Use `styled_link` on registration/login views

* Don't crash when there's no conn.private carried over

* Format

* Push "Determining installation type" message a bit lower

* Use links and footer lists uniformly

This commit introduces a `<.focus_list/>` component
for rendering focus box footer links with colored
discs. It also equips generic link components
with the ability of sending non-GET requests
along with CSRF token, so we can apply uniform
styling and stop using legacy Phoenix link tags.

cc @zoldar @apata

* ws 👾

* Render more descriptive flashes on script config change

---------

Co-authored-by: Marko Saric <34340819+metmarkosaric@users.noreply.github.com>
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
Co-authored-by: Artur Pata <artur.pata@gmail.com>
2024-09-02 12:49:54 +02:00
RobertJoonas
f04c47f881
Support realtime periods in API v2 (#4469)
* add realtime date_ranges into the private API schema

This commit starts parsing date ranges into a new NaiveDateTimeRange
struct, rather than a simple Date.Range.

* transform realtime labels into negative integers + test

* move schema type argument to last position in helper functions

* allow passing a date param + tests

* Update test/plausible/stats/query_parser_test.exs

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* Update test/plausible/stats/query_parser_test.exs

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* Update test/plausible/stats/query_parser_test.exs

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* Update test/plausible/stats/query_parser_test.exs

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* keep test file structure consistent

* Turn NaiveDateTimeRange into DateTimeRange

* change 'now' field from NaiveDateTime to DateTime in v2 query

* fix minute interval labels + add missing tests

* return query_result.date_range as iso8601 timestamps with timezone

* allow timestamps with tz as date_range arguments in API v2

* delete Plausible.Timezones.to_utc_datetime

* simplify returning comparison periods

* add comment about realtime not supported in comparisons

* pass only now instead of test_opts

* drop redundant else branch

* separate tests

* stick to a single check_date_range function in tests

* fix credo error

---------

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
2024-09-02 12:56:58 +03:00
Adrian Gruntkowski
e9dd895d6c
Do not try querying imported data for unsupported props (#4467)
* Do not try querying imported data for unsupported props

* Remove unnecessary `unquote`s

Co-authored-by: hq1 <hq@mtod.org>

* Add respective regression test to APIv2 tests

---------

Co-authored-by: hq1 <hq@mtod.org>
2024-08-30 10:55:31 +02:00
Uku Taht
2e89a50188
Map lowercase tagged sources to capitalized form during ingestion (#4417)
* Map lowercase tagged sources to capitalized form during ingestion

* @moduledoc

* Add changlog entry
2024-08-27 14:03:15 +03:00
Karl-Aksel Puulmann
9c71161eab
APIv2: JSON schema validation, separate internal and public API validation (#4464)
* Restore `date` internal parameter, validate via json schema

* Improved error formatting from json schema, get most tests passing

* Handle internal overrides to JSON schema

* Parsing tests all pass

* Remove some repeated code, enforce length/uniqueness in schema

* Explicit separation between internal and public API validation

* Mark file as external_resource

* map_join

* Update query tests

* Update query tests

* Serve schema under an /api/docs/query/schema.json endpoint

* dotify errors
2024-08-26 14:01:27 +03:00
Adrian Gruntkowski
bd93cf3b46
Refactor and consolidate user session logic (#4452)
* Extract session management from AuthController

* Don't explicitly pass `current_user_id` to `live_render`'s session

* Add ability to retrieve session and user from token via `UserAuth`

* Always fetch current user (or just id) via `UserAuth` API

* Introduce `UserSession` as an embedded schema for now

* Make `UserAuth.get_user/1` accept `UserSession` as an input

* Introduce LV auth context populating user data from session on mount

* Refactor `AuthPlug` and make it populate `current_user_session` as well

* Rely on authenticated user data provided by auth plug or LV context

* Make `Sites.get_for_user(!)` accept `User` struct as well

* Set `logged_in` cookie explicitly when it's out of sync with session

* Expand modules documentation a bit

* Improve and extend tests slightly
2024-08-23 10:53:33 +02:00
RobertJoonas
039790b82d
Prepare for graph v2 (#4451)
* add the ability to pass date param into Query.build

* stop returning interval from main_graph controller action

* globally rename 'date' interval to 'day'

* Allow parsing query date range from a 'period' param

The 'period' param will not be exposed in the public API, but makes it
possible to construct a "realtime" query.

* Revert "Allow parsing query date range from a 'period' param"

This reverts commit c5630eaef9.

* call beginning_of_time for first_datetime instead of last

* allow 'realtime' and '30m' date_range shortcuts

* evaluate date_range to 'realtime' or '30m' instead of Date.range(today, today)

* Revert "evaluate date_range to 'realtime' or '30m' instead of Date.range(today, today)"

This reverts commit a887569ec5.

* Revert "allow 'realtime' and '30m' date_range shortcuts"

This reverts commit 91ae0fa5e6.

* fix graph tooltips
2024-08-22 14:33:05 +03:00
hq1
572f8abac5
First pass: unify onboarding UI (#4445)
* First pass: unify onboarding UI

* Dark mode fixes

* Format

* Update self-hosted index

* Remove unnecessary child box in Enter Your 2FA screen

* Fixup Enter Your 2FA screen
2024-08-19 09:10:54 +02:00
Adrian Gruntkowski
a20cb39652
Unify and refactor login regardless of trigger source (explicit/register) (#4434)
* Unify and refactor login regardless of trigger source (explicit or registration)

* Fix code formatting
2024-08-16 10:59:31 +02:00
hq1
0440d36577
Implement new onboarding/provisioning flow progress layout (#4441)
* Implement new onboarding/provisioning flow progress layout

* Delete dead template

* Fixup & test FlowProgress component
2024-08-16 10:07:48 +02:00
Karl-Aksel Puulmann
ee3d1e770e
APIv2: visit:country_name, visit:region_name, visit:city_name dimensions (#4328)
* Add data migration for creating and syncing location_data table and dictionary

* Migration to populate location data

* Daily cron to refresh location dataset if changed

* Add support for visit:country_name, visit:region_name and visit:city_name dimensions

Under the hood this relies on a `location_data` table in clickhouse being regularly synced with
plausible/location repo and dictionary lookups used in ALIAS columns

* Update queue name

* Update documentation

* Explicit structs

* Improve docs further

* Migration comment

* Add queues

* Add error when already loaded

* Test for filtering by new dimensions

* Update deps

* dimension -> select_dimension

* Update a test
2024-08-13 09:44:58 +03:00
Adrian Gruntkowski
4c5ce0f1fe
Fix flaky auth rate limit tests and refactor auth rate limiting (#4401)
* Add safeguard against flaky auth rate limit tests

* Fix typo in a log message

* Extract and abstract rate limiting from `AuthController`

* Fix flaky rate limit test tag

* Don't leak prefix from auth rate limit

* Use more compact map syntax

* Remove special tag in favor of `eventually` test util function
2024-08-09 14:58:34 +02:00
hq1
cc769dfb3d
Edit goals with display names (#4415)
* Update Goal schema

* Equip ComboBox with the ability of JS selection callbacks

* Update factory so display_name is always present

* Extend Goals context interface

* Update seeds

Also farming unsuspecting BEAM programmers for better
sample page paths :)

* Update ComboBox test

* Unify error message color class with helpers seen elsewhere

* Use goal.display_name where applicable

* Implement LiveView extensions for editing goals

* Sprinkle display name in external stats controller tests

* Format

* Fix goal list mobile view

* Update lib/plausible_web/live/goal_settings/list.ex

Co-authored-by: Artur Pata <artur.pata@gmail.com>

* Update lib/plausible_web/live/goal_settings/form.ex

Co-authored-by: Artur Pata <artur.pata@gmail.com>

* Update the APIs: plugins and external

* Update test so the intent is clearer

* Format

* Update CHANGELOG

* Simplify form tabs tests

* Revert "Format"

This reverts commit c1647b5307.

* Fixup format commit that went too far

* ComboBox: select the input contents on first focus

* Update lib/plausible/goal/schema.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update lib/plausible/goals/goals.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update lib/plausible_web/live/goal_settings/form.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Pass form goal instead of just ID

* Make tab component dumber

* Extract separate render functions for edit and create forms

* Update test to account for extracted forms

* Inline goal get query

* Extract revenue goal settings to a component and avoid computing assigns in flight

* Make LV modal preload optional

* Disable preload for goal settings form modal

* Get rid of phash component ID hack

* For another render after render_submit when testing goal updates

* Fix LV preload option

* Enable preload back for goals modal for now

* Make formatter happy

* Implement support for preopening of LV modal

* Preopen goals modal to avoid feedback gap on loading edited goal

* Remove `console.log` call from modal JS

* Clean up display name input IDs

* Make revenue settings functional on first edit again

* Display names: 2nd stage migration

* Update migration with data backfill

---------

Co-authored-by: Artur Pata <artur.pata@gmail.com>
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-08-09 11:12:00 +02:00
RobertJoonas
1f8662438d
Add Details modals for all Devices reports (#4419)
* return concatenated browser names and versions directly from the API

* return concatenated os names and versions directly from the API

* add detailed views for all device reports

* extract put_combined_name_with_version function

* return only version under the name key when detailed=true

* update changelog

* add more metrics into detailed views of Devices reports

* split up different devices modals into separate files
2024-08-02 15:50:22 +03:00
Uku Taht
a1ee58426e
Implement goal contains filter + search in Conversions > Details (#4404)
* Implement goal contains

* add search into ConversionsModal

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
2024-07-31 10:21:30 +03:00
Uku Taht
05b70e37a8
Generate goal filter SQL from postgres goals (#4391)
* Use goals from postgres to build goal filter

* Remove unnecessary goal preloading

* Add contains filter for goals

* Make sure the correct imported tables are used when goals are in filters

* Remove 'contains' filter type for now

* Remove TODO comment

* Fix QueryParser test

* move goals.ex into a goals subfolder

* extract goal filtering logic into a separate module

* remove duplication from Imported.Base

* Extract `get_filter_goals` function

* Credo suggestions

* Credo ignore nesting

* apply suggestion - remove Sql.Util

* apply suggestion - pass imported? via opts

* remove duplicate function

* Uncomment tests

* Fix 500 error with goal suggestions

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
2024-07-30 12:07:23 +03:00
Adrian Gruntkowski
556c896936
Add ability to lookup by email and site domain in HelpScout integration (#4377)
* Add ability to lookup by email and site domain in HelpScout integration

* Handle callback with invalid parameters gracefully

* Add routine adjusting iframe height to the content of HS integration

* Add requestStorageAccess routine to support iframe cookies in Safari

* Fill first email pick when on failed user lookup

* Fix a typo in a comment
2024-07-26 16:26:51 +02:00
Uku Taht
49bb57f601
Fix email report sentry errors (#4381)
* Fix missing email param on email unsubscribe

* Guard against email report being deleted
2024-07-23 12:05:42 +03:00
RobertJoonas
2f87832532
Implement search and pagination in Google Keywords modal (#4378)
* Extend the GSC API with search functionality

* Fix typo in error tuple atom

* log GSC errors

* rename confusing variable name

* Fix the API response format and error handling

* Read results under the `results` key to be consistent with other
  endpoints

* Return errors with a non-200 status code, and with an error payload
  that will be well constructed into ApiError

* rebuild Google Keywords modal with useAPIClient

* Add pagination support in Search Terms API

* delete unused fixture file

* rename fixture files

* fix tests
2024-07-23 11:53:52 +03:00
Adrian Gruntkowski
0310cecef8
Add listing sites, goals and custom props to Sites API (#4302)
* Add listing sites, goals and custom props to Sites API

* Rename exmaple props in tests

* Rename `allowed_custom_props` -> `custom_properties`

* Expose goal name in GET endpoints for goals in Sites API

* Bump default pagination limit to 100 and max to 1000

* Introduce Goal.name/display_name and use the first one for name in API

* Extend goal list response but hide currency

* Settle on `display_name` instead of `name`

* Allow viewer members to get site details and list site goals

* Don't include currency in goal's display name
2024-07-19 10:05:54 +02:00
Uku Taht
28787198b2
Fix present_index for daily, weekly and monthly intervals (#4358)
* Fix present_index for daily, weekly and monthly intervals

* Add test for present_index
2024-07-16 12:12:38 +03:00
Adrian Gruntkowski
1c5c4a25aa
HelpScout integration (#4327)
* Implement basic HelpScout integration

* Set 127.0.0.1 as a default customer IP in `Plans.with_prices/2`

* Use `is_nil/1` instead of `... == nil` (h/t @aerosol)

* Use `Path.join/1,2` to build API URLs a bit more safely (h/t @aerosol)

* Check for locked sites entirely within query logic

* Move conditional start of HelpScout vault to compile-time

* Include customer_id in error sent to Sentry

* Use `Plug.Crypto.secure_compare/2` for constant-time signature comparison

* Refactor token request function

* Use `Path.join/1` in one more spot

* Use route helper to build CRM URL
2024-07-12 12:01:59 +02:00
hq1
d56bb2b4d5
Implement traffic drop notifications (#4300)
* Expose current visitors 12h aggregate

* Remove unused site association

* Distinct drop/spike notification factories

* Rename modules accordingly + implement drop handling

* Rename periodic oban service

* Implement drop email

* Rest of the owl

* Update changelog

* Update moduledoc

* Update moduledoc

* Min threshold to 1

* Threshold 1

* Remove merge artifact

* Put panel behind a feature flag

* Format
2024-07-11 14:55:18 +02:00
Karl-Aksel Puulmann
0114689779
fix(TimeSeries): Work around weird time zones (#4337)
* Refactor Expression.dimension to accept q

* Handle quarter- and half-hour timezones

Previously APIv2 output didn't start at a full hour for these time zones
and main graph was blank

The core reasoning is that ClickHouse `timeSlots` is not time-zone
aware and works off of unix epoch - meaning that in time zones which
have an offset of 5:45 the "hours" reported would start at minute :45.

The fix is kind of silly - we now divide each hour into 4 and handle
things that way.

Related basecamp issue: https://3.basecamp.com/5308029/buckets/36789884/card_tables/cards/7590936581

* Fix test typo
2024-07-11 11:48:16 +03:00
Karl-Aksel Puulmann
645b81376c
fix(main-graph): fix 12mo and other comparisons with blanks (#4333)
Some comparisons return __blank__ values. After recent time series fix,
this blew up when trying to parse __blank__ as a date.

This fixes the issue and adds a test for a period with __blank__ values

Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
2024-07-10 15:46:31 +03:00
Karl-Aksel Puulmann
4d19603693
fix: Calculate revenue metrics correctly when including imports (#4335)
Previously, revenue metrics were reported as 0.0$ when imports were included with
APIv2. This is because the metrics were not properly included/forwarded by the
imports query.

Not sure how long this bug has persisted, but eh.
2024-07-10 15:10:29 +03:00
Karl-Aksel Puulmann
ecc882dc22
APIv2: Bugfixes to location filtering (#4321)
* Fix filter ["contains", "visit:country", ["E"]]

This blew up since FixedString columns dont natively support matching functions for some reason

* Dont blow up for 3-letter country codes, instead fail validation
2024-07-10 11:34:24 +03:00
Karl-Aksel Puulmann
7aff0760aa
APIv2: Allow sending numbers for visit:city along with strings (#4313)
* Allow sending numbers for visit:city along with strings

This simplifies querying as we pass the list of cities as numbers to clients

* use &is_integer/1

* Remove redundancy
2024-07-10 11:29:21 +03:00
RobertJoonas
7d0321fd22
Implement search in Details views (#4318)
* Create a new BreakdownModal component and use it for Entry Pages

* Add search functionality into the new component

* Adjust FilterLink component and use it in BreakdownModal

* pass addSearchFilter fn through props

* pass fn props as useCallback

* add a function doc to BreakdownModal

* refactor: create a Metric class

* Fixup: use Metric class for defining BreakdownModal metrics

* keep revenueAvailable state in the Dashboard component

* move query context into a higher-order component

* fix react key error in BreakdownModal

* use BreakdownModal in PropsModal

* adjust EntryPagesModal to use query context

* fix variable name typo

* fixup: BreakdownModal function doc

* use BreakdownModal in SourcesModal

* use Breakdown modal in ReferrerDrilldownModal

* use BreakdownModal in PagesModal

* use BreakdownModal in ExitPagesModal

* replace ModalTable with LocationsModal and use BreakdownModal in it

* use BreakdownModal in Conversions

* make sure next pages are loaded with 'detailed: true'

* replace loading spinner logic in BreakdownModal

* fix two flaky tests

* unfocus search input element on Escape keyup event

* ignore Escape keyup handling when search disabled

* Review suggestion: remove redundant state

* do not fetch data on every search input change

* use longer variable names

* do not define renderIcon callbacks conditionally

* deconstruct props in function header of BreakdownModal

* refactor searchEnabled being true by default
2024-07-09 15:01:52 +03:00
Karl-Aksel Puulmann
a9676546dc
APIv2: Cleanup (#4308)
* Remove a dead method

* Move select_event/session_metrics to within QueryBuilder

* Make a method private

* Move page_regex util around

* Move utc_boundaries helper around

* Fixups for utc_boundaries

* Add legacy notices

* Move Stats.query method around

* include_sentry_replay_info consistently

* Move filters out of select_group_fields

* Collapse conditions under select_group_fields

* Shorten some imported methods

* Use dimension over dim in group_by

* Separate SQL query building from imported.ex

* props.ex -> legacy_dimensions.ex

* Move some query building out of Query.ex

* Remove unneeded method

* put_filter -> add_filter

* Remove some query setters

* Moduledoc

* Split out validations and import tests from query_test

* Move tests around

* Split event:goal tests from query_test

* Remove redundant filters

* Remove dead code

* Split special metrics tests from query_test

* Legacy module
2024-07-09 14:31:45 +03:00
Karl-Aksel Puulmann
a181f3eab3
APIv2: TimeSeries using QueryBuilder, release experimental_session_count (#4305)
* Move fragments module under Plausible.Stats.SQL

* Introduce select_merge_as macro

This simplifies some select_merge calls

* Simplify select_join_fields

* Remove a needless dynamic

* wrap_select_columns macro

* Move metrics from base.ex to expression.ex

* Move WhereBuilder under Plausible.Stats.SQL

* Moduledoc

* Improved macros

* Wrap more code

* select_merge_as more

* Move defp to the end

* include.time_labels parsing

* include.time_labels in result

Note that the previous implementation of the labels from TimeSeries.ex was broken

* Apply consistent function in imports and timeseries.ex

* Remove boilerplate

* WIP: Limited support for timeseries-with-querybuilder

* time:week dimension

* cleanup: property -> dimension

* Make querying with time series work

* Refactor: Move special metrics (percentage, conversion rate) to own module

* Explicitly format datetimes

* Consistent include_imported in special metrics

* Solve week-related crash

* conversion_rate hacking

* Keep include_imported consistent after splitting the query

* Simplify do_decide_tables

* Handle time dimensions in imports cleaner

* Allow time dimensions in custom property queries

* time:week handling continued

* cast_revenue_metrics_to_money

* fix `full_intervals` support

* Handle minute/realtime graphs

* experimental_session_count? with timeseries

This becomes required as we try to include visits from sessions by default

* Support hourly data in imports

* Update bounce_rate in more csv tests

* Update some time-series query tests

* Fix for meta.warning being included incorrectly

* Simplify imported.ex

* experimental_session_count flag removal

* moduledoc

* Split interval and time modules
2024-07-09 14:25:02 +03:00
Adrian Gruntkowski
35596e8692
Remove Universal Analytics import logic (#4312) 2024-07-05 13:58:42 +02:00
Adrian Gruntkowski
2f2602e316
Stop listing UA views in Google data import flow (#4309)
* Stop listing UA views in Google data import flow

* Update CHANGELOG

* Trigger preview deploy
2024-07-04 13:08:32 +02:00
Adrian Gruntkowski
790984e1ad
Refactor Sites and Stats API authorization logic (#4297)
* Refactor and unify auth plugs for Stats and Sites APIs

* Expose get site Sites API endpoint to all API keys

* Test the new plug

* Add test for endpoint with modified scope

* Fix typos

Co-authored-by: hq1 <hq@mtod.org>

* Rename plug for consistency (h/t @aerosol)

---------

Co-authored-by: hq1 <hq@mtod.org>
2024-07-02 15:09:23 +02:00
Karl-Aksel Puulmann
0594478add
APIv2: Replace breakdown module with QueryBuilder (#4293)
* Revert "Revert "APIv2: Replace breakdown module with QueryBuilder (#4283)" (#4292)"

This reverts commit ef5e0e0382.

* Allow querying events and pageviews from sessions table

This is not strictly accurate, especially with shorter time frames, but
is useful for a fallback mechanism. I'll figure out something around
shorter time frames in the future.

See also: https://github.com/plausible/analytics/pull/4292

* Only query events and pageviews in legacy breakdowns
2024-07-01 12:50:01 +03:00
Karl-Aksel Puulmann
ef5e0e0382
Revert "APIv2: Replace breakdown module with QueryBuilder (#4283)" (#4292)
This reverts commit 7dd12d1dd6.
2024-07-01 10:50:44 +03:00
Karl-Aksel Puulmann
7dd12d1dd6
APIv2: Replace breakdown module with QueryBuilder (#4283)
* WIP: Breakdown using QueryBuilder

* Revert "Remove problematic test"

This reverts commit b442bb5d1f.

* Get more breakdown tests passing

* Preload goals, sort when dealing with time_on_page

* Handle conversion_rate in breakdowns

* Simplify ordering by using selected_as consistently for dimensions

* Get breakdown tests passing

* Strings to atoms in keys for StatsController.transform_keys calls to work

* Handle revenue metrics removal

* Add test for nil-removal case

* Include percentage metric

* Fix and test with imported locations

* Fixup time-on-page

* Fix country/region automatic filters

* Handle multiple imports (os/browser version) in importsv2

* Filter goals

* Default to ordering by page as well

* Calculate conversion rate on sessions if needed

* Order by event dimensions - handles event:page special case

* Update tests

* Update more tests, handle goal=0 case in imports

* Handle event:goal breakdowns correctly with filters

* Revenue to money

* Improved table deciding

* Also update event:page filters on event:page breakdown

* bounce_rate to 0

Previous behavior relied on two queries being made - new query leads to 0 naturally

* Update pagination test

* dont count non-pageviews as path goal completions

* Make revenue logic breakdown-specific

Its hard to fit into the new schema and likely needs a rethink for apiv2

* Retain previous behavior for TimeSeries module

* Get GA4 test passing

Most failures are related to ordering, pageviews shouldnt be read off of sessions

* Clean up old methods

* Simplify imported.ex

* Dont crash on garbage filters

* Reflect ordering-related change in test

* Fix test data

* Update table_decider

* Re-simplify get_revenue_tracking_currency

* Revert revenue changes

* Use Query.set

* Remove a TODO

* csv importer: no pageviews

Pageviews were incorrectly fetched from sessions table before, causing issues

* csv importer tweaking

* Remove use Plausible

* to_existing_atom
2024-07-01 09:03:33 +03:00
Karl-Aksel Puulmann
2eeaf7a152
APIv2: Aggregates, timeseries, conversion_rate, hostname (#4251)
* Add some aggregates tests

* Port aggregates tests to do with filtering

* Session metrics can be queried with event: filters

* Solve a typo

* Update a validation message

* Add validations for views_per_visit

* Port an aggregation/imports test

* Optimize time dimension, add tests

* Add first timeseries test, update parsing tests

* Docs for SQL.Expression

* Test timeseries more

* Allow time explicitly in order_by

* Add multiple breakdowns test

* Refactor QueryOptimizer not to care about time dimension placement in dimensions array

* Add test breaking down by event:hostname

* Add hostname filtering logic to QueryOptimizer, unblock some tests

* WIP: Breakdown by goal

* conversion rate logic for query api

* Update more tests

* Set default order_by

* dimension_label

* preloaded_goals in tests

* inline load_goals

* Use Date functions over Timex

* Comments

* is_binary

* Remove special form used in tests

* Fix defmodule

* WIP: Fix memory leak, event:page breakdown logic

* Enable more tests, fix for group_conversion_rate without explicit visitors metric

* Re-enable a partially commented test

* Re-enable a partially commented test

* Get last test passing

* No imports order_by in apiv2

* Add a TODO

* Remove redundant Util call

* Update aggregate.ex

* Remove problematic test
2024-06-28 08:59:54 +03:00
Karl-Aksel Puulmann
58a66a952c
APIv2 - initial PR (#4216)
* WIP new querying

* WIP: Move some aggregate code under new command

* WIP: Add joins, handling less metrics

* join events table to sessions if needed

* Merge imported results with built query

* Remove dead code

* WIP: /api/v2/query

* Allow grouping by time

* Use JOIN for main query

* Build query result

* update parse_time

* Make joinless order by work

* First test

* more breakdown tests

* Serialize event:goal filters in an json-encodable way/reflection

* Handle inner vs outer ORDER BY clauses properly

* Handle single conversion_rate metric

* Update more tests

* Get parsing tests passing again

* Validate filtered goal filter is configured

* Enable more validation tests

* Enable more event:name breakdown tests

* Enable more breakdown tests

* Validate site has access to custom props

* Validate conversion_rate metric which is only allowed in some situations

* Validate that empty event:props: is not valid

* handle query.dimensions properly in table_decider

* test more validations on metrics/dimensions

* Validate session metrics in combination with event dimension(s)

* Tests cleanup

* Parse include.imports

* Get imports working with new querying

* Make more imports tests work

* Make event:props:path imports-adjacent test work

* Get query imports warning-related tests running

* Remove dead pagination tests

* Solve dead import

* Solve some warnings

* Update aggregate metrics tests

* credo

* Improve test naming

* Lazy goal loading

* Use datetime methods

* Ecto -> SQL module name

* Remove Expression.dimension mode option
2024-06-25 09:27:19 +03:00