Commit Graph

1130 Commits

Author SHA1 Message Date
Uku Taht
05b70e37a8
Generate goal filter SQL from postgres goals (#4391)
* Use goals from postgres to build goal filter

* Remove unnecessary goal preloading

* Add contains filter for goals

* Make sure the correct imported tables are used when goals are in filters

* Remove 'contains' filter type for now

* Remove TODO comment

* Fix QueryParser test

* move goals.ex into a goals subfolder

* extract goal filtering logic into a separate module

* remove duplication from Imported.Base

* Extract `get_filter_goals` function

* Credo suggestions

* Credo ignore nesting

* apply suggestion - remove Sql.Util

* apply suggestion - pass imported? via opts

* remove duplicate function

* Uncomment tests

* Fix 500 error with goal suggestions

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
2024-07-30 12:07:23 +03:00
Adrian Gruntkowski
783740468d
Update ua_inspector fork to sanitize multi-part prereleases properly (#4398)
* Update `ua_inspector` fork to sanitize multi-part prereleases properly

* Add regression tests

* Make another version bump to include a fix for a missed corner case
2024-07-29 11:10:27 +02:00
Adrian Gruntkowski
556c896936
Add ability to lookup by email and site domain in HelpScout integration (#4377)
* Add ability to lookup by email and site domain in HelpScout integration

* Handle callback with invalid parameters gracefully

* Add routine adjusting iframe height to the content of HS integration

* Add requestStorageAccess routine to support iframe cookies in Safari

* Fill first email pick when on failed user lookup

* Fix a typo in a comment
2024-07-26 16:26:51 +02:00
RobertJoonas
37014c86a8
Tracking improvements (#4392)
* track browser_language, theme, and logged_in for 'Filter Menu: Open'

* track props for 'Signup' and 'Signup via invitation'

* do not track theme as 'system' for non-logged-in users

* fix tests
2024-07-26 15:24:55 +02:00
Adrian Gruntkowski
590785794f
Add search_query to internal props (#4390) 2024-07-26 09:40:23 +02:00
Uku Taht
49bb57f601
Fix email report sentry errors (#4381)
* Fix missing email param on email unsubscribe

* Guard against email report being deleted
2024-07-23 12:05:42 +03:00
RobertJoonas
2f87832532
Implement search and pagination in Google Keywords modal (#4378)
* Extend the GSC API with search functionality

* Fix typo in error tuple atom

* log GSC errors

* rename confusing variable name

* Fix the API response format and error handling

* Read results under the `results` key to be consistent with other
  endpoints

* Return errors with a non-200 status code, and with an error payload
  that will be well constructed into ApiError

* rebuild Google Keywords modal with useAPIClient

* Add pagination support in Search Terms API

* delete unused fixture file

* rename fixture files

* fix tests
2024-07-23 11:53:52 +03:00
Adrian Gruntkowski
0310cecef8
Add listing sites, goals and custom props to Sites API (#4302)
* Add listing sites, goals and custom props to Sites API

* Rename exmaple props in tests

* Rename `allowed_custom_props` -> `custom_properties`

* Expose goal name in GET endpoints for goals in Sites API

* Bump default pagination limit to 100 and max to 1000

* Introduce Goal.name/display_name and use the first one for name in API

* Extend goal list response but hide currency

* Settle on `display_name` instead of `name`

* Allow viewer members to get site details and list site goals

* Don't include currency in goal's display name
2024-07-19 10:05:54 +02:00
hq1
1488683d26
Implement UI to edit existing funnels (#4369)
* Update tailwind config

Turns out `extra` wasn't scanned properly with
previous wildcards. I don't care any more, it's not slow.

* Add interface for updating funnels

* Extend ComboBox with the ability to preselect an initial value

* Implement editing funnels UI

* Update CHANGELOG

* s/add_funnel/setup_funnel

* modal width 2/5 => 2/3

* Let's not make the list disappear on modal pop-up

* Fix the damn modal width again

* Watch extra dir

* Format

* Remove commented code

The test is implemented elsewhere

* Track funnel modified to drop default selection

* Fix screen size adoption and format large numbers

* Preserve currency so that string casting includes it

* Format

* Fix ComboBox attribute for preselected option
2024-07-17 22:04:36 +02:00
Adrian Gruntkowski
1da1da9421
Fix flaky tests in Plausible.Auth.TOTPTest (#4362) 2024-07-16 16:50:52 +02:00
Uku Taht
28787198b2
Fix present_index for daily, weekly and monthly intervals (#4358)
* Fix present_index for daily, weekly and monthly intervals

* Add test for present_index
2024-07-16 12:12:38 +03:00
Adrian Gruntkowski
7015075e12
Show number of owned sites in HS integration with a link to sites list in CRM (#4355) 2024-07-16 09:18:34 +02:00
ruslandoga
b988364203
Improve PG unix sockets configuration (#4202)
* improve pg unix sockets configuration

* changelog

* don't use test env in config tests
2024-07-15 13:05:31 +02:00
ruslandoga
14d5a8028f
Don't require TOTP vault key in CE (#4317) 2024-07-15 12:02:31 +02:00
ruslandoga
530951780c
Fix relay-without-auth MUA config (#4298) 2024-07-15 12:01:57 +02:00
Adrian Gruntkowski
1c5c4a25aa
HelpScout integration (#4327)
* Implement basic HelpScout integration

* Set 127.0.0.1 as a default customer IP in `Plans.with_prices/2`

* Use `is_nil/1` instead of `... == nil` (h/t @aerosol)

* Use `Path.join/1,2` to build API URLs a bit more safely (h/t @aerosol)

* Check for locked sites entirely within query logic

* Move conditional start of HelpScout vault to compile-time

* Include customer_id in error sent to Sentry

* Use `Plug.Crypto.secure_compare/2` for constant-time signature comparison

* Refactor token request function

* Use `Path.join/1` in one more spot

* Use route helper to build CRM URL
2024-07-12 12:01:59 +02:00
hq1
d565175c13
Remove verification related feature flags (#4345)
- `:verification` flag is no longer needed,
   however we'll keep the old "awaiting first pageview" flow
   for Community Edition where verification is not available
   by default
 - `:traffic_drop_notifications` have been tested and should
   be enabled for everyone
2024-07-12 09:43:54 +02:00
hq1
a2497f6f8c
Open verification modal when following drop notification link (#4344)
* Open verification modal when following drop notification link

* Add test

* Fixup

* Move `open-modal` event to Components.Modal API
2024-07-12 06:26:40 +02:00
hq1
d56bb2b4d5
Implement traffic drop notifications (#4300)
* Expose current visitors 12h aggregate

* Remove unused site association

* Distinct drop/spike notification factories

* Rename modules accordingly + implement drop handling

* Rename periodic oban service

* Implement drop email

* Rest of the owl

* Update changelog

* Update moduledoc

* Update moduledoc

* Min threshold to 1

* Threshold 1

* Remove merge artifact

* Put panel behind a feature flag

* Format
2024-07-11 14:55:18 +02:00
Karl-Aksel Puulmann
0114689779
fix(TimeSeries): Work around weird time zones (#4337)
* Refactor Expression.dimension to accept q

* Handle quarter- and half-hour timezones

Previously APIv2 output didn't start at a full hour for these time zones
and main graph was blank

The core reasoning is that ClickHouse `timeSlots` is not time-zone
aware and works off of unix epoch - meaning that in time zones which
have an offset of 5:45 the "hours" reported would start at minute :45.

The fix is kind of silly - we now divide each hour into 4 and handle
things that way.

Related basecamp issue: https://3.basecamp.com/5308029/buckets/36789884/card_tables/cards/7590936581

* Fix test typo
2024-07-11 11:48:16 +03:00
Karl-Aksel Puulmann
645b81376c
fix(main-graph): fix 12mo and other comparisons with blanks (#4333)
Some comparisons return __blank__ values. After recent time series fix,
this blew up when trying to parse __blank__ as a date.

This fixes the issue and adds a test for a period with __blank__ values

Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
2024-07-10 15:46:31 +03:00
Karl-Aksel Puulmann
4d19603693
fix: Calculate revenue metrics correctly when including imports (#4335)
Previously, revenue metrics were reported as 0.0$ when imports were included with
APIv2. This is because the metrics were not properly included/forwarded by the
imports query.

Not sure how long this bug has persisted, but eh.
2024-07-10 15:10:29 +03:00
Karl-Aksel Puulmann
ecc882dc22
APIv2: Bugfixes to location filtering (#4321)
* Fix filter ["contains", "visit:country", ["E"]]

This blew up since FixedString columns dont natively support matching functions for some reason

* Dont blow up for 3-letter country codes, instead fail validation
2024-07-10 11:34:24 +03:00
Karl-Aksel Puulmann
7aff0760aa
APIv2: Allow sending numbers for visit:city along with strings (#4313)
* Allow sending numbers for visit:city along with strings

This simplifies querying as we pass the list of cities as numbers to clients

* use &is_integer/1

* Remove redundancy
2024-07-10 11:29:21 +03:00
RobertJoonas
7d0321fd22
Implement search in Details views (#4318)
* Create a new BreakdownModal component and use it for Entry Pages

* Add search functionality into the new component

* Adjust FilterLink component and use it in BreakdownModal

* pass addSearchFilter fn through props

* pass fn props as useCallback

* add a function doc to BreakdownModal

* refactor: create a Metric class

* Fixup: use Metric class for defining BreakdownModal metrics

* keep revenueAvailable state in the Dashboard component

* move query context into a higher-order component

* fix react key error in BreakdownModal

* use BreakdownModal in PropsModal

* adjust EntryPagesModal to use query context

* fix variable name typo

* fixup: BreakdownModal function doc

* use BreakdownModal in SourcesModal

* use Breakdown modal in ReferrerDrilldownModal

* use BreakdownModal in PagesModal

* use BreakdownModal in ExitPagesModal

* replace ModalTable with LocationsModal and use BreakdownModal in it

* use BreakdownModal in Conversions

* make sure next pages are loaded with 'detailed: true'

* replace loading spinner logic in BreakdownModal

* fix two flaky tests

* unfocus search input element on Escape keyup event

* ignore Escape keyup handling when search disabled

* Review suggestion: remove redundant state

* do not fetch data on every search input change

* use longer variable names

* do not define renderIcon callbacks conditionally

* deconstruct props in function header of BreakdownModal

* refactor searchEnabled being true by default
2024-07-09 15:01:52 +03:00
Karl-Aksel Puulmann
a9676546dc
APIv2: Cleanup (#4308)
* Remove a dead method

* Move select_event/session_metrics to within QueryBuilder

* Make a method private

* Move page_regex util around

* Move utc_boundaries helper around

* Fixups for utc_boundaries

* Add legacy notices

* Move Stats.query method around

* include_sentry_replay_info consistently

* Move filters out of select_group_fields

* Collapse conditions under select_group_fields

* Shorten some imported methods

* Use dimension over dim in group_by

* Separate SQL query building from imported.ex

* props.ex -> legacy_dimensions.ex

* Move some query building out of Query.ex

* Remove unneeded method

* put_filter -> add_filter

* Remove some query setters

* Moduledoc

* Split out validations and import tests from query_test

* Move tests around

* Split event:goal tests from query_test

* Remove redundant filters

* Remove dead code

* Split special metrics tests from query_test

* Legacy module
2024-07-09 14:31:45 +03:00
Karl-Aksel Puulmann
a181f3eab3
APIv2: TimeSeries using QueryBuilder, release experimental_session_count (#4305)
* Move fragments module under Plausible.Stats.SQL

* Introduce select_merge_as macro

This simplifies some select_merge calls

* Simplify select_join_fields

* Remove a needless dynamic

* wrap_select_columns macro

* Move metrics from base.ex to expression.ex

* Move WhereBuilder under Plausible.Stats.SQL

* Moduledoc

* Improved macros

* Wrap more code

* select_merge_as more

* Move defp to the end

* include.time_labels parsing

* include.time_labels in result

Note that the previous implementation of the labels from TimeSeries.ex was broken

* Apply consistent function in imports and timeseries.ex

* Remove boilerplate

* WIP: Limited support for timeseries-with-querybuilder

* time:week dimension

* cleanup: property -> dimension

* Make querying with time series work

* Refactor: Move special metrics (percentage, conversion rate) to own module

* Explicitly format datetimes

* Consistent include_imported in special metrics

* Solve week-related crash

* conversion_rate hacking

* Keep include_imported consistent after splitting the query

* Simplify do_decide_tables

* Handle time dimensions in imports cleaner

* Allow time dimensions in custom property queries

* time:week handling continued

* cast_revenue_metrics_to_money

* fix `full_intervals` support

* Handle minute/realtime graphs

* experimental_session_count? with timeseries

This becomes required as we try to include visits from sessions by default

* Support hourly data in imports

* Update bounce_rate in more csv tests

* Update some time-series query tests

* Fix for meta.warning being included incorrectly

* Simplify imported.ex

* experimental_session_count flag removal

* moduledoc

* Split interval and time modules
2024-07-09 14:25:02 +03:00
Adrian Gruntkowski
35596e8692
Remove Universal Analytics import logic (#4312) 2024-07-05 13:58:42 +02:00
Adrian Gruntkowski
2f2602e316
Stop listing UA views in Google data import flow (#4309)
* Stop listing UA views in Google data import flow

* Update CHANGELOG

* Trigger preview deploy
2024-07-04 13:08:32 +02:00
RobertJoonas
81047ba06d
Fix recommending a tier on the upgrade page (#4287)
* fix recommending a plan

* Fix credo issue

* Review suggestions

* remove dead code
2024-07-04 11:57:57 +03:00
Karl-Aksel Puulmann
05ac840078
APIv2: macros, SQL cleanup (#4286)
* Move fragments module under Plausible.Stats.SQL

* Introduce select_merge_as macro

This simplifies some select_merge calls

* Simplify select_join_fields

* Remove a needless dynamic

* wrap_select_columns macro

* Move metrics from base.ex to expression.ex

* Move WhereBuilder under Plausible.Stats.SQL

* Moduledoc

* Improved macros

* Wrap more code

* select_merge_as more

* Move defp to the end

* wrap_alias
2024-07-03 16:32:25 +03:00
Adrian Gruntkowski
790984e1ad
Refactor Sites and Stats API authorization logic (#4297)
* Refactor and unify auth plugs for Stats and Sites APIs

* Expose get site Sites API endpoint to all API keys

* Test the new plug

* Add test for endpoint with modified scope

* Fix typos

Co-authored-by: hq1 <hq@mtod.org>

* Rename plug for consistency (h/t @aerosol)

---------

Co-authored-by: hq1 <hq@mtod.org>
2024-07-02 15:09:23 +02:00
Karl-Aksel Puulmann
0594478add
APIv2: Replace breakdown module with QueryBuilder (#4293)
* Revert "Revert "APIv2: Replace breakdown module with QueryBuilder (#4283)" (#4292)"

This reverts commit ef5e0e0382.

* Allow querying events and pageviews from sessions table

This is not strictly accurate, especially with shorter time frames, but
is useful for a fallback mechanism. I'll figure out something around
shorter time frames in the future.

See also: https://github.com/plausible/analytics/pull/4292

* Only query events and pageviews in legacy breakdowns
2024-07-01 12:50:01 +03:00
Karl-Aksel Puulmann
ef5e0e0382
Revert "APIv2: Replace breakdown module with QueryBuilder (#4283)" (#4292)
This reverts commit 7dd12d1dd6.
2024-07-01 10:50:44 +03:00
Karl-Aksel Puulmann
7dd12d1dd6
APIv2: Replace breakdown module with QueryBuilder (#4283)
* WIP: Breakdown using QueryBuilder

* Revert "Remove problematic test"

This reverts commit b442bb5d1f.

* Get more breakdown tests passing

* Preload goals, sort when dealing with time_on_page

* Handle conversion_rate in breakdowns

* Simplify ordering by using selected_as consistently for dimensions

* Get breakdown tests passing

* Strings to atoms in keys for StatsController.transform_keys calls to work

* Handle revenue metrics removal

* Add test for nil-removal case

* Include percentage metric

* Fix and test with imported locations

* Fixup time-on-page

* Fix country/region automatic filters

* Handle multiple imports (os/browser version) in importsv2

* Filter goals

* Default to ordering by page as well

* Calculate conversion rate on sessions if needed

* Order by event dimensions - handles event:page special case

* Update tests

* Update more tests, handle goal=0 case in imports

* Handle event:goal breakdowns correctly with filters

* Revenue to money

* Improved table deciding

* Also update event:page filters on event:page breakdown

* bounce_rate to 0

Previous behavior relied on two queries being made - new query leads to 0 naturally

* Update pagination test

* dont count non-pageviews as path goal completions

* Make revenue logic breakdown-specific

Its hard to fit into the new schema and likely needs a rethink for apiv2

* Retain previous behavior for TimeSeries module

* Get GA4 test passing

Most failures are related to ordering, pageviews shouldnt be read off of sessions

* Clean up old methods

* Simplify imported.ex

* Dont crash on garbage filters

* Reflect ordering-related change in test

* Fix test data

* Update table_decider

* Re-simplify get_revenue_tracking_currency

* Revert revenue changes

* Use Query.set

* Remove a TODO

* csv importer: no pageviews

Pageviews were incorrectly fetched from sessions table before, causing issues

* csv importer tweaking

* Remove use Plausible

* to_existing_atom
2024-07-01 09:03:33 +03:00
hq1
f4e091452c
Require src attribute when seeking snippet for verification (#4290) 2024-07-01 07:38:25 +02:00
Karl-Aksel Puulmann
2eeaf7a152
APIv2: Aggregates, timeseries, conversion_rate, hostname (#4251)
* Add some aggregates tests

* Port aggregates tests to do with filtering

* Session metrics can be queried with event: filters

* Solve a typo

* Update a validation message

* Add validations for views_per_visit

* Port an aggregation/imports test

* Optimize time dimension, add tests

* Add first timeseries test, update parsing tests

* Docs for SQL.Expression

* Test timeseries more

* Allow time explicitly in order_by

* Add multiple breakdowns test

* Refactor QueryOptimizer not to care about time dimension placement in dimensions array

* Add test breaking down by event:hostname

* Add hostname filtering logic to QueryOptimizer, unblock some tests

* WIP: Breakdown by goal

* conversion rate logic for query api

* Update more tests

* Set default order_by

* dimension_label

* preloaded_goals in tests

* inline load_goals

* Use Date functions over Timex

* Comments

* is_binary

* Remove special form used in tests

* Fix defmodule

* WIP: Fix memory leak, event:page breakdown logic

* Enable more tests, fix for group_conversion_rate without explicit visitors metric

* Re-enable a partially commented test

* Re-enable a partially commented test

* Get last test passing

* No imports order_by in apiv2

* Add a TODO

* Remove redundant Util call

* Update aggregate.ex

* Remove problematic test
2024-06-28 08:59:54 +03:00
RobertJoonas
07a54ef65c
Count pending ownership sites towards account usage on the upgrade page (#4277)
* add notice about pending ownerships counting towards usage

* Refactor feature usage to take site_ids too

* Add option to query usage from pending ownerships

* count pending ownerships towards usage on the choose-plan page

* turn 'eligible_for_upgrade?' into a predicate

* fix variable names on ce

* fix unused alias on ce

* fix ce test
2024-06-27 12:09:15 +03:00
Karl-Aksel Puulmann
58a66a952c
APIv2 - initial PR (#4216)
* WIP new querying

* WIP: Move some aggregate code under new command

* WIP: Add joins, handling less metrics

* join events table to sessions if needed

* Merge imported results with built query

* Remove dead code

* WIP: /api/v2/query

* Allow grouping by time

* Use JOIN for main query

* Build query result

* update parse_time

* Make joinless order by work

* First test

* more breakdown tests

* Serialize event:goal filters in an json-encodable way/reflection

* Handle inner vs outer ORDER BY clauses properly

* Handle single conversion_rate metric

* Update more tests

* Get parsing tests passing again

* Validate filtered goal filter is configured

* Enable more validation tests

* Enable more event:name breakdown tests

* Enable more breakdown tests

* Validate site has access to custom props

* Validate conversion_rate metric which is only allowed in some situations

* Validate that empty event:props: is not valid

* handle query.dimensions properly in table_decider

* test more validations on metrics/dimensions

* Validate session metrics in combination with event dimension(s)

* Tests cleanup

* Parse include.imports

* Get imports working with new querying

* Make more imports tests work

* Make event:props:path imports-adjacent test work

* Get query imports warning-related tests running

* Remove dead pagination tests

* Solve dead import

* Solve some warnings

* Update aggregate metrics tests

* credo

* Improve test naming

* Lazy goal loading

* Use datetime methods

* Ecto -> SQL module name

* Remove Expression.dimension mode option
2024-06-25 09:27:19 +03:00
ruslandoga
5a0718aa75
update runtime to elixir-1.17.1 and erlang-27.0 (#4255)
* update runtime to elixir-1.17.1 and erlang-27.0

* update credo

* fix some warnings in tests

* temp fix (backport)

---------

Co-authored-by: hq1 <hq@mtod.org>
2024-06-24 13:22:33 +02:00
Karl-Aksel Puulmann
284b4ebbf6
does_not_contain, new filters wildcard matching (#4228)
* Make goals accept wildcards in :is queries

* Add support for contains/does_not_contains in the backend

* Support all operations for event custom properties

* Support does_not_contain on the frontend

* Changelog entry

* Render filter operations nicely for does not contain

Found 3 extra pixels for operation dropdown

* Remove multiple_filters feature flag
2024-06-21 11:32:05 +03:00
Karl-Aksel Puulmann
ed8abf426b
Revert "Revert "Re-apply multiple filters in FE (#4219)" (#4223)" (#4227)
This reverts commit 2c05676d8e.
2024-06-21 11:31:02 +03:00
Adrian Gruntkowski
766807e9fd
Add backlinks between CRM, dashboard and site members (#4252)
* Add button to dashboard in site CRM edit form

* Add links to CRM for site and each user in "People" settingsg
2024-06-20 14:03:58 +02:00
RobertJoonas
db1e755b0e
Hide 20M and 50M legacy plans from the upgrade page (#4248)
* Stop showing 20M and 50M plans unless user is already on one

* Add tests
2024-06-20 13:04:04 +03:00
Adrian Gruntkowski
fc20782287
Fix pagination is Sites CRM (#4242) 2024-06-20 09:32:38 +02:00
ruslandoga
9cbb9ba79c
Fix CE Stats API access (#4244)
* fix ce stats api access

* changelog

* tag groth tests ee only
2024-06-20 09:32:21 +02:00
RobertJoonas
b5c5ab8e61
ignore irrelevant paddle webhook (#4240) 2024-06-18 11:47:16 +03:00
hq1
1c820ba2af
Verification tweaks (#4234)
* Prioritize CSP over GTM error in case both detected

* Outline better message for persistent headless timeout

* Bump headless waiting times slightly

* Add generic type of error

For known cases, that we rather not reveal much
details about.

* Add the malformed script tag case
2024-06-18 05:58:56 +02:00
RobertJoonas
f9a4cc784d
Make billing cycles less confusing (#4233)
* Set default selected tab to last cycle when rendering usage

...And never disable this tab.

* rename Ongoing cycle to Upcoming cycle

* fix tests

* mix format
2024-06-17 13:51:30 +02:00
Adrian Gruntkowski
e65e37afc0
Implement CRM enterprise plan definition QoL improvements (#4230)
* Implement autoprefill of enterprise plan fields on user change

* Implement sanitizing input attrs in enterprise plan CRM form

* Implement number formatting for monthly pageview limit input in CRM form
2024-06-17 13:11:53 +02:00
Adrian Gruntkowski
aadf528459
Add CRM QoL improvements around usage info and linking Paddle (#4229)
* Link subscription status to Paddle profile whenever available

* Display usage info directly in user form in CRM

* Handle pageview limit fallback gracefully when rendering subscription quota

* Remove usage page link from users list view in CRM

* Add tests for usage page

* Apply `raw/1` only to the one (and only) element of the list
2024-06-17 09:21:24 +02:00
RobertJoonas
dd1d74ccb7
Always recommend a suitable plan on the choose-plan page (#4222)
* pull last_bill_date from paddle sandbox in mix task

* move cycle usage checks to Quota module

* move quota.ex to a subfolder

* split up Quota module

* set choose-plan pageview slider according to usage

* silence credo
2024-06-17 09:25:46 +03:00
hq1
86d7031336
Unify default pricing currency selection (#4221)
* Start cache meant to store customer currencies

* Expose caching fetch_or_store interface

* Improve IP picking strategy - skip empty header values

* Use customer IP in determining pricing currency

* Expose /api/paddle/currency API

* Remove cache-control header

* Tidy up
2024-06-14 14:49:22 +02:00
Karl-Aksel Puulmann
2c05676d8e
Revert "Re-apply multiple filters in FE (#4219)" (#4223)
This reverts commit 77520390f8.
2024-06-13 12:45:50 +03:00
Karl-Aksel Puulmann
77520390f8
Re-apply multiple filters in FE (#4219)
* Revert "Revert "Multiple filters on the frontend (#4174)" (#4218)"

This reverts commit 2f7dcae991.

* Make city links work again

By enforcing everything sent to the BE is stringified. We do this at serialization-time to ensure old dashboard links still work
2024-06-13 09:56:15 +03:00
hq1
2f7dcae991
Revert "Multiple filters on the frontend (#4174)" (#4218)
This reverts commit e1f0002d2e.
2024-06-11 17:42:07 +03:00
hq1
4603aa7a51
Reset grace period if usage drops within the last cycle (#4212)
So that `LockSites` job unlocks the site,
encountering no grace period and no upgrade needed.
2024-06-11 15:49:31 +02:00
hq1
9f1f826801
Allow site creation for users owning the domain_changed_from (#4215)
* Allow site creation for users owning the `domain_changed_from`

* Format
2024-06-11 15:48:46 +02:00
Karl-Aksel Puulmann
e1f0002d2e
Multiple filters on the frontend (#4174)
* Add new logic for query parsing

Note this is not yet final, but this scaffolding will soon be used

* Send filters to BE in new encoding, flag add more button

* query.property -> query.dimensions

* Reduce number of operations for filtering

This helps ensure we can be consistent/simple with apiv2

* Update search console filter mapping

* Update filters sent from frontend

* Update new filter parsing

* Remove redundant clause

* Make filtering by event:goal work

* Handle * as old backend did - prefix/suffix by **, if not using wildcards already

* Update imports logic

* Credo warning

* Spacing for add row button in filter modals

* query fix

* LegacyDashboardFilterParser

* only single hostname filter allowed

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
2024-06-11 14:22:53 +03:00
Adrian Gruntkowski
d5a1d6765e
Implement custom events suggestions (#4092)
* Start suggesting event names in goal settings form

* Fix tests

* Bump phoenix_live_view to 0.20.12

* Implement a criminal hack to track removal of modal's child live components

* Revert "Implement a criminal hack to track removal of modal's child live components"

This reverts commit f34ceb78f1.

* Remove redundant closing brace from currency combo input

* Hide batch goal add button when tab selection is in progress

* Implement unique modal ID regenerated on every modal close/open cycle

* Use unique modal ID as ID suffix to live components in goal settings form

* Reset suffix on tab selection to reset live components state on switch

* Revert "Bump phoenix_live_view to 0.20.12"

This reverts commit 1b1c801981.

* Make unique IDs more predictable

* Fix tests for `GoalSettings.Form`

* Use unique modal ID in country rule modal

* Use unique modal ID in hostname rule modal

* Use unique modal ID in page rule form modal

* Don't limit detected goals when fetching them for autoconfigure

* Escape interpolated Alpine state function argument

* Exclude goals with whitespace on either end or consisting only of whitespace

* Ensure event name suggestions update after goal deletion

* Avoid showing loading spinner when closing modal

* Don't enable spinner when new combobox selection is identical

* Revert "Don't enable spinner when new combobox selection is identical"

This reverts commit a041ba8542.
2024-06-11 08:28:25 +02:00
RobertJoonas
7b11b71dcf
Minor refactor: Remove skip_refresh_imported_opts from query modifying functions (#4204)
* Remove skip_refresh_imported_opts from query modifying functions

+ add test coverage for timeseries conversion rate.

* format

* improve comment
2024-06-10 13:03:12 +03:00
hq1
92d5092518
Verification improvements next batch (#4203)
* Upgrade req

* Fetch body even on non-200 statuses

Turns out sites, will happily go with 307 and no location
header for example. In which case saying we can't reach it
is an overstatement. After all we should let the headless
check determine the rest.

* Update callback status interpretation cases

* Update tests

* Fix flaky test

* Update expectation
2024-06-10 10:40:19 +02:00
ruslandoga
22ea56a779
increase api_keys.hourly_request_limit in CE (#4200)
* increase api_keys.hourly_request_limit in CE

* changelog

* add tests
2024-06-10 07:21:58 +02:00
hq1
823f7a8800
Plugins API: Get/Create Funnels (#4192)
* Bootstrap OpenAPI Funnel schemas

* Implement Plugins API Funnel view

* Allow casting funnel step directly from `%Goal{}`

* Check feature availability on funnel creation

just like it's done when inserting goals

* Implement Plugins API context module for Funnels

* Implement GET/PUT funnels via Plugins API

* Fix typo

* A rare event in which dialyzer found an actual bug, wow!

* Format

* Wrap creation request with a root `funnel` key

* Format

* Extract common funnel get query

* Remove redundant tag

* Refactor queries a bit
2024-06-07 13:10:01 +02:00
RobertJoonas
2a529b4eaa
Fix with imported switch + warning bubble renders (#4187)
* prioritize other skip_imported_reasons over not requested

* return with_imported_switch info from top stats

* fix the with-imported-switch component

* Refactor ImportedQueryUnsupportedWarning

* do not render warning bubble when imported data not requested

* stop displaying warning bubble before the API response is received

* fix tab switch pills jumping due to bubble height

* drop loading condition of ImportedQueryUnsupportedWarning for Funnels

* add explicit check for realtime when rendering the bubble

* remove unused JS imports

* rename snake_case to camelCase
2024-06-06 11:50:54 +03:00
Adrian Gruntkowski
fd81ef2105
Include imported page paths in pageview goal suggestions (#4193) 2024-06-06 10:40:48 +02:00
Adrian Gruntkowski
a764938928
Fix handling filtering by imported special properties (#4179)
* Handle custom prop queries separately when deciding imported table

* Add Query level tests
2024-06-04 13:06:31 +02:00
hq1
5fd4547382
More verification tweaks (#4177)
* Stop reporting multiple snippets in case manual extension is in use

* Try detecting CookieBot/GTM case
2024-06-03 13:49:04 +02:00
RobertJoonas
1d3b068233
Implement filtering for imported data (#4118)
* move imported.ex to imported subfolder

* move constructing base imported query into a separate module

* Implement imported table deciding and filtering

+ tests for pages, entry_pages, exit_pages and common filter types

* add top stats test with country filter

* add timeseries test

* Drop bounce_rate and time_on_page from imported & page-filtered Top Stats

* rename field returned by top stats

* turn pages into a fn comp

* Move dashboard API results under a results key

...and also return the skip_imported_reason to the frontend to be used
for displaying warnings.

* extend ListReport component with an optional afterFetchData prop

* turn Devices into a fn comp

* add not_requested as a skip_imported_reason

* display warning icons in the dashboard

* Implement filtering suggestions and translate filter fields for imported

* WIP

* Improve and cover filtering suggestions with tests

* Rename imported suggestions query helpers

* fix screen size breakdown with screen size filter

* support filtering by the same suggestion property

* support location filters when fetching location suggestions

* support filtering by multiple props from the same table

* Implement filtering by goals

* Make views per visit metric work for import entry and exit pages

* Get rid of circular dependencies between Stats.Imported and Stats.Imported.Base

* Clean up Query struct manipulation in Breakdown

* Rename helper function for clarity

* Automatically refresh query struct state after modifications

* Shutup credo

* display imported warning bubble in prop breakdown section

* Render warning bubble for funnels whenever imported data is in the view

* Transform any operator on respective goal filters

* Fix percentage and conversion_rate calculation in presence of custom props

* add tests for for combining page and pageview goal filters

* add skip_refresh option to query tweaking functions

* add imported CR support for timeseries

* still show url breakdown when special goal + url in filter

* rename Query.refresh

* use flat_map instead of map and concat

* fix darkmode color

* Handle invalid imported region codes in suggestions gracefully

* Add an entry to CHANGELOG.md

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-06-03 12:29:08 +02:00
ruslandoga
8588776df2
Add newline to csv importer insert sql (#4172)
* add newline to csv importer insert sql

* add test

* changelog

* fix test typo

* fix test typo x2
2024-06-03 09:48:16 +02:00
hq1
bb7b0d9f94
Tidy up verification code a little (#4156) 2024-05-29 08:51:10 +02:00
hq1
9d77e970ef
Redo site verification UI (#4153)
* Redo site verification UI

* format
2024-05-28 13:43:14 +02:00
hq1
d9cf037866
Handle no snippet WordPress case (#4152) 2024-05-28 09:37:01 +02:00
hq1
78c5762761
Verification: prioritize domain mismatch over multiple snippets (#4146) 2024-05-27 10:52:22 +02:00
hq1
0381bc6121
Use codespell (#4145)
* Fix typos

* Add .codespellignore

* Add codespell GH action
2024-05-27 10:52:01 +02:00
hq1
206a6543f6
First pass: extract verification errors (#4138)
* First pass: extract verification errors

We like to make them easier to maintain.
An update fixing documentation URLs will follow.

* Update moduledoc
2024-05-24 16:23:25 +02:00
hq1
a7b69719c1
More fixes to the verification process (#4136)
* More fixes to the verification process:
 - extend the list of known attributes
 - follow redirect chain host to verify data-domain difference
 - display the actual data-domain in error message, not URL

* format

* fix test

* remove redundant pattern matching

* Update lib/plausible/verification/checks/snippet.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-05-24 12:45:31 +02:00
hq1
ee61094023
Verification improvements (#4132)
* Simplify browserless script and catch errors

* Improve fetching body:
    - follow up on max. 4 redirects
    - rely on Req default timeouts (wait much longer to account for slow
      sites)

* Improve installation check:
  - rely on Req default timeouts, wait longer
  - log service errors as warnings
  - use stealth mode to bypass captchas

* Stop cutting off body too large

* Improve diagnostics for known failures

* Another round of diagnostic improvements

* Format

* Add a test for callback status 500
2024-05-24 10:40:59 +02:00
hq1
c81cb16933
Snippet integration verification (#4106)
* Allow running browserless.io locally

* Compile tailwind classes based on extra/ too

* Add browserless runtime configuration

* Ignore verification events on ingestion

* Improve extracting HTML text in tests

* Update dependencies

- Floki will be used on production to parse site contents
- Req will be used to handle redundant stuff like retrying etc.

* Add shuttle SVG to generic components

Later on we'll use it to indicate verification errors

* Connect live socket & allow skipping awaiting the first pageview

* Connect live socket in general settings

* Implement verification checks & diagnostics

* Stub remote services with Req for testing

* Change snippet screen copy

* Update tracker script, so that:

1. headless browsers aren't ignored if `window.__plausible` is defined
2. callback optionally supplies the event response HTTP status

This will be later used to check whether the server acknowledged
the verification event.

* Implement LiveView verification UI

* Embed the verification UIs into settings and onboarding

* Implement browserless puppeteer verification script

It:
 - tries to visit the site
 - defines window.__plausible, so the tracker doesn't ignore test events
 - sends a verification event and instruments the callback
 - awaits the callback to fire and returns the result

* Improve diagnostics for CSP

Only report CSP error if the snippet is already found

* Put verification behind a feature flag/env setting

* Contact Us hint only for Enterprise Edition

* For headless code, use JS context instead of EEx interpolation

* Update diagnostics test with WordPress scenarios

* Shorten exception/throw interception

* Rename test

* Tidy up

* Bust URL always on headless check

* Update moduledoc

* Detect official Plausible WordPress Plugin

and act accordingly on diagnostics interoperation

* Stop using 'rating' in favour of 'interpretation'

* Only report CSP error if no proxy is likely

* Update CHANGELOG

* Allow event-* attributes on snippet elements

* Improve naive GTM detection, not to confuse it with GA4

* Update lib/plausible/verification.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update test/plausible/site/verification/checks_test.exs

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* s/perform_wrapped/perform_safe

* Update lib/plausible/verification/checks/installation.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Remove garbage

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-05-23 15:00:50 +02:00
ruslandoga
9687fa786d
remove Plausible Analytics metions from CE (#4121)
* remove Plausible Analytics metions from CE

* update tests

* still mention Plausible Analytics on landing page
2024-05-23 09:43:01 +02:00
hq1
cd4d1d06bb
Rename Plugins API Token(s) to Plugin Token(s) (#4075)
Deployment needs to be coordinated with WordPress plugin
update. Reasoning: there's too much confusion coming
from the use of the "API" terminology, leading to wrong
keys being generated by users.
2024-05-22 07:45:09 +02:00
Adrian Gruntkowski
2d23db2982
Return warning when imported stats are skipped in Stats API due to unsupported query (#4116) 2024-05-21 11:45:18 +02:00
Adrian Gruntkowski
7876802d05
Improve CSV importer tests and visit duration query for imported data (#4114)
* Tidy up breakdown helper functions in CSV importer tests

* Fix a typo

* Extend city breakdown tests in CSV importer test suite

* Make visit duration computation consistent across native and imported data
2024-05-21 09:24:10 +02:00
Adrian Gruntkowski
756d9c28ce
Improve test coverage and fix stats API and dashboard CSV export (#4110)
* Fix broken hostname property and handle missing imported metrics gracefully

* Add test for CSV export of imported data

* Add extra coverage for property and metric combox which were failing

* Compute visit duration and bounce rate for exit pages in imported data

* Drop support for breaking down by `event:hostname` property for now
2024-05-20 16:11:35 +01:00
RobertJoonas
df178ea2d5
Introduce :skip_imported_reason field into Query struct (#4115)
* introduce new query field

* improve tests
2024-05-20 15:46:25 +01:00
Karl-Aksel Puulmann
cfdd769984
Fix StatsAPI access for new accounts on business tier (#4105)
Currently, business tier users created after business tier launch can't
access Stats API due to faulty grandfathering logic. This change should
fix that.
2024-05-15 11:50:41 +03:00
Adrian Gruntkowski
9374a95cf2
Export and import custom events via CSV (#4096)
* Export and import custom events via CSV

* Add prop support of url for cloaked links and path for 404s in imported queries

* Handle custom events with empty URL and path properties gracefully

* Make events with properties logic DRY and fix missed cloaked link

* Add test for path property breakdown

* Update raw CH data fixture and extend CSV importer tests

* Fix broken query condition after rebase

* Update CHANGELOG.md
2024-05-14 13:50:30 +02:00
Karl-Aksel Puulmann
baa99652f6
Refactor internal Query schema and introduce WhereBuilder (#4082)
* New struct format for query after parsing

* WIP refactoring

* WIP: Validations working

* WIP: tuple to list

* continued refactoring

* WIP: parsing defaults

* Breakdown tests pass

* Window functions fix

* Fix default

* Remove dead argument

* Update filters tests

* Update query_test.exs

* Fix table_decider

* sources tests pass

* Filter suggestions fix

* revenue/goal filter applied refactor

* Update top_stats matching

* Get stats_controller tests passing

* Update neighbor_aggregate_time_on_page

* Refactor Query.remove_event_filters into Query.remove_filters, add new callsites

* Move goal where clause building to new WhereBuilder module

* Move event:name filters

* Move more filters to WhereBuilder

* Update fragment to allow non-static meta columns

* Build where clause for events table using WhereBuilder

* Build sessions table where clause using WhereBuilder

* Move time range filtering and site checking to WhereBuilder

* WhereBuilder.build_condition method

* Remove TODO

* _rest pattern for TableDecider, Query pattern matching

Future-proofing in a tiny way

* Hacky fix to get tests passing for Google API tests

* Typespec fix

* Merge conflict

* refactor special goal filter logic in imported.ex

* Docs feedback

* put_filter

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
2024-05-14 11:58:10 +03:00
Uku Taht
06e8118dab
Filtering Search Console keywords (#4077)
* Apply filters in search console request

* Remove dead code from search console modal

* Remove unimportant information from keyword modal

* Show invalid filters from search console

* Fix tests

* Add/Fix tests

* Fix typo

* Remove unused variable

* Fix typo

* Changelog entry

* Fix Credo

* Display impressions, CTR and position in keyword modal

* Undo change that should not have been committed

* Fix test

* Fix test

* filters -> search_console_filters
2024-05-14 09:56:55 +03:00
ruslandoga
1f4346f4df
Add DATA_DIR (#4100)
* add DATA_DIR

* add test

* changelog

* fix test in CI where PERSISTENT_CACHE_DIR is always set

* consistent fallback
2024-05-13 10:17:56 +02:00
ruslandoga
7af8273702
Handle s3 timeout on settings page (#4036)
* handle s3 timeout

* no-async approach

* fewer changes

* add tests

* make s3 failure test EE-only
2024-05-13 10:17:27 +02:00
RobertJoonas
a6dcd19ccc
Autoconfigure event goals (#4093)
* add new goal suggestions API

* silence credo

* Order suggestions from subqueries explicitly

* allow autoconfiguring goals

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Fix form modal tab switching behavior

* add test

* Remove redundant and invalid action link title

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-05-10 11:48:27 +02:00
Adrian Gruntkowski
6bbc8d69a4
Avoid crashing on empty GA4 HTTP report response (#4094) 2024-05-10 09:12:22 +02:00
Adrian Gruntkowski
4e7e932a75
Add support for imported custom events (#4033)
* Add Ecto schema for imported custom events

* Start importing custom events from GA4

* query imported goals

* make it possible to query events metric from imported

* make it possible to query pageviews in goal breakdown

* make it possible to query conversion rate

* fix rate limiting test

* add CR tests for dashboard API

* implement imported link_url breakdown

* override special custom event names coming from GA4

* allow specific goal filters in imported_q

* update GA4 import tests to use Stats API

* Improve tests slightly

* Update CHANGELOG.md

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
2024-05-09 13:13:19 +01:00
Adrian Gruntkowski
d8435f2e01
Remove imports_exports and csv_imports_exports feature flags (#4089) 2024-05-09 10:09:24 +02:00
ruslandoga
84ed7988a8
Drop local_start_date/1 (#4088) 2024-05-09 08:59:23 +02:00
Karl-Aksel Puulmann
0a883f10e7
Refactor: Use common current_visitors code (#4071)
* Use common module for counting current visitors in external stats controller

* Refactor spike notifier, remove now-dead code
2024-05-07 15:03:37 +03:00
Karl-Aksel Puulmann
64850cd00f
Remove maybe_drop_prop_filter (#4066)
* Fix event props paygate

Previous code wasn't properly omitting event property filters from
queries.

Discovered while refactoring the code. Extracting fix from refactor for easier reviewability.

* a

* Drop function
2024-05-07 15:03:16 +03:00
ruslandoga
02d4709be7
add csv fixture for e2e export/import test (#4037)
* add inline csv fixture

* use new csvs

* cleanup csv reading and site_id replacing

* perform comparisons between native and imported queries

* help help help

* help help

* help

* eh

* fin

* exclude export/import e2e test when experimental_reduced_joins flag is enabled

* adapt to new pageviews

* adapt to experimental_reduced_joins

* credo is formatter

* cleanup

* assert bounce rates equal in city breakdown

* fix rebase against master

* clean-up dataset

* update comment

* fix typo

* apply csv changes to the files

* use sessions timestamp for exports' dates

---------

Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
2024-05-07 08:48:22 +01:00
Adrian Gruntkowski
d7ca8d9600
Revert "Debug queries for super-admins (#4010)" (#4073)
This reverts commit dd493fdad2.
2024-05-06 15:05:17 +02:00
Karl-Aksel Puulmann
17f812443d
Return session in each time bucket its active in for hourly/minute timeseries (#4052)
* Fix typo in test name

* Update test_helper, enable experimental_session_count together with experimental_reduced_joins

* Return session in each time bucket its active in for hourly/minute timeseries

The behavior is behind experimental_session_count flag

This results in more accurate visitor showing compared to previous approach of showing each user only
active the _last_ time they did a pageview.

Were not doing this for monthly/weekly graphs due to query performance cost and it having a small effect there.
See also https://3.basecamp.com/5308029/buckets/35611491/messages/7085680123

* Add tests for new behavior

Note the new behavior mimics the old one precisely, these tests fail if only
experimental_reduced_joins is on, but not experimental_session_count

* Type erasure

* Dead comment remove

* Expected_plot change
2024-05-05 11:44:43 +03:00
ruslandoga
972dd5d150
redirect to s3 url when downloading exports (#4002)
* redirect to s3 url

* use new on_ee macro, reduce wait time for email to five seconds
2024-05-02 19:53:12 +01:00
ruslandoga
8712e91bcb
drop time on page from exports (#4051) 2024-05-02 08:29:24 +01:00
RobertJoonas
bfdadc2eee
Include breakdown property in the Query struct (#4053)
* keep breakdown prop in the query struct

* Explicitly ignore property param in aggregate and timeseries

Since parameter validation depends on the breakdown property, we need to
make sure it doesn't have any unexpected effect in endpoints where it's
not expected.
2024-04-30 18:43:46 +01:00
Adrian Gruntkowski
41fef85d29
Implement resumable GA4 imports to work around rate limiting (#4049)
* Add support for resuming import to GA4 importer

* Handle rate limiting gracefully for all remainig GA4 HTTP requests

* Show notice tooltip for long running imports

* Bump resume job schedule delay to 65 minutes

* Fix tooltip styling
2024-04-30 18:06:18 +02:00
hq1
dd493fdad2
Debug queries for super-admins (#4010)
* Debug queries for super-admins

* Fixup

* Update lib/plausible/clickhouse_repo.ex

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>

* Try again with https://github.com/plausible/analytics/pull/3699

It's still clunky 😅

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
Co-authored-by: ruslandoga <doga.ruslan@gmail.com>

* Move headers injection to a separate plug module

* Add tests

* Update repo test

* Format

* Add moduledoc

* Don't assume order in query_log

* Be patient about query_log maybe?

* huh?

* huh2

* Wait longer

* Guard against \x00 in response header - testing on stage

* Fixup

* fixup

* fixup

* s/debug_label/label

* Include `site_id` and `metadata` in `log_comment`

* Tolerate non-serializable log_comment contents

---------

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
2024-04-30 09:57:28 +02:00
hq1
ad9141a9d0
Display tooltips on plan change when limits exceeded (#4048)
* Reapply "Display upgrade tooltips for exceeded limits (#4032)"

This reverts commit 76e910d45c.

* Switch to alpinejs controlled tooltips

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>

* Remove unused selector

* Refactor plan limits warning and extract tooltip component

* Remove redundant check

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-04-29 11:16:15 +02:00
hq1
b2009aa158
Rely on con_cache telemetry (#4019)
* Rely on con_cache telemetry

Now that https://github.com/sasa1977/con_cache/pull/76
is released, we don't have to use low-level operations
to emit hit/miss events.

This PR also wraps cache processes with
a function returning appropriate child specs lists.

Ideally each cache will have its own supervisor/child specs
going forward. This is an intermediate step in that direction.

* Update lib/plausible/application.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Declare caches without warmers with plain child specs

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-04-29 11:00:53 +02:00
RobertJoonas
726fe2d982
fix test (#4038) 2024-04-29 09:30:30 +01:00
hq1
d6824de1ad
Rename internal build symbols (#3942)
* Rename internal build symbols

* Rename remaining + add `on_ce` macro

cc @ruslandoga
2024-04-29 08:05:33 +02:00
Adrian Gruntkowski
ca25b6c764
Bump GA4 import page limit to max 250k (#4045)
* Bump GA4 page limit to 250k

* Handle empty properties list response gracefully
2024-04-26 20:41:42 +02:00
Adam Rutkowski
76e910d45c Revert "Display upgrade tooltips for exceeded limits (#4032)"
This reverts commit 4a372a1528.
2024-04-23 12:53:26 +02:00
hq1
4a372a1528
Display upgrade tooltips for exceeded limits (#4032) 2024-04-23 10:42:48 +02:00
Adrian Gruntkowski
a8ea4ce54b
Add new actions to CRM (#4030)
* Implement `Auth.TOTP.force_disable/1`

* Add "Reset 2FA" action to users CRM

* Add `Purge.reset!/2` variant allowing to set arbitrary cutoff time

* Add ability to set native stats start time from CRM

* Revert "Add `Purge.reset!/2` variant allowing to set arbitrary cutoff time"

This reverts commit 6f294d5d58.

* Add test for CRM site update action
2024-04-23 10:29:49 +02:00
hq1
148413afbb
Fix dogfooding invitation event (#4023)
* Fix dogfooding invitation event

* Fix small build
2024-04-22 16:43:04 +02:00
Adrian Gruntkowski
3023cb12fd
Introduce active_visitors to imported_pages and start populating it with activeUsers from GA4 imports (#4027)
* Import `activeUsers` into `imported_pages.active_visitors` for GA4

* Add test for active visitors

* Simplify assertion in active visitors test

* Improve assertion for active visitors further
2024-04-22 10:18:16 +02:00
Adrian Gruntkowski
fede2f0a8a
Make final touches to Imports & Exports (#4025)
* Make final touches to Imports & Exports

* Change import content copy depending on CSV imports and exports flag state

* Remove unused aliases
2024-04-19 11:40:13 +02:00
Adrian Gruntkowski
c10580777e
Remove references to site.imported_data (#4006)
* Remove references to `site.imported_data`

* Count pre-existing ID 0 imports when showing pageview count summary for legacy imports

* Fix tests after rebase

* Dry `delete_imported_stats!`

* Clean up remaining imported data references and add notes
2024-04-19 11:15:51 +02:00
Adrian Gruntkowski
9bae3ccce3
Improve UI/UX of imports view and GA import flow (#4017)
* Add runtime config option for enabled/disabling csv imports and exports

* Use the new option to toggle rendering exports UI

* Disable import buttons when at maximum imports or when option disabled for CSV

* Improve forms for GA import flow

* Add test for maximum imports reached

* Remove "Changed your mind?" prefixing back button

* Hide UA imports in Integrations when `imports_exports` flag is enabled

* Implement `csv_imports_exports` feature flag

* Revert "Add runtime config option for enabled/disabling csv imports and exports"

This reverts commit e30f202dd3.

* Send import notification email only to the user who ran the import

* Improve rendering of disabled button state

* Put import status heroicon in front of import label
2024-04-18 12:12:48 +02:00
RobertJoonas
3a371fdf4d
Test new imported metrics (GA4) (#4014)
* test fixture imported data with stats requests

* take visits metric from the events table in event:page breakdown

* Remove assert_referrers after all

pageReferrer is an event scoped property in GA4, which when queried
along with session-level dimensions will return unexpected data.

Adding the pageReferrer dimension to the GA4 Data API request, it will
cause the selected metric totals to increase significantly, even though
they shouldn't.

* Adjust sources and utm_mediums assertions

* adjust assert_pages

* Make formatter happy

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-04-18 12:12:24 +02:00
Adrian Gruntkowski
9849743407
Always sort occupied date ranges in Imported.clamp_dates/3 (#4018) 2024-04-18 11:15:51 +02:00
Adam Rutkowski
2b1fbf0a0e Fix typo (and kick docker build) 2024-04-16 20:57:29 +02:00
hq1
6fb56dc1cc
Stats api hostname filter (#4008)
* Update Stats API tests

* Revert "Remove hostname filter from the external API (#3991)"

This reverts commit 884daa7943.
2024-04-16 20:36:57 +02:00
hq1
f635f0a6d3
Hostnames shield (#3990)
* Add shield hostname rules migration

* Add hostname rule schema

* Initialize hostname rules cache

* Extend Shields context with hostname related functions

* Instrument ingestion pipeline with hostname rule lookups

* Limit hostname suggestions by shield patterns

* Add LiveView for hostname rules management

* Test hostname cache

* Rename feature flag - should be separate from hostname filter

* Remove :shield_pages feature flag

* Update CHANGELOG

* Format

* Update lib/plausible/shield/hostname_rule.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Move tests from `lib/` 🤦

* Use plain `assign` where no short-circuit is necessary

* Fine tune the copy a little bit

* Prevent misplaced tests

* Treat a test with common sense

* Fixup another test that hasn't been really run before

* Make the form hint dynamic depending on rules count

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-04-16 20:30:20 +02:00
RobertJoonas
fce909d041
Improve merge imported (#4003)
* Pass the actual date args that are returned in fixtures (just for clarity)

* Change select key from operating_system to os

* Select (not set) from imported data as done from native stats

* Support breakdown by imported os_version

* Remove dead code

The `query.include_imported` is set to `false` from the start when
filters are included.

* Refactor Plausible.Stats.Imported

Extract a function that does group_by and dimension select, instead of
doing both separately inside the merge_imported function body

* Further refactor of Plausible.Stats.Imported

Get rid of code repetition

* Support breakdown by imported browser_version

* Support breakdown by imported referrer

* Support breakdown by imported referrer

* remove redundant if in select

* use greatest instead of coalesce

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>

* add back the :member filter handling in Stats.Imported

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
2024-04-16 16:58:22 +01:00
Adrian Gruntkowski
c07f00636d
Stop importing page referrer from GA4 (#4012)
* Stop importing page referrer from GA4

* Update GA4 import fixture

* Update fixture-based test
2024-04-16 15:35:36 +02:00
Adrian Gruntkowski
c1c03b729c
Reapply "Local CSV exports/imports and S3/UI updates (#3989)" (#3995) (#3996)
* Reapply "Local CSV exports/imports and S3/UI updates (#3989)" (#3995)

This reverts commit aee69e44c8.

* remove unused functions

* eh, that one was actually used

* ugh, they were both used

---------

Co-authored-by: ruslandoga <67764432+ruslandoga@users.noreply.github.com>
2024-04-11 09:15:01 +02:00
RobertJoonas
5163880968
fix test (#4001)
* fix test

* Update test/plausible_web/controllers/api/stats_controller/suggestions_test.exs

Co-authored-by: hq1 <hq@mtod.org>

* format

---------

Co-authored-by: hq1 <hq@mtod.org>
2024-04-10 12:40:42 +01:00
hq1
378d3bc6f5
Discard sessions switching hostnames for UTM/referrer breakdowns (#4000)
* Discard sessions switching hostnames for UTM/referrer breakdowns

Co-authored-by: Uku Taht <uku.taht@gmail.com>

* Format

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
2024-04-10 11:38:15 +02:00
Adrian Gruntkowski
aee69e44c8
Revert "Local CSV exports/imports and S3/UI updates (#3989)" (#3995)
This reverts commit 1a0cb52f95.
2024-04-09 21:26:23 +02:00
ruslandoga
1a0cb52f95
Local CSV exports/imports and S3/UI updates (#3989)
* local CSV exports/imports and S3 updates

* credo

* dialyzer

* refactor input columns

* fix ci minio/clickhouse tests

* Update lib/plausible_web/live/csv_export.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* fix date range filter in export_pages_q and process only pageviews

* remove toTimeZone(zero_timestamp) note

* use SiteImport.pending(), SiteImport.importing()

* escape [SiteImport.pending(), SiteImport.importing()]

* use random s3 keys for imports to avoid collisions (sometimes makes the upload get stuck)

* clamp import date ranges

* site is already in assigns

* recompute cutoff date each time

* use toDate(timestamp[, timezone]) shortcut

* show alreats on export cancel/delete and extract hint into a component

* switch to Imported.clamp_dates/4

* reprocess tables when imports are added

* recompute cutoff_date on each call

* actually use clamped_date_range on submit

* add warning message

* add expiry rules to buckets in make minio

* add site_id to imports notifications and use it in csv_importer

* try/catch safer

* return :ok

* date range is not available when no uploads

* improve ui and warning messages

* use Generic.notice

* fix flaky exports test

* begin tests

* Improve `Importer` notification payload shape

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-04-09 20:59:48 +02:00
Adrian Gruntkowski
bb108450cb
Fix flaky google import API tests due to hardcoded import ID (#3994) 2024-04-09 19:55:26 +02:00
hq1
884daa7943
Remove hostname filter from the external API (#3991) 2024-04-09 18:03:06 +02:00
Karl-Aksel Puulmann
441412a164
Return 400 when using invalid filters for stats api (#3986)
Currently a 500 is returned instead and logged to sentry.
2024-04-09 15:36:17 +03:00
Adrian Gruntkowski
14b00c6ac3
Fix dry run mode in DataMigration.SiteImports (#3988) 2024-04-09 13:04:17 +02:00
Adrian Gruntkowski
1c1ea95e16
Ensure only complete imports are considered in site imports data migration (#3987)
* Ensure only complete imports are considered in site imports data migration

* Refactor `SiteImports` data migration for clarity (h/t @RobertJoonas)

* Fix tests
2024-04-09 11:49:28 +02:00
Adrian Gruntkowski
d796788715
Keep sites.imported_data in sync with backfilled SiteImport when migrating (#3979)
* Keep `sites.imported_data` in sync with backfilled `SiteImport` when migrating

* Consider only completed site imports in data migration
2024-04-09 09:04:51 +02:00
ruslandoga
94deb89b9d
remove Plausible Team footer from self-hosted emails (#3980)
* remove Plausible Team footer from self-hosted

* don't test unsubscribe placeholder in small build
2024-04-09 09:04:23 +02:00
Adrian Gruntkowski
b951065724
Refactor Imported.check_dates (->clamp_dates) for better felxibility (#3983) 2024-04-09 09:04:11 +02:00
Karl-Aksel Puulmann
a6d4786959
Worker to clean site data from ClickHouse (#3959)
* Create a worker to clean clickhouse deleted sites data

The plan is to run this weekly, but going to trigger it manually the first few times on cloud

* Make asserting count more reliable

* credo

* PR feedback

* Fixes
2024-04-08 12:26:38 +03:00
Adrian Gruntkowski
a7603c9e49
Improve import procedure to ensure no time range overlaps (#3970)
* Always scope import ID by site as well

* Do not schedule new import job if there are any site imports in progress

* Disable import buttons when any import is in progress

* Simplify `schedule_job/4` (h/t @RobertJoonas)
2024-04-04 18:56:36 +02:00
Adrian Gruntkowski
cffff0340c
Handle Google API timeouts gracefully during imports (#3975) 2024-04-04 18:55:39 +02:00
Adrian Gruntkowski
33eed9d7db
Delete imports which have no stats (#3972) 2024-04-04 18:55:14 +02:00
hq1
f9f0407d68
Remove experimtnal_hostname_filter and keep it on by default (#3973)
* Remove `experimental_hostname_filter` and keep it on by default

* Catch up with changes done via e5b56dbe6
2024-04-04 17:20:16 +02:00
RobertJoonas
e5b56dbe62
Refactor VisitorGraph (#3936)
* Give a more semantic name to a function

* Make the LineGraph component thinner

* Move LineGraph into a separate file

* Move interval logic into interval-picker.js

This commit also fixes a bug where the interval name displayed inside
the picker component flickers the default interval when the graph is
loading.

The problem was that we were counting on graphData for returning us the
current interval: `let currentInterval = graphData?.interval`

We should always know the default interval before making the main-graph
request. Sending graphData to IntervalPicker component does not make
sense anyway.

* extract data fetching functions out of VisitorGraph component

* Return graph_metric key from Top Stats API

This commit introduces no behavioral changes - only starts returning an
additional field, allowing us to avoid the following logic in React:

1. Finding the metric names, given a stat display name. E.g.
   `Unique visitors (last 30 min) -> visitors`

2. Checking if a metric is graphable or not

* Move metric state into localStorage

This commit gets rid of the internal `metric` state in the VisitorGraph
component and starts using localStorage for that instead.

This commit also chains the main-graph request into the top-stats request
callback - meaning that we'll always fetch new graph data after top stats
are updated. And we do it all in a single function.

Doing so simplifies the loading state significantly, and also helps to
make it clear, that at all times, existing top stats are required before
we can fetch the graph. That's because the metric is determined by which
Top stats are returned (for example, we can't be sure whether revenue
metrics will be returned or not).

* Make sure graph tooltip says "Converted Visitors"

* Extract a StatsExport function component

Again, instead of relying on `graphData?.interval` we can read it from
localStorage, or default to the largest interval available. The export
should not be dependant on the graph.

* Extract SamplingNotice function component

* Extract WithImportedSwitch function component

* Stop "lazy-loading" the graph and top stats

Since the container is always on top on the page, it will be visible on
the first render in any case - no matter the screen size.

* Turn VisitorGraph into a function component

* Display empty container until everything has loaded

* Do not display loading spinner on realtime ticks

* Turn Top Stats into a fn component

* fetch top stats and graph async

* Make sure revenue metrics can remain on the graph

* Add an extra check to canMetricBeGraphed

* fix typo

* remove redundant double negation
2024-04-04 13:39:55 +01:00
Karl-Aksel Puulmann
3115c6e7a8
Reducing JOINs in queries (#3966)
* Move experimental_session_count? logic to within query object

* WIP new querying system for deciding what tables to query

* both -> either

* Include sample_percent in both tables

* Remove a hanging TODO

* Allow filtering by visit props on event queries if flag is on

* Make default sessions join more conditional

* Simplify events_join_sessions?

* Add some TODOs

* Fix assignment

* Handle entry/exit page visit props separately from props stored in events table

* Update test which created sessions/events differently from everyone else

* Make query_events private

* Dont filter by session properties on events table if querying sessions and joining in events

* Handle visits, pageviews, events and visitors metrics from other table

* both -> either

* events, pageviews are strictly event metrics

* Add support for (plain) breakdowns deciding which table to use

* Run tests with experimental_reduced_joins as a separate job

Also refactor which tests are run with postgres:15 to reduce number of jobs

* moduledocs for TableDecider

* Fix matrix

* Custom build name

* Move TEST_EXPERIMENTAL_REDUCED_JOINS check

* Handle percentage separately from other metrics

* Remove debug code

* TableDecider tests

* both => sample_percent

* Improve naming

* Simplify code

* Breakdowns retain old behavior if getting metric visitors

* Unify behavior of entry/exit page hostnames with rest

* Fix test naming
2024-04-04 13:54:23 +03:00
hq1
6af80dd246
Filter by hostnames (#3963)
* CH Migration: exit/entry hostnames in sessions_v2

* Leave only exit_page_hostname, we already record hostnames

* Use ClickHouse DDL in favour of ecto so that cluster is included

* Compress with ZSTD(3)

* Expose Hostname filter in the dashboard dropdown

* Add `exit_page_hostname` to ClickHouse `sessions_v2` schema

* Start tracking hostname changes in sessions

* Implement hostname filter suggestions

* Enable filtering by `event:hostname`

* Add tests for filtering by hostnames

* Ensure filter suggestions work for exit pages too

* Allow overriding hostnames with `send_pageview` mix task

* Remove `:window_time_on_page` flag

It seems that we can remove it after all?

* Initialize `experimental_hostname_filter` query parameter

* Rewrite cache store behaviour with regards to session hostnames

* Work around inconsistent session merging

So that `populate_stats` can get closer to actual ingestion

* Improve top stats test

* Make it possible to filter sessions by entry/exit hostnames

* Update pages tests

* Expose `experimental_hostname_filtering` temporarily in the UI

* Untested yet: also apply experimental filtering to sources

* Introduce `hostname_filter` feature flag

* Format

* Test top sources with hostname filter + experimental flag
2024-04-04 10:48:30 +02:00
Adrian Gruntkowski
e6d83e946f
Populate new columns in imports and exports (#3969)
* Extend `Imported*` schemas with newly added columns

* Populate newly added `Imported*` fields in GA4 imports

* Extend exports with newly added fields

* Extend CSV importer to ingest new fields

* Fix alias shadowing error

* Add more extensive GA4 import fixtures

* Apply rounding and casting to sampled visits
2024-04-04 10:33:19 +02:00
Adrian Gruntkowski
9f27fa303c
Fix dry run mode in DataMigration.SiteImports (#3965) 2024-04-02 14:05:34 +02:00
Adrian Gruntkowski
23a3699dd7
Improve import stats toggle and with_imported flag computation (#3960)
* Check import presence across all imports and not just the first one

Also, simplify imported data toggle rendering to not explicitly
refer to the earliest import source.

* Change imported stats toggle icon in dashboard

* Test `Imported.get_imports_date_range/1`

* Simplify failed UA/GA import email copy
2024-04-02 12:53:19 +02:00
Adrian Gruntkowski
71fe541359
Implement script for backfilling legacy site import entries and adjusting end dates of site imports (#3954)
* Always select and clear import ID 0 when referring to legacy imports

* Implement script for adding site import entries and adjusting end dates

* Log cases where end date computation is using fallback

* Don't log queries when running the migration to reduce noise
2024-04-02 12:53:02 +02:00
Adrian Gruntkowski
5bf59d1d8a
Implement adjusting imported date range to actual and existing stats (#3943)
* Implement adjusting imported date range to actual and existing stats

* Drop redundant prefix from import list entries

* Make pageview numbers in imports list formatted for readability

* Test and improve date range cropping

* DRY UA and GA4 stats start and end date API calls

* Extend UA/GA import controller tests and improve error handling

* refactor finding longest open range without existing data

* Fix typo in test description

Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>

* Rename `open_ranges` to `free_ranges`

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
2024-03-28 09:32:41 +01:00
hq1
b31433a7bf
Ensure all the react container attributes are strings (#3948) 2024-03-26 11:01:59 +01:00
hq1
edf70d14b6
Use sessionStorage for "dashboard first launch" banner tracking (#3892)
* Use sessionStorage for offer e-mail report banner tracking

Keeping it within the cookie is problematic, as the banners don't
expire and overflow the cookie with data when enough new sites
are added.

Ref https://github.com/plausible/analytics/issues/3762

* Update changelog

* Extract a component

* Make is_dbip evaluate to quoted boolean
2024-03-26 09:49:15 +01:00
Karl-Aksel Puulmann
4af7019011
Ignore sessions without entry/exit pages when breaking down entry/exit pages (#3933)
* Ignore sessions without entry/exit pages when breaking down entry/exit pages

* Update stats controller tests to have more realistic test data (pageview followed by event)
2024-03-26 09:01:07 +02:00
hq1
2fae0146a4
Reapply 3918 (#3940)
* Reapply "Pages shield (#3918)"

This reverts commit 33b5c10654.

* Make the FF check work against the site actor
2024-03-25 10:36:22 +01:00
hq1
9989ce6927
Migration for 3918 (#3939)
* Revert "Pages shield (#3918)"

This reverts commit 53f94a9f82.

* Migration: Shield page rules
2024-03-25 10:19:50 +01:00
hq1
53f94a9f82
Pages shield (#3918)
* Migration: Shield page rules

* Add Ecto schema for Page Rules

* Add Page Rule cache

* Fix typo

* BTW: Use already imported function

* Extend Shields context interface + split existing tests

* Ingestion: filter matching patches + refactor shield actions

* Add LV section for adding Page Rules

* Validate max page path length

* Put Pages Shield behind a feature flag

* Update CHANGELOG

* Update docs link anchor

As per https://github.com/plausible/docs/pull/477

* Update lib/plausible/shields.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update lib/plausible_web/live/shields/page_rules.ex

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>

* Update lib/plausible_web/live/shields/page_rules.ex

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
2024-03-25 09:48:56 +01:00
Adrian Gruntkowski
ba5b80a8c0
Add label to site imports and populate it (#3914) 2024-03-22 11:17:02 +01:00
RobertJoonas
d6e1e8bebd
Put total conversions on the graph + goal-filtered CSV export improvements (#3929)
* Add validation for the events metric in main_graph

* Test the already existing events metric support in main-graph

* Put total conversions on the graph

* extract main_graph_csv function (refactor only)

* add total_conversions and conversion_rate to goal-filtered visitors.csv

* update changelog
2024-03-22 09:35:23 +00:00
Uku Taht
fd879eeb16
Store referrers from android apps (#3715)
* Store referrers from android apps

* Add test for unknown referrer protocol

* Store android referrer protocol
2024-03-21 17:45:34 +02:00
RobertJoonas
c32779a3e5
Timeseries for conversion rate (#3919)
* add conversion rate to Stats API timeseries

* make sure CR can be queried as the only metric

* add a test asserting zeros are returned

* add tests for filtering by other properties at the same time

* Remove unnecessary validation of params

1. It doesn't make to validate `interval` (and its granularity) in all
   endpoints. It's only relevant for the main graph.

2. The plug (renamed to `date_validation_plug`) already makes sure that
   the dates are validated. No need to call the same function again in
   Top Stats and Funnel endpoints.

* add metric validation to main graph

* Add tests for main graph API

* put conversion rate on the graph

* update changelog

* Add revenue metrics into metrics.ex

* make fn private

* avoid setting graph metric to visitors in goal-filtered view
2024-03-21 13:58:00 +00:00
Adrian Gruntkowski
d6e81670e4
Unify UA and GA4 import flow into one (#3888)
* Unify GA4 and UA import flow into one

* Clean up property and view data retrieval via Google HTTP APIs

* Turn `Map.get` into `Map.fetch!` in API response processing code

* Bump list account summaries page size limit to max of 200

* Show only views in legacy flow and fix legacy redirect after import start

* Move google analytics import actions tests to a separate module

* Extend Google Analytics controller tests

* DRY up `property?` predicate (h/t @RobertJoonas)
2024-03-21 11:37:10 +01:00
ruslandoga
5f9465614b
Include domain and dates in zip archive filename (#3921)
* include domain and dates in zip archive filename

* adapt to comments
2024-03-21 11:35:42 +01:00
Karl-Aksel Puulmann
32ab138301
Fix issue with name clash (#3925)
Unexpectedly, table.name caused a name clash after CR refactor, so using a unique name
for the output column

Sentry issue: https://sentry.plausible.io/organizations/sentry/issues/5612
2024-03-21 10:11:29 +00:00
Karl-Aksel Puulmann
c219652dae
Re-apply Move conversion_rate logic from elixir to clickhouse (#3924)
* Revert "Revert "Move conversion_rate logic from elixir to clickhouse (#3887)"…"

This reverts commit 253fb5d67d.

* Fix issue with missing columns

The issue came from refactoring event:goal UNION ALL logic and trying to move
name select from first to last. If any other tables were joined, the incorrect
item would be used as an array index, causing this issue.

Added a relevant test.
2024-03-21 10:48:41 +02:00
Karl-Aksel Puulmann
253fb5d67d
Revert "Move conversion_rate logic from elixir to clickhouse (#3887)" (#3923)
This reverts commit 1909743b90.
2024-03-21 09:53:31 +02:00
Karl-Aksel Puulmann
1909743b90
Move conversion_rate logic from elixir to clickhouse (#3887)
* Separate out query building from pagination/execution logic.

* Refactor pageview_goals breakdown query, removing index column from results

* Remove zip_columns logic

* Use common pagination util

* Do everything in a single query for breakdowns for goals

* Order in DB

* Make sure column order is identical

* Calculate CR within the goal breakdown query

* Calculate CR for property breakdowns

* WIP: Calculate group CR

* CR with order_by

* Compatibility fix

* Import Ecto.Query and cleanup

* handle total_visitors the same way as add_percentage

* Handle conversion_rate in aggregate.ex

* Solve rebase fail

* Simplify maybe_add_group_conversion_rate

* Add conversion_rate defaults to 0 test

* Add test for conversion_rate should not be calculated with imported data (failing here and on master)

* Dont include imported data when breakdown by prop or goal

* Remove revenue_nils
2024-03-21 09:38:44 +02:00
ruslandoga
279e89c693
CSV imports (no UI) (#3895)
* encode/decode date range in filenames

* Update lib/plausible/imported/csv_importer.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update lib/plausible/imported/csv_importer.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* drop unused functions

* send failure email if there is no data to export

* use PlausibleWeb.Email.mailer_email_from()

* ensure we get dates from minmax date query

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-03-19 12:06:47 +01:00
ruslandoga
4242b52be4
Allow importing extra config (#3906)
* allow importing extra config

* changelog

* fix typo

* add test
2024-03-19 12:02:52 +01:00
Karl-Aksel Puulmann
02d2256483
Set exit_page only on pageviews (#3870)
* Set exit_page only on pageviews

* Update tests

* Update entry_page on first pageview

* Update CHANGELOG.md
2024-03-18 11:11:15 +02:00
ruslandoga
07b714a143
Update Sentry (#3843)
* update Sentry

* Sentry.HTTPClient.child_spec is now optional

* Sentry.EventFilter is deprecated

* update sentry to 10.2.0

* fix dialyzer warnings
2024-03-18 10:10:20 +01:00
ruslandoga
5e74b1cf74
CSV exports (UI) (#3875)
* ui

* fix redirect link

* improve make minio

* use implicit button form for csv export

* add exports_bucket helper

* read S3_EXPORTS_BUCKET

* supply s3_bucket in export_csv job args

* make plausible_minio use unprotected port

* move s3_csv_export queue to base queues

* Update lib/plausible_web/controllers/site_controller.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-03-18 08:52:57 +01:00
hq1
59afa20955
Reapply #3878 + bugfix hit rate tracking (#3891)
* Reapply "Replace caching engine (#3878)" (#3883)

This reverts commit c5881cdc6d.

* Ensure hit rate is tracked on `get_or_store`

* Remove :wx and :observer

* Remove unused deps

* Use `:set` table type
2024-03-14 08:06:12 +01:00
RobertJoonas
e8f3946dde
Fix division by zero in imported queries (#3890)
* prevent division by 0 in merge_imported queries

* Revert "fix bounce_rate change bug (#3886)"

This reverts commit 6eef32a8ff.

After 02aa0b2, we can keep on assuming that bounce rate is always numeric.
2024-03-13 10:37:14 +00:00
Adrian Gruntkowski
4d7d88cfec
Implement basics of GA4 import (#3851)
* Implement LV date input using flatpickr

* Implement basics of GA4 import (very dirty WIP)

* Split Google HTTP API into UA and GA4 specific parts

* Add a quick way to record GA4 API responses

* Add first GA4 import fixtures with GA4 Data API responses

* Extract GA4 and UA specific logic form Google API

* Extract UA and GA4 specific actions to distinct controllers

* Add integration test for GA4 importer

* Update GA4 fixtures

* Test GA4 API

* Add debug logging and fix paginating through API results in in GA4 import

* Revert "Implement LV date input using flatpickr"

This reverts commit c696f8ee39d5702f27015c09a4f079ca124cc7bb.

* Fix note
2024-03-12 18:08:25 +01:00
ruslandoga
f2350b5165
Add /tmp/ to .gitignore and simplify s3 cleanups in tests (#3889) 2024-03-12 17:58:05 +01:00
ruslandoga
5a3072ca21
CSV exports (no UI) (#3836)
* csv exports

* use ex_unit's tmp_dir
2024-03-12 17:27:27 +01:00
RobertJoonas
7641c66a2b
Stats api time on page (#3858)
* add metric validation + support in aggregate

* add a test ensuring comparison works

* disallow time_on_page with a goal filter

* Return time_on_page as `nil` from aggregate API

In case time_on_page cannot be calculated, we'll return it as `nil` from
the Stats API.

This is to make the behaviour consistent between breakdown and aggregate
endpoints. As for the UI, we'll still continue to report time_on_page as
0 - not changing any UI behaviour as discussed with Marko.

* add tests for time_on_page in event:page breakdown

* update changelog

* invalidate time_on_page with event:name filter

* add the ability to only query time_on_page in page breakdown

We'll need the visitors metric to get the list of pages to calculate the
time_on_page for.
2024-03-12 10:00:32 +00:00
hq1
c5881cdc6d
Revert "Replace caching engine (#3878)" (#3883)
This reverts commit 437a3350ff.
2024-03-12 08:30:16 +01:00
hq1
437a3350ff
Replace caching engine (#3878)
* Dependencies: swap Cachex for ConCache

* Implement Cache adapter wrapping ConCache

* Implement cache stats tracker, for metrics

* Use Cache.Adapter in Plausible.Cache

Marking the test as not slow anymore

* Use Cache Adapter when tracking sessions

* Use Cache Adapter for UA parsing

* Rename child identifiers - cachex is obsolete now

* Test stats tracking

* Update grafana metrics

* Put all caches under common child specification

* Try less

* Shorten the function delegation path
2024-03-12 07:58:12 +01:00
Karl-Aksel Puulmann
a9d3c03782
Validate the same metric isnt queried multiple times in external stats API (#3871)
* Validate the same metric isnt queried multiple times in external stats API

Issue: https://3.basecamp.com/5308029/buckets/35611491/card_tables/cards/7161347855

* Changelog entry

* Make credo happy
2024-03-08 10:46:18 +02:00
Karl-Aksel Puulmann
0cdba7d407
Fix broken tests (#3867) 2024-03-06 17:14:04 +02:00
Karl-Aksel Puulmann
c6d98397a8
Move add_percentage logic into clickhouse (#3854)
* Remove `add_percentage`, calculate percentages in clickhouse queries

This simplifies querying logic and avoids doing extra queries and avoids
race conditions.

* Remove special none handling from breakdowns, handling percentages correctly

* Add (failing) test showing expected add_percentage behavior for user making multiple sessions

* Update add_percentage behavior to use separate subqueries
2024-03-06 11:08:25 +02:00
Karl-Aksel Puulmann
c60a2faee4
Write event table session columns (#3865)
* Write event table session columns

* Update testing factory rig
2024-03-06 10:59:24 +02:00
Karl-Aksel Puulmann
8d977e0f76
Tests: session properties without the prefixes (#3863)
* Undo event session attributes renaming

* Rename session_ attributes in tests
2024-03-05 12:44:33 +02:00
Karl-Aksel Puulmann
d5048fd6b4
Stop writing session properties into events table (#3800)
* Refactor: Explicitly add field names to INSERT

This avoids issues when code schema is out of sync with real schema

* Dont write session parameters to events

These would only be stored on first event anyways. Work remains to be done
on tests which have their own helper

* Remove writes to country_code in a test

* Remove old columns from being accessible in elixir code

* Update most tests to use new way of adding session props to events

* Update testing harness

* Update stats controller test

* Update for shield rules

* update breakdown tests

* Fix typing of state for dialyzer

* Drop support for old session attributes code

* Update remaining tests

* cond -> if
2024-03-01 10:53:56 +02:00
hq1
5eb9d724e5
Plugins API: mark data_domain as nullable in capabilities schema (#3840)
* Plugins API: mark data_domain as nullable in capabilities schema

* fixup
2024-02-28 10:26:01 +01:00
Adrian Gruntkowski
39aa81a16f
Implement UI for multiple imports (#3727)
* Create a stub of site settings section for imports and exports

* Use legacy site import indication to determine UA import handling

* Add provisional logos for upcoming import sources

* Stub basics of import page

* Add very rudimentary support for multiple UA imports

* Implement imports list as live view

* Add support for opening LV modal from backend and closing from frontend

* Introduce notion of themes to `button` and `button_link` components

* Add confirmation modal on deleting import

* Swap GA4 logo

* Implement disabled state support for `button_link` component

* Disable export and non-implemented import sources

* Use native starts start date for upper boundary of import time range

* Ensure integrations view uses legacy UA import flow

* Remove unnecessary preload in SiteController

* Remove unnecessary exception for legacy imports

* Move API controller stats tests under PlausibleWeb

* Test listing imports

* Add test for explicit listener setup

* Add tests for legacy flag state in UA importer

* Add test for purging legacy import data

* Add tests for `Sites.native_stats_start_date`

* Test forgetting imports

* Add `Stats.Clickhouse.imported_pageview_counts/1` and fix test flakiness

* Show page view counts on imports list

* Add tests for static imports and exports view

* Adjust button look slightly

* Use `case` instead of `cond`

* Make feature flag customisable per site

* Fix buttons and empty state styling

* Add another import to seeds

* Use JS confirm dialog instead of modal for deletion confirmations

* Revert "Add support for opening LV modal from backend and closing from frontend"

This reverts commit 260e6c753032b451542e24be9edc2118790b5a00.

* Default `legacy` to false when inserting new import jobs

* Drop `method` attribute from `button_link` and `unstyled_link` components
2024-02-28 09:34:04 +01:00
ruslandoga
f3423aefec
Add csv importer (#3795)
* add csv importer

* make table validation explicit

* update some docs

* improve docs

* add minio container to ci

* more tests

* eh

* continue

* add passing test

* add failing test

* add config test

* add minio to Makefile

* testcontainers

* remove extra whitespace

* explain the implementation a bit

* account for async deletes in tests

* bounces is UInt32

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-02-27 14:19:09 +01:00
ruslandoga
31cf3e54f8
Add Bamboo.Mua (#3654) 2024-02-27 14:18:36 +01:00
hq1
f1b6a672d4
Fix added_by not saving on adding country rules (#3835)
* Fix `added_by` not saving on adding country rules

* Format

* Remove dupe attr
2024-02-27 12:36:16 +01:00
hq1
518cdb3307
Shield: Country Rules (#3828)
* Migration: add country rules

* Add CountryRule schema

* Implement CountryRule cache

* Add country rules context interface

* Start country rules cache

* Lookup country rules on ingestion

* Remove :shields feature flag from test helpers

* Add nested sidebar menu for Shields

* Fix typo

* IP Rules: hide description on mobile view

* Prepare SiteController to handle multiple shield types

* Seed some country shield

* Implement LV for country rules

* Remove "YOU" indicator from country rules

* Fix small build

* Format

* Update typespecs

* Make docs link point at /countries

* Fix flash on top of modal for Safari

* Build the rule struct with site_id provided up-front

* Clarify why we're messaging the ComboBox component

* Re-open combobox suggestions after pressing Escape

* Update changelog

* Fix font size in country table cells

* Pass `added_by` via rule add options

* Display site's timezone timestamps in rule tooltips

* Display formatted timestamps in site's timezone

And simplify+test Timezone module; an input timestamp converted
to UTC can never be ambiguous.

* Remove no-op atom

* Display the maximum number of rules when reached

* Improve readability of remove button tests

* Credo

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-02-27 12:08:13 +01:00
RobertJoonas
dd428430f5
Query imported data for views_per_visit (#3830)
* query imported data for views_per_visit

* changelog update
2024-02-26 15:54:49 +00:00
RobertJoonas
52f584efa9
Group os_version by os (#3806)
* improve test

* add os to os_version breakdown

...and add operating_system_versions.csv to the CSV export

* fix conversion rate for os_version breakdown

* update changelog

* fix existing CSV tests

* use case instead of cond
2024-02-22 15:58:50 +00:00
RobertJoonas
d74b1d5e60
Reapply conversion rate into Stats API + bugfixes (#3805)
* Revert "Revert api conversion rate (#3789)"

This reverts commit 8e8790dd30.

* fix browser_version CR breakdown bug

* changelog bugfix

* inspect data structures before sending to sentry
2024-02-21 15:53:05 +00:00
hq1
6035618213
Add GET /capabilities to Plugins API (#3808)
* Add `GET /capabilities` to Plugins API

It aims to:

 - help the client verify the data-domain the token is associated with
 - list all the features available for the site's owner
   (and therefore determine availability of the subset of those for the current
   Plugins API caller)

The endpoint does not require authentication, in the sense that it'll
always respond with 200 OK. However when the token is provided,
a verification lookup is made.

* Remove IO.inspect() call

* Credo

* Aesthetics

* s/send_resp/send_error/

* Call preload just once
2024-02-21 12:41:56 +01:00
Karl-Aksel Puulmann
5cdca6f408
Remove code that updates session browser/geo/OS attributes (#3796)
This will makes it impossible to store session attributes on events.

Looking at production data also revealed no cases where these updates
are effective.
2024-02-20 09:35:12 +02:00
hq1
eceac8afd5
Allow inviting users who are members already (#3797)
* Allow e-mail exclusion in team members quota

* Exclude invitee from quota on invitation create

* Enable invitation submission but report errors on quota violation

* Use a single interface for team members quota

* Check the `Keyword.validate/2` result

* Update test/plausible_web/controllers/site/membership_controller_test.exs

Co-authored-by: Uku Taht <Uku.taht@gmail.com>

---------

Co-authored-by: Uku Taht <Uku.taht@gmail.com>
2024-02-19 12:12:31 +01:00
RobertJoonas
8e8790dd30
Revert api conversion rate (#3789)
* Revert "Unify percentage change for CR and bounce_rate (#3781)"

This reverts commit a6b1a6ebc7.

* Revert "Bring Stats API up to speed: Add `conversion_rate` to Aggregate and Breakdown (#3739)"

This reverts commit 672d682e95.
2024-02-15 17:43:35 +00:00