Commit Graph

901 Commits

Author SHA1 Message Date
ruslandoga
9687fa786d
remove Plausible Analytics metions from CE (#4121)
* remove Plausible Analytics metions from CE

* update tests

* still mention Plausible Analytics on landing page
2024-05-23 09:43:01 +02:00
hq1
cd4d1d06bb
Rename Plugins API Token(s) to Plugin Token(s) (#4075)
Deployment needs to be coordinated with WordPress plugin
update. Reasoning: there's too much confusion coming
from the use of the "API" terminology, leading to wrong
keys being generated by users.
2024-05-22 07:45:09 +02:00
Adrian Gruntkowski
2d23db2982
Return warning when imported stats are skipped in Stats API due to unsupported query (#4116) 2024-05-21 11:45:18 +02:00
Adrian Gruntkowski
7876802d05
Improve CSV importer tests and visit duration query for imported data (#4114)
* Tidy up breakdown helper functions in CSV importer tests

* Fix a typo

* Extend city breakdown tests in CSV importer test suite

* Make visit duration computation consistent across native and imported data
2024-05-21 09:24:10 +02:00
Adrian Gruntkowski
756d9c28ce
Improve test coverage and fix stats API and dashboard CSV export (#4110)
* Fix broken hostname property and handle missing imported metrics gracefully

* Add test for CSV export of imported data

* Add extra coverage for property and metric combox which were failing

* Compute visit duration and bounce rate for exit pages in imported data

* Drop support for breaking down by `event:hostname` property for now
2024-05-20 16:11:35 +01:00
RobertJoonas
df178ea2d5
Introduce :skip_imported_reason field into Query struct (#4115)
* introduce new query field

* improve tests
2024-05-20 15:46:25 +01:00
Karl-Aksel Puulmann
cfdd769984
Fix StatsAPI access for new accounts on business tier (#4105)
Currently, business tier users created after business tier launch can't
access Stats API due to faulty grandfathering logic. This change should
fix that.
2024-05-15 11:50:41 +03:00
Adrian Gruntkowski
9374a95cf2
Export and import custom events via CSV (#4096)
* Export and import custom events via CSV

* Add prop support of url for cloaked links and path for 404s in imported queries

* Handle custom events with empty URL and path properties gracefully

* Make events with properties logic DRY and fix missed cloaked link

* Add test for path property breakdown

* Update raw CH data fixture and extend CSV importer tests

* Fix broken query condition after rebase

* Update CHANGELOG.md
2024-05-14 13:50:30 +02:00
Karl-Aksel Puulmann
baa99652f6
Refactor internal Query schema and introduce WhereBuilder (#4082)
* New struct format for query after parsing

* WIP refactoring

* WIP: Validations working

* WIP: tuple to list

* continued refactoring

* WIP: parsing defaults

* Breakdown tests pass

* Window functions fix

* Fix default

* Remove dead argument

* Update filters tests

* Update query_test.exs

* Fix table_decider

* sources tests pass

* Filter suggestions fix

* revenue/goal filter applied refactor

* Update top_stats matching

* Get stats_controller tests passing

* Update neighbor_aggregate_time_on_page

* Refactor Query.remove_event_filters into Query.remove_filters, add new callsites

* Move goal where clause building to new WhereBuilder module

* Move event:name filters

* Move more filters to WhereBuilder

* Update fragment to allow non-static meta columns

* Build where clause for events table using WhereBuilder

* Build sessions table where clause using WhereBuilder

* Move time range filtering and site checking to WhereBuilder

* WhereBuilder.build_condition method

* Remove TODO

* _rest pattern for TableDecider, Query pattern matching

Future-proofing in a tiny way

* Hacky fix to get tests passing for Google API tests

* Typespec fix

* Merge conflict

* refactor special goal filter logic in imported.ex

* Docs feedback

* put_filter

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
2024-05-14 11:58:10 +03:00
Uku Taht
06e8118dab
Filtering Search Console keywords (#4077)
* Apply filters in search console request

* Remove dead code from search console modal

* Remove unimportant information from keyword modal

* Show invalid filters from search console

* Fix tests

* Add/Fix tests

* Fix typo

* Remove unused variable

* Fix typo

* Changelog entry

* Fix Credo

* Display impressions, CTR and position in keyword modal

* Undo change that should not have been committed

* Fix test

* Fix test

* filters -> search_console_filters
2024-05-14 09:56:55 +03:00
ruslandoga
1f4346f4df
Add DATA_DIR (#4100)
* add DATA_DIR

* add test

* changelog

* fix test in CI where PERSISTENT_CACHE_DIR is always set

* consistent fallback
2024-05-13 10:17:56 +02:00
ruslandoga
7af8273702
Handle s3 timeout on settings page (#4036)
* handle s3 timeout

* no-async approach

* fewer changes

* add tests

* make s3 failure test EE-only
2024-05-13 10:17:27 +02:00
RobertJoonas
a6dcd19ccc
Autoconfigure event goals (#4093)
* add new goal suggestions API

* silence credo

* Order suggestions from subqueries explicitly

* allow autoconfiguring goals

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Fix form modal tab switching behavior

* add test

* Remove redundant and invalid action link title

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-05-10 11:48:27 +02:00
Adrian Gruntkowski
6bbc8d69a4
Avoid crashing on empty GA4 HTTP report response (#4094) 2024-05-10 09:12:22 +02:00
Adrian Gruntkowski
4e7e932a75
Add support for imported custom events (#4033)
* Add Ecto schema for imported custom events

* Start importing custom events from GA4

* query imported goals

* make it possible to query events metric from imported

* make it possible to query pageviews in goal breakdown

* make it possible to query conversion rate

* fix rate limiting test

* add CR tests for dashboard API

* implement imported link_url breakdown

* override special custom event names coming from GA4

* allow specific goal filters in imported_q

* update GA4 import tests to use Stats API

* Improve tests slightly

* Update CHANGELOG.md

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
2024-05-09 13:13:19 +01:00
Adrian Gruntkowski
d8435f2e01
Remove imports_exports and csv_imports_exports feature flags (#4089) 2024-05-09 10:09:24 +02:00
ruslandoga
84ed7988a8
Drop local_start_date/1 (#4088) 2024-05-09 08:59:23 +02:00
Karl-Aksel Puulmann
0a883f10e7
Refactor: Use common current_visitors code (#4071)
* Use common module for counting current visitors in external stats controller

* Refactor spike notifier, remove now-dead code
2024-05-07 15:03:37 +03:00
Karl-Aksel Puulmann
64850cd00f
Remove maybe_drop_prop_filter (#4066)
* Fix event props paygate

Previous code wasn't properly omitting event property filters from
queries.

Discovered while refactoring the code. Extracting fix from refactor for easier reviewability.

* a

* Drop function
2024-05-07 15:03:16 +03:00
ruslandoga
02d4709be7
add csv fixture for e2e export/import test (#4037)
* add inline csv fixture

* use new csvs

* cleanup csv reading and site_id replacing

* perform comparisons between native and imported queries

* help help help

* help help

* help

* eh

* fin

* exclude export/import e2e test when experimental_reduced_joins flag is enabled

* adapt to new pageviews

* adapt to experimental_reduced_joins

* credo is formatter

* cleanup

* assert bounce rates equal in city breakdown

* fix rebase against master

* clean-up dataset

* update comment

* fix typo

* apply csv changes to the files

* use sessions timestamp for exports' dates

---------

Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
2024-05-07 08:48:22 +01:00
Adrian Gruntkowski
d7ca8d9600
Revert "Debug queries for super-admins (#4010)" (#4073)
This reverts commit dd493fdad2.
2024-05-06 15:05:17 +02:00
Karl-Aksel Puulmann
17f812443d
Return session in each time bucket its active in for hourly/minute timeseries (#4052)
* Fix typo in test name

* Update test_helper, enable experimental_session_count together with experimental_reduced_joins

* Return session in each time bucket its active in for hourly/minute timeseries

The behavior is behind experimental_session_count flag

This results in more accurate visitor showing compared to previous approach of showing each user only
active the _last_ time they did a pageview.

Were not doing this for monthly/weekly graphs due to query performance cost and it having a small effect there.
See also https://3.basecamp.com/5308029/buckets/35611491/messages/7085680123

* Add tests for new behavior

Note the new behavior mimics the old one precisely, these tests fail if only
experimental_reduced_joins is on, but not experimental_session_count

* Type erasure

* Dead comment remove

* Expected_plot change
2024-05-05 11:44:43 +03:00
ruslandoga
972dd5d150
redirect to s3 url when downloading exports (#4002)
* redirect to s3 url

* use new on_ee macro, reduce wait time for email to five seconds
2024-05-02 19:53:12 +01:00
ruslandoga
8712e91bcb
drop time on page from exports (#4051) 2024-05-02 08:29:24 +01:00
RobertJoonas
bfdadc2eee
Include breakdown property in the Query struct (#4053)
* keep breakdown prop in the query struct

* Explicitly ignore property param in aggregate and timeseries

Since parameter validation depends on the breakdown property, we need to
make sure it doesn't have any unexpected effect in endpoints where it's
not expected.
2024-04-30 18:43:46 +01:00
Adrian Gruntkowski
41fef85d29
Implement resumable GA4 imports to work around rate limiting (#4049)
* Add support for resuming import to GA4 importer

* Handle rate limiting gracefully for all remainig GA4 HTTP requests

* Show notice tooltip for long running imports

* Bump resume job schedule delay to 65 minutes

* Fix tooltip styling
2024-04-30 18:06:18 +02:00
hq1
dd493fdad2
Debug queries for super-admins (#4010)
* Debug queries for super-admins

* Fixup

* Update lib/plausible/clickhouse_repo.ex

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>

* Try again with https://github.com/plausible/analytics/pull/3699

It's still clunky 😅

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
Co-authored-by: ruslandoga <doga.ruslan@gmail.com>

* Move headers injection to a separate plug module

* Add tests

* Update repo test

* Format

* Add moduledoc

* Don't assume order in query_log

* Be patient about query_log maybe?

* huh?

* huh2

* Wait longer

* Guard against \x00 in response header - testing on stage

* Fixup

* fixup

* fixup

* s/debug_label/label

* Include `site_id` and `metadata` in `log_comment`

* Tolerate non-serializable log_comment contents

---------

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
2024-04-30 09:57:28 +02:00
hq1
ad9141a9d0
Display tooltips on plan change when limits exceeded (#4048)
* Reapply "Display upgrade tooltips for exceeded limits (#4032)"

This reverts commit 76e910d45c.

* Switch to alpinejs controlled tooltips

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>

* Remove unused selector

* Refactor plan limits warning and extract tooltip component

* Remove redundant check

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-04-29 11:16:15 +02:00
hq1
b2009aa158
Rely on con_cache telemetry (#4019)
* Rely on con_cache telemetry

Now that https://github.com/sasa1977/con_cache/pull/76
is released, we don't have to use low-level operations
to emit hit/miss events.

This PR also wraps cache processes with
a function returning appropriate child specs lists.

Ideally each cache will have its own supervisor/child specs
going forward. This is an intermediate step in that direction.

* Update lib/plausible/application.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Declare caches without warmers with plain child specs

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-04-29 11:00:53 +02:00
RobertJoonas
726fe2d982
fix test (#4038) 2024-04-29 09:30:30 +01:00
hq1
d6824de1ad
Rename internal build symbols (#3942)
* Rename internal build symbols

* Rename remaining + add `on_ce` macro

cc @ruslandoga
2024-04-29 08:05:33 +02:00
Adrian Gruntkowski
ca25b6c764
Bump GA4 import page limit to max 250k (#4045)
* Bump GA4 page limit to 250k

* Handle empty properties list response gracefully
2024-04-26 20:41:42 +02:00
Adam Rutkowski
76e910d45c Revert "Display upgrade tooltips for exceeded limits (#4032)"
This reverts commit 4a372a1528.
2024-04-23 12:53:26 +02:00
hq1
4a372a1528
Display upgrade tooltips for exceeded limits (#4032) 2024-04-23 10:42:48 +02:00
Adrian Gruntkowski
a8ea4ce54b
Add new actions to CRM (#4030)
* Implement `Auth.TOTP.force_disable/1`

* Add "Reset 2FA" action to users CRM

* Add `Purge.reset!/2` variant allowing to set arbitrary cutoff time

* Add ability to set native stats start time from CRM

* Revert "Add `Purge.reset!/2` variant allowing to set arbitrary cutoff time"

This reverts commit 6f294d5d58.

* Add test for CRM site update action
2024-04-23 10:29:49 +02:00
hq1
148413afbb
Fix dogfooding invitation event (#4023)
* Fix dogfooding invitation event

* Fix small build
2024-04-22 16:43:04 +02:00
Adrian Gruntkowski
3023cb12fd
Introduce active_visitors to imported_pages and start populating it with activeUsers from GA4 imports (#4027)
* Import `activeUsers` into `imported_pages.active_visitors` for GA4

* Add test for active visitors

* Simplify assertion in active visitors test

* Improve assertion for active visitors further
2024-04-22 10:18:16 +02:00
Adrian Gruntkowski
fede2f0a8a
Make final touches to Imports & Exports (#4025)
* Make final touches to Imports & Exports

* Change import content copy depending on CSV imports and exports flag state

* Remove unused aliases
2024-04-19 11:40:13 +02:00
Adrian Gruntkowski
c10580777e
Remove references to site.imported_data (#4006)
* Remove references to `site.imported_data`

* Count pre-existing ID 0 imports when showing pageview count summary for legacy imports

* Fix tests after rebase

* Dry `delete_imported_stats!`

* Clean up remaining imported data references and add notes
2024-04-19 11:15:51 +02:00
Adrian Gruntkowski
9bae3ccce3
Improve UI/UX of imports view and GA import flow (#4017)
* Add runtime config option for enabled/disabling csv imports and exports

* Use the new option to toggle rendering exports UI

* Disable import buttons when at maximum imports or when option disabled for CSV

* Improve forms for GA import flow

* Add test for maximum imports reached

* Remove "Changed your mind?" prefixing back button

* Hide UA imports in Integrations when `imports_exports` flag is enabled

* Implement `csv_imports_exports` feature flag

* Revert "Add runtime config option for enabled/disabling csv imports and exports"

This reverts commit e30f202dd3.

* Send import notification email only to the user who ran the import

* Improve rendering of disabled button state

* Put import status heroicon in front of import label
2024-04-18 12:12:48 +02:00
RobertJoonas
3a371fdf4d
Test new imported metrics (GA4) (#4014)
* test fixture imported data with stats requests

* take visits metric from the events table in event:page breakdown

* Remove assert_referrers after all

pageReferrer is an event scoped property in GA4, which when queried
along with session-level dimensions will return unexpected data.

Adding the pageReferrer dimension to the GA4 Data API request, it will
cause the selected metric totals to increase significantly, even though
they shouldn't.

* Adjust sources and utm_mediums assertions

* adjust assert_pages

* Make formatter happy

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-04-18 12:12:24 +02:00
Adrian Gruntkowski
9849743407
Always sort occupied date ranges in Imported.clamp_dates/3 (#4018) 2024-04-18 11:15:51 +02:00
Adam Rutkowski
2b1fbf0a0e Fix typo (and kick docker build) 2024-04-16 20:57:29 +02:00
hq1
6fb56dc1cc
Stats api hostname filter (#4008)
* Update Stats API tests

* Revert "Remove hostname filter from the external API (#3991)"

This reverts commit 884daa7943.
2024-04-16 20:36:57 +02:00
hq1
f635f0a6d3
Hostnames shield (#3990)
* Add shield hostname rules migration

* Add hostname rule schema

* Initialize hostname rules cache

* Extend Shields context with hostname related functions

* Instrument ingestion pipeline with hostname rule lookups

* Limit hostname suggestions by shield patterns

* Add LiveView for hostname rules management

* Test hostname cache

* Rename feature flag - should be separate from hostname filter

* Remove :shield_pages feature flag

* Update CHANGELOG

* Format

* Update lib/plausible/shield/hostname_rule.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Move tests from `lib/` 🤦

* Use plain `assign` where no short-circuit is necessary

* Fine tune the copy a little bit

* Prevent misplaced tests

* Treat a test with common sense

* Fixup another test that hasn't been really run before

* Make the form hint dynamic depending on rules count

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-04-16 20:30:20 +02:00
RobertJoonas
fce909d041
Improve merge imported (#4003)
* Pass the actual date args that are returned in fixtures (just for clarity)

* Change select key from operating_system to os

* Select (not set) from imported data as done from native stats

* Support breakdown by imported os_version

* Remove dead code

The `query.include_imported` is set to `false` from the start when
filters are included.

* Refactor Plausible.Stats.Imported

Extract a function that does group_by and dimension select, instead of
doing both separately inside the merge_imported function body

* Further refactor of Plausible.Stats.Imported

Get rid of code repetition

* Support breakdown by imported browser_version

* Support breakdown by imported referrer

* Support breakdown by imported referrer

* remove redundant if in select

* use greatest instead of coalesce

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>

* add back the :member filter handling in Stats.Imported

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
2024-04-16 16:58:22 +01:00
Adrian Gruntkowski
c07f00636d
Stop importing page referrer from GA4 (#4012)
* Stop importing page referrer from GA4

* Update GA4 import fixture

* Update fixture-based test
2024-04-16 15:35:36 +02:00
Adrian Gruntkowski
c1c03b729c
Reapply "Local CSV exports/imports and S3/UI updates (#3989)" (#3995) (#3996)
* Reapply "Local CSV exports/imports and S3/UI updates (#3989)" (#3995)

This reverts commit aee69e44c8.

* remove unused functions

* eh, that one was actually used

* ugh, they were both used

---------

Co-authored-by: ruslandoga <67764432+ruslandoga@users.noreply.github.com>
2024-04-11 09:15:01 +02:00
RobertJoonas
5163880968
fix test (#4001)
* fix test

* Update test/plausible_web/controllers/api/stats_controller/suggestions_test.exs

Co-authored-by: hq1 <hq@mtod.org>

* format

---------

Co-authored-by: hq1 <hq@mtod.org>
2024-04-10 12:40:42 +01:00
hq1
378d3bc6f5
Discard sessions switching hostnames for UTM/referrer breakdowns (#4000)
* Discard sessions switching hostnames for UTM/referrer breakdowns

Co-authored-by: Uku Taht <uku.taht@gmail.com>

* Format

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
2024-04-10 11:38:15 +02:00