Commit Graph

2668 Commits

Author SHA1 Message Date
ruslandoga
7af8273702
Handle s3 timeout on settings page (#4036)
* handle s3 timeout

* no-async approach

* fewer changes

* add tests

* make s3 failure test EE-only
2024-05-13 10:17:27 +02:00
ruslandoga
c41e5980fb
Update links to configs (#4080) 2024-05-10 14:00:51 +02:00
RobertJoonas
a6dcd19ccc
Autoconfigure event goals (#4093)
* add new goal suggestions API

* silence credo

* Order suggestions from subqueries explicitly

* allow autoconfiguring goals

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Fix form modal tab switching behavior

* add test

* Remove redundant and invalid action link title

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-05-10 11:48:27 +02:00
Adrian Gruntkowski
6bbc8d69a4
Avoid crashing on empty GA4 HTTP report response (#4094) 2024-05-10 09:12:22 +02:00
RobertJoonas
370aeb4b02
Refactor filter suggestions with a more DRY approach (#4090)
* refactor filter suggestions with a more DRY approach

* Avoid DRYing props string->atom translation

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-05-10 09:12:02 +02:00
Adrian Gruntkowski
4e7e932a75
Add support for imported custom events (#4033)
* Add Ecto schema for imported custom events

* Start importing custom events from GA4

* query imported goals

* make it possible to query events metric from imported

* make it possible to query pageviews in goal breakdown

* make it possible to query conversion rate

* fix rate limiting test

* add CR tests for dashboard API

* implement imported link_url breakdown

* override special custom event names coming from GA4

* allow specific goal filters in imported_q

* update GA4 import tests to use Stats API

* Improve tests slightly

* Update CHANGELOG.md

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
2024-05-09 13:13:19 +01:00
Adrian Gruntkowski
d8435f2e01
Remove imports_exports and csv_imports_exports feature flags (#4089) 2024-05-09 10:09:24 +02:00
ruslandoga
b1c13617b3
Allow empty city geonames in exports (#4079) 2024-05-09 09:17:28 +02:00
ruslandoga
84ed7988a8
Drop local_start_date/1 (#4088) 2024-05-09 08:59:23 +02:00
ruslandoga
1114686a0e
refactor filtering in export queries (#4086) 2024-05-09 08:55:12 +02:00
ruslandoga
953af278d8
Fix type in date_range/2 (#4087) 2024-05-09 08:47:06 +02:00
Cenk Kücük
435d708289
Update join_algorithm (#4078) 2024-05-07 16:35:22 +02:00
Karl-Aksel Puulmann
0a883f10e7
Refactor: Use common current_visitors code (#4071)
* Use common module for counting current visitors in external stats controller

* Refactor spike notifier, remove now-dead code
2024-05-07 15:03:37 +03:00
Karl-Aksel Puulmann
9f6ea00a72
Clean up deleted sites from clickhouse cron (#4065) 2024-05-07 15:03:32 +03:00
Karl-Aksel Puulmann
64850cd00f
Remove maybe_drop_prop_filter (#4066)
* Fix event props paygate

Previous code wasn't properly omitting event property filters from
queries.

Discovered while refactoring the code. Extracting fix from refactor for easier reviewability.

* a

* Drop function
2024-05-07 15:03:16 +03:00
RobertJoonas
c106595be0
Migration: add imported custom events (#4076)
* Add `imported_custom_events` to CH

* remove redundant table setting

* add path column

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-05-07 11:46:27 +01:00
ruslandoga
02d4709be7
add csv fixture for e2e export/import test (#4037)
* add inline csv fixture

* use new csvs

* cleanup csv reading and site_id replacing

* perform comparisons between native and imported queries

* help help help

* help help

* help

* eh

* fin

* exclude export/import e2e test when experimental_reduced_joins flag is enabled

* adapt to new pageviews

* adapt to experimental_reduced_joins

* credo is formatter

* cleanup

* assert bounce rates equal in city breakdown

* fix rebase against master

* clean-up dataset

* update comment

* fix typo

* apply csv changes to the files

* use sessions timestamp for exports' dates

---------

Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
2024-05-07 08:48:22 +01:00
Adrian Gruntkowski
62138e0dad
Apply sampling factor multiplication consistently across CSV export queries (#4074)
* Apply sampling factor multiplication consistently across CSV export queries

* Fix casting

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>

---------

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
2024-05-07 09:15:43 +02:00
Adrian Gruntkowski
d7ca8d9600
Revert "Debug queries for super-admins (#4010)" (#4073)
This reverts commit dd493fdad2.
2024-05-06 15:05:17 +02:00
Adrian Gruntkowski
70ed30e871
Add override on CSV exports for super admins (#4070) 2024-05-06 11:12:47 +02:00
ruslandoga
b0fc527eb2
Run Oban in CE too (#4061) 2024-05-06 10:40:09 +02:00
ruslandoga
c49500204c
use released ecto_ch v0.3.5 (#4054) 2024-05-06 10:39:45 +02:00
ruslandoga
6e3556d2aa
Use sessions timestamp for exports' dates (#4063) 2024-05-06 10:39:00 +02:00
Adrian Gruntkowski
39357150e9
Improve styling of imports list (#4069) 2024-05-06 10:38:37 +02:00
Karl-Aksel Puulmann
035cb3a379
Use ALIAS columns to simplify querying API (#4059)
* Cleanup: remove @api_prop_name_to_db-related cruft

* Use aliases elsewhere in code
2024-05-05 12:03:00 +03:00
Karl-Aksel Puulmann
17f812443d
Return session in each time bucket its active in for hourly/minute timeseries (#4052)
* Fix typo in test name

* Update test_helper, enable experimental_session_count together with experimental_reduced_joins

* Return session in each time bucket its active in for hourly/minute timeseries

The behavior is behind experimental_session_count flag

This results in more accurate visitor showing compared to previous approach of showing each user only
active the _last_ time they did a pageview.

Were not doing this for monthly/weekly graphs due to query performance cost and it having a small effect there.
See also https://3.basecamp.com/5308029/buckets/35611491/messages/7085680123

* Add tests for new behavior

Note the new behavior mimics the old one precisely, these tests fail if only
experimental_reduced_joins is on, but not experimental_session_count

* Type erasure

* Dead comment remove

* Expected_plot change
2024-05-05 11:44:43 +03:00
Karl-Aksel Puulmann
850a843d82
Add migration to add ALIAS columns to common session/visit properties (#4058)
This allows for simplifications in the API code
2024-05-05 11:30:39 +03:00
ruslandoga
972dd5d150
redirect to s3 url when downloading exports (#4002)
* redirect to s3 url

* use new on_ee macro, reduce wait time for email to five seconds
2024-05-02 19:53:12 +01:00
ruslandoga
07cf2b4682
use events_v2 for pageviews export (#4035) 2024-05-02 14:50:31 +01:00
Karl-Aksel Puulmann
bcde2b8cf2
Add Yesterday as an time range option in the dashboard (#4040) 2024-05-02 14:57:10 +03:00
ruslandoga
8712e91bcb
drop time on page from exports (#4051) 2024-05-02 08:29:24 +01:00
RobertJoonas
bfdadc2eee
Include breakdown property in the Query struct (#4053)
* keep breakdown prop in the query struct

* Explicitly ignore property param in aggregate and timeseries

Since parameter validation depends on the breakdown property, we need to
make sure it doesn't have any unexpected effect in endpoints where it's
not expected.
2024-04-30 18:43:46 +01:00
Adrian Gruntkowski
41fef85d29
Implement resumable GA4 imports to work around rate limiting (#4049)
* Add support for resuming import to GA4 importer

* Handle rate limiting gracefully for all remainig GA4 HTTP requests

* Show notice tooltip for long running imports

* Bump resume job schedule delay to 65 minutes

* Fix tooltip styling
2024-04-30 18:06:18 +02:00
hq1
dd493fdad2
Debug queries for super-admins (#4010)
* Debug queries for super-admins

* Fixup

* Update lib/plausible/clickhouse_repo.ex

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>

* Try again with https://github.com/plausible/analytics/pull/3699

It's still clunky 😅

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
Co-authored-by: ruslandoga <doga.ruslan@gmail.com>

* Move headers injection to a separate plug module

* Add tests

* Update repo test

* Format

* Add moduledoc

* Don't assume order in query_log

* Be patient about query_log maybe?

* huh?

* huh2

* Wait longer

* Guard against \x00 in response header - testing on stage

* Fixup

* fixup

* fixup

* s/debug_label/label

* Include `site_id` and `metadata` in `log_comment`

* Tolerate non-serializable log_comment contents

---------

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
2024-04-30 09:57:28 +02:00
hq1
ad9141a9d0
Display tooltips on plan change when limits exceeded (#4048)
* Reapply "Display upgrade tooltips for exceeded limits (#4032)"

This reverts commit 76e910d45c.

* Switch to alpinejs controlled tooltips

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>

* Remove unused selector

* Refactor plan limits warning and extract tooltip component

* Remove redundant check

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-04-29 11:16:15 +02:00
hq1
b2009aa158
Rely on con_cache telemetry (#4019)
* Rely on con_cache telemetry

Now that https://github.com/sasa1977/con_cache/pull/76
is released, we don't have to use low-level operations
to emit hit/miss events.

This PR also wraps cache processes with
a function returning appropriate child specs lists.

Ideally each cache will have its own supervisor/child specs
going forward. This is an intermediate step in that direction.

* Update lib/plausible/application.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Declare caches without warmers with plain child specs

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-04-29 11:00:53 +02:00
RobertJoonas
726fe2d982
fix test (#4038) 2024-04-29 09:30:30 +01:00
hq1
d6824de1ad
Rename internal build symbols (#3942)
* Rename internal build symbols

* Rename remaining + add `on_ce` macro

cc @ruslandoga
2024-04-29 08:05:33 +02:00
Adrian Gruntkowski
ca25b6c764
Bump GA4 import page limit to max 250k (#4045)
* Bump GA4 page limit to 250k

* Handle empty properties list response gracefully
2024-04-26 20:41:42 +02:00
dependabot[bot]
805b5acbc7
Bump peaceiris/actions-gh-pages from 3 to 4 (#4007)
Bumps [peaceiris/actions-gh-pages](https://github.com/peaceiris/actions-gh-pages) from 3 to 4.
- [Release notes](https://github.com/peaceiris/actions-gh-pages/releases)
- [Changelog](https://github.com/peaceiris/actions-gh-pages/blob/main/CHANGELOG.md)
- [Commits](https://github.com/peaceiris/actions-gh-pages/compare/v3...v4)

---
updated-dependencies:
- dependency-name: peaceiris/actions-gh-pages
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-04-24 00:23:48 +02:00
Adam Rutkowski
76e910d45c Revert "Display upgrade tooltips for exceeded limits (#4032)"
This reverts commit 4a372a1528.
2024-04-23 12:53:26 +02:00
Adrian Gruntkowski
2a35a0099f
Remove "Stats start date" and adjust "Native stats start time" in Sites CRM (#4034) 2024-04-23 12:52:55 +02:00
hq1
4a372a1528
Display upgrade tooltips for exceeded limits (#4032) 2024-04-23 10:42:48 +02:00
Adrian Gruntkowski
a8ea4ce54b
Add new actions to CRM (#4030)
* Implement `Auth.TOTP.force_disable/1`

* Add "Reset 2FA" action to users CRM

* Add `Purge.reset!/2` variant allowing to set arbitrary cutoff time

* Add ability to set native stats start time from CRM

* Revert "Add `Purge.reset!/2` variant allowing to set arbitrary cutoff time"

This reverts commit 6f294d5d58.

* Add test for CRM site update action
2024-04-23 10:29:49 +02:00
hanghuge
87ae9d807f
Signed-off-by: hanghuge <cmoman@outlook.com> (#3981)
Signed-off-by: hanghuge <cmoman@outlook.com>
2024-04-23 11:18:49 +03:00
hq1
148413afbb
Fix dogfooding invitation event (#4023)
* Fix dogfooding invitation event

* Fix small build
2024-04-22 16:43:04 +02:00
Adrian Gruntkowski
7ae855cda5
Improve UI for failed import entry case (#4029)
* Improve UI for failed import entry case

* Make failed request debugging for UA imports on par with the one for GA4

* Refine failed import state further
2024-04-22 15:29:13 +02:00
Adrian Gruntkowski
3023cb12fd
Introduce active_visitors to imported_pages and start populating it with activeUsers from GA4 imports (#4027)
* Import `activeUsers` into `imported_pages.active_visitors` for GA4

* Add test for active visitors

* Simplify assertion in active visitors test

* Improve assertion for active visitors further
2024-04-22 10:18:16 +02:00
Adrian Gruntkowski
069170eb1d
Add active_visitors column to imported_pages CH table (#4028) 2024-04-22 09:34:01 +02:00
Adrian Gruntkowski
fede2f0a8a
Make final touches to Imports & Exports (#4025)
* Make final touches to Imports & Exports

* Change import content copy depending on CSV imports and exports flag state

* Remove unused aliases
2024-04-19 11:40:13 +02:00