Commit Graph

2656 Commits

Author SHA1 Message Date
Karl-Aksel Puulmann
0a883f10e7
Refactor: Use common current_visitors code (#4071)
* Use common module for counting current visitors in external stats controller

* Refactor spike notifier, remove now-dead code
2024-05-07 15:03:37 +03:00
Karl-Aksel Puulmann
9f6ea00a72
Clean up deleted sites from clickhouse cron (#4065) 2024-05-07 15:03:32 +03:00
Karl-Aksel Puulmann
64850cd00f
Remove maybe_drop_prop_filter (#4066)
* Fix event props paygate

Previous code wasn't properly omitting event property filters from
queries.

Discovered while refactoring the code. Extracting fix from refactor for easier reviewability.

* a

* Drop function
2024-05-07 15:03:16 +03:00
RobertJoonas
c106595be0
Migration: add imported custom events (#4076)
* Add `imported_custom_events` to CH

* remove redundant table setting

* add path column

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-05-07 11:46:27 +01:00
ruslandoga
02d4709be7
add csv fixture for e2e export/import test (#4037)
* add inline csv fixture

* use new csvs

* cleanup csv reading and site_id replacing

* perform comparisons between native and imported queries

* help help help

* help help

* help

* eh

* fin

* exclude export/import e2e test when experimental_reduced_joins flag is enabled

* adapt to new pageviews

* adapt to experimental_reduced_joins

* credo is formatter

* cleanup

* assert bounce rates equal in city breakdown

* fix rebase against master

* clean-up dataset

* update comment

* fix typo

* apply csv changes to the files

* use sessions timestamp for exports' dates

---------

Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
2024-05-07 08:48:22 +01:00
Adrian Gruntkowski
62138e0dad
Apply sampling factor multiplication consistently across CSV export queries (#4074)
* Apply sampling factor multiplication consistently across CSV export queries

* Fix casting

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>

---------

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
2024-05-07 09:15:43 +02:00
Adrian Gruntkowski
d7ca8d9600
Revert "Debug queries for super-admins (#4010)" (#4073)
This reverts commit dd493fdad2.
2024-05-06 15:05:17 +02:00
Adrian Gruntkowski
70ed30e871
Add override on CSV exports for super admins (#4070) 2024-05-06 11:12:47 +02:00
ruslandoga
b0fc527eb2
Run Oban in CE too (#4061) 2024-05-06 10:40:09 +02:00
ruslandoga
c49500204c
use released ecto_ch v0.3.5 (#4054) 2024-05-06 10:39:45 +02:00
ruslandoga
6e3556d2aa
Use sessions timestamp for exports' dates (#4063) 2024-05-06 10:39:00 +02:00
Adrian Gruntkowski
39357150e9
Improve styling of imports list (#4069) 2024-05-06 10:38:37 +02:00
Karl-Aksel Puulmann
035cb3a379
Use ALIAS columns to simplify querying API (#4059)
* Cleanup: remove @api_prop_name_to_db-related cruft

* Use aliases elsewhere in code
2024-05-05 12:03:00 +03:00
Karl-Aksel Puulmann
17f812443d
Return session in each time bucket its active in for hourly/minute timeseries (#4052)
* Fix typo in test name

* Update test_helper, enable experimental_session_count together with experimental_reduced_joins

* Return session in each time bucket its active in for hourly/minute timeseries

The behavior is behind experimental_session_count flag

This results in more accurate visitor showing compared to previous approach of showing each user only
active the _last_ time they did a pageview.

Were not doing this for monthly/weekly graphs due to query performance cost and it having a small effect there.
See also https://3.basecamp.com/5308029/buckets/35611491/messages/7085680123

* Add tests for new behavior

Note the new behavior mimics the old one precisely, these tests fail if only
experimental_reduced_joins is on, but not experimental_session_count

* Type erasure

* Dead comment remove

* Expected_plot change
2024-05-05 11:44:43 +03:00
Karl-Aksel Puulmann
850a843d82
Add migration to add ALIAS columns to common session/visit properties (#4058)
This allows for simplifications in the API code
2024-05-05 11:30:39 +03:00
ruslandoga
972dd5d150
redirect to s3 url when downloading exports (#4002)
* redirect to s3 url

* use new on_ee macro, reduce wait time for email to five seconds
2024-05-02 19:53:12 +01:00
ruslandoga
07cf2b4682
use events_v2 for pageviews export (#4035) 2024-05-02 14:50:31 +01:00
Karl-Aksel Puulmann
bcde2b8cf2
Add Yesterday as an time range option in the dashboard (#4040) 2024-05-02 14:57:10 +03:00
ruslandoga
8712e91bcb
drop time on page from exports (#4051) 2024-05-02 08:29:24 +01:00
RobertJoonas
bfdadc2eee
Include breakdown property in the Query struct (#4053)
* keep breakdown prop in the query struct

* Explicitly ignore property param in aggregate and timeseries

Since parameter validation depends on the breakdown property, we need to
make sure it doesn't have any unexpected effect in endpoints where it's
not expected.
2024-04-30 18:43:46 +01:00
Adrian Gruntkowski
41fef85d29
Implement resumable GA4 imports to work around rate limiting (#4049)
* Add support for resuming import to GA4 importer

* Handle rate limiting gracefully for all remainig GA4 HTTP requests

* Show notice tooltip for long running imports

* Bump resume job schedule delay to 65 minutes

* Fix tooltip styling
2024-04-30 18:06:18 +02:00
hq1
dd493fdad2
Debug queries for super-admins (#4010)
* Debug queries for super-admins

* Fixup

* Update lib/plausible/clickhouse_repo.ex

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>

* Try again with https://github.com/plausible/analytics/pull/3699

It's still clunky 😅

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
Co-authored-by: ruslandoga <doga.ruslan@gmail.com>

* Move headers injection to a separate plug module

* Add tests

* Update repo test

* Format

* Add moduledoc

* Don't assume order in query_log

* Be patient about query_log maybe?

* huh?

* huh2

* Wait longer

* Guard against \x00 in response header - testing on stage

* Fixup

* fixup

* fixup

* s/debug_label/label

* Include `site_id` and `metadata` in `log_comment`

* Tolerate non-serializable log_comment contents

---------

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
2024-04-30 09:57:28 +02:00
hq1
ad9141a9d0
Display tooltips on plan change when limits exceeded (#4048)
* Reapply "Display upgrade tooltips for exceeded limits (#4032)"

This reverts commit 76e910d45c.

* Switch to alpinejs controlled tooltips

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>

* Remove unused selector

* Refactor plan limits warning and extract tooltip component

* Remove redundant check

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-04-29 11:16:15 +02:00
hq1
b2009aa158
Rely on con_cache telemetry (#4019)
* Rely on con_cache telemetry

Now that https://github.com/sasa1977/con_cache/pull/76
is released, we don't have to use low-level operations
to emit hit/miss events.

This PR also wraps cache processes with
a function returning appropriate child specs lists.

Ideally each cache will have its own supervisor/child specs
going forward. This is an intermediate step in that direction.

* Update lib/plausible/application.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Declare caches without warmers with plain child specs

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-04-29 11:00:53 +02:00
RobertJoonas
726fe2d982
fix test (#4038) 2024-04-29 09:30:30 +01:00
hq1
d6824de1ad
Rename internal build symbols (#3942)
* Rename internal build symbols

* Rename remaining + add `on_ce` macro

cc @ruslandoga
2024-04-29 08:05:33 +02:00
Adrian Gruntkowski
ca25b6c764
Bump GA4 import page limit to max 250k (#4045)
* Bump GA4 page limit to 250k

* Handle empty properties list response gracefully
2024-04-26 20:41:42 +02:00
dependabot[bot]
805b5acbc7
Bump peaceiris/actions-gh-pages from 3 to 4 (#4007)
Bumps [peaceiris/actions-gh-pages](https://github.com/peaceiris/actions-gh-pages) from 3 to 4.
- [Release notes](https://github.com/peaceiris/actions-gh-pages/releases)
- [Changelog](https://github.com/peaceiris/actions-gh-pages/blob/main/CHANGELOG.md)
- [Commits](https://github.com/peaceiris/actions-gh-pages/compare/v3...v4)

---
updated-dependencies:
- dependency-name: peaceiris/actions-gh-pages
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-04-24 00:23:48 +02:00
Adam Rutkowski
76e910d45c Revert "Display upgrade tooltips for exceeded limits (#4032)"
This reverts commit 4a372a1528.
2024-04-23 12:53:26 +02:00
Adrian Gruntkowski
2a35a0099f
Remove "Stats start date" and adjust "Native stats start time" in Sites CRM (#4034) 2024-04-23 12:52:55 +02:00
hq1
4a372a1528
Display upgrade tooltips for exceeded limits (#4032) 2024-04-23 10:42:48 +02:00
Adrian Gruntkowski
a8ea4ce54b
Add new actions to CRM (#4030)
* Implement `Auth.TOTP.force_disable/1`

* Add "Reset 2FA" action to users CRM

* Add `Purge.reset!/2` variant allowing to set arbitrary cutoff time

* Add ability to set native stats start time from CRM

* Revert "Add `Purge.reset!/2` variant allowing to set arbitrary cutoff time"

This reverts commit 6f294d5d58.

* Add test for CRM site update action
2024-04-23 10:29:49 +02:00
hanghuge
87ae9d807f
Signed-off-by: hanghuge <cmoman@outlook.com> (#3981)
Signed-off-by: hanghuge <cmoman@outlook.com>
2024-04-23 11:18:49 +03:00
hq1
148413afbb
Fix dogfooding invitation event (#4023)
* Fix dogfooding invitation event

* Fix small build
2024-04-22 16:43:04 +02:00
Adrian Gruntkowski
7ae855cda5
Improve UI for failed import entry case (#4029)
* Improve UI for failed import entry case

* Make failed request debugging for UA imports on par with the one for GA4

* Refine failed import state further
2024-04-22 15:29:13 +02:00
Adrian Gruntkowski
3023cb12fd
Introduce active_visitors to imported_pages and start populating it with activeUsers from GA4 imports (#4027)
* Import `activeUsers` into `imported_pages.active_visitors` for GA4

* Add test for active visitors

* Simplify assertion in active visitors test

* Improve assertion for active visitors further
2024-04-22 10:18:16 +02:00
Adrian Gruntkowski
069170eb1d
Add active_visitors column to imported_pages CH table (#4028) 2024-04-22 09:34:01 +02:00
Adrian Gruntkowski
fede2f0a8a
Make final touches to Imports & Exports (#4025)
* Make final touches to Imports & Exports

* Change import content copy depending on CSV imports and exports flag state

* Remove unused aliases
2024-04-19 11:40:13 +02:00
Adrian Gruntkowski
c10580777e
Remove references to site.imported_data (#4006)
* Remove references to `site.imported_data`

* Count pre-existing ID 0 imports when showing pageview count summary for legacy imports

* Fix tests after rebase

* Dry `delete_imported_stats!`

* Clean up remaining imported data references and add notes
2024-04-19 11:15:51 +02:00
hq1
83643450c4
Remove obsolete feature flags (#4026) 2024-04-18 20:18:38 +02:00
Marko Saric
1b06bb5fd2
Update site_setup_success_email.html.eex (#4021) 2024-04-18 16:06:38 +02:00
RobertJoonas
f49bdef7dd
Fix main graph intervals (#4024)
* make sure valid intervals consider custom range

* make default intervals per period more logical

* remove unused function

* adjust defaults for custom range
2024-04-18 15:01:47 +01:00
Marko Saric
219f1f5538
Add link to settings on the "Waiting for first pageview" screen (#4020)
* Update waiting_first_pageview.html.heex

* Improve text formatting

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-04-18 12:13:19 +02:00
Adrian Gruntkowski
9bae3ccce3
Improve UI/UX of imports view and GA import flow (#4017)
* Add runtime config option for enabled/disabling csv imports and exports

* Use the new option to toggle rendering exports UI

* Disable import buttons when at maximum imports or when option disabled for CSV

* Improve forms for GA import flow

* Add test for maximum imports reached

* Remove "Changed your mind?" prefixing back button

* Hide UA imports in Integrations when `imports_exports` flag is enabled

* Implement `csv_imports_exports` feature flag

* Revert "Add runtime config option for enabled/disabling csv imports and exports"

This reverts commit e30f202dd3.

* Send import notification email only to the user who ran the import

* Improve rendering of disabled button state

* Put import status heroicon in front of import label
2024-04-18 12:12:48 +02:00
RobertJoonas
3a371fdf4d
Test new imported metrics (GA4) (#4014)
* test fixture imported data with stats requests

* take visits metric from the events table in event:page breakdown

* Remove assert_referrers after all

pageReferrer is an event scoped property in GA4, which when queried
along with session-level dimensions will return unexpected data.

Adding the pageReferrer dimension to the GA4 Data API request, it will
cause the selected metric totals to increase significantly, even though
they shouldn't.

* Adjust sources and utm_mediums assertions

* adjust assert_pages

* Make formatter happy

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-04-18 12:12:24 +02:00
Adrian Gruntkowski
9849743407
Always sort occupied date ranges in Imported.clamp_dates/3 (#4018) 2024-04-18 11:15:51 +02:00
Adam Rutkowski
b373c36dcc Fix docs link for hostname shield 2024-04-17 07:01:16 +02:00
Adam Rutkowski
2b1fbf0a0e Fix typo (and kick docker build) 2024-04-16 20:57:29 +02:00
hq1
6fb56dc1cc
Stats api hostname filter (#4008)
* Update Stats API tests

* Revert "Remove hostname filter from the external API (#3991)"

This reverts commit 884daa7943.
2024-04-16 20:36:57 +02:00
hq1
f635f0a6d3
Hostnames shield (#3990)
* Add shield hostname rules migration

* Add hostname rule schema

* Initialize hostname rules cache

* Extend Shields context with hostname related functions

* Instrument ingestion pipeline with hostname rule lookups

* Limit hostname suggestions by shield patterns

* Add LiveView for hostname rules management

* Test hostname cache

* Rename feature flag - should be separate from hostname filter

* Remove :shield_pages feature flag

* Update CHANGELOG

* Format

* Update lib/plausible/shield/hostname_rule.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Move tests from `lib/` 🤦

* Use plain `assign` where no short-circuit is necessary

* Fine tune the copy a little bit

* Prevent misplaced tests

* Treat a test with common sense

* Fixup another test that hasn't been really run before

* Make the form hint dynamic depending on rules count

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-04-16 20:30:20 +02:00