Commit Graph

921 Commits

Author SHA1 Message Date
Adrian Gruntkowski
d5a1d6765e
Implement custom events suggestions (#4092)
* Start suggesting event names in goal settings form

* Fix tests

* Bump phoenix_live_view to 0.20.12

* Implement a criminal hack to track removal of modal's child live components

* Revert "Implement a criminal hack to track removal of modal's child live components"

This reverts commit f34ceb78f1.

* Remove redundant closing brace from currency combo input

* Hide batch goal add button when tab selection is in progress

* Implement unique modal ID regenerated on every modal close/open cycle

* Use unique modal ID as ID suffix to live components in goal settings form

* Reset suffix on tab selection to reset live components state on switch

* Revert "Bump phoenix_live_view to 0.20.12"

This reverts commit 1b1c801981.

* Make unique IDs more predictable

* Fix tests for `GoalSettings.Form`

* Use unique modal ID in country rule modal

* Use unique modal ID in hostname rule modal

* Use unique modal ID in page rule form modal

* Don't limit detected goals when fetching them for autoconfigure

* Escape interpolated Alpine state function argument

* Exclude goals with whitespace on either end or consisting only of whitespace

* Ensure event name suggestions update after goal deletion

* Avoid showing loading spinner when closing modal

* Don't enable spinner when new combobox selection is identical

* Revert "Don't enable spinner when new combobox selection is identical"

This reverts commit a041ba8542.
2024-06-11 08:28:25 +02:00
RobertJoonas
7b11b71dcf
Minor refactor: Remove skip_refresh_imported_opts from query modifying functions (#4204)
* Remove skip_refresh_imported_opts from query modifying functions

+ add test coverage for timeseries conversion rate.

* format

* improve comment
2024-06-10 13:03:12 +03:00
hq1
92d5092518
Verification improvements next batch (#4203)
* Upgrade req

* Fetch body even on non-200 statuses

Turns out sites, will happily go with 307 and no location
header for example. In which case saying we can't reach it
is an overstatement. After all we should let the headless
check determine the rest.

* Update callback status interpretation cases

* Update tests

* Fix flaky test

* Update expectation
2024-06-10 10:40:19 +02:00
ruslandoga
22ea56a779
increase api_keys.hourly_request_limit in CE (#4200)
* increase api_keys.hourly_request_limit in CE

* changelog

* add tests
2024-06-10 07:21:58 +02:00
hq1
823f7a8800
Plugins API: Get/Create Funnels (#4192)
* Bootstrap OpenAPI Funnel schemas

* Implement Plugins API Funnel view

* Allow casting funnel step directly from `%Goal{}`

* Check feature availability on funnel creation

just like it's done when inserting goals

* Implement Plugins API context module for Funnels

* Implement GET/PUT funnels via Plugins API

* Fix typo

* A rare event in which dialyzer found an actual bug, wow!

* Format

* Wrap creation request with a root `funnel` key

* Format

* Extract common funnel get query

* Remove redundant tag

* Refactor queries a bit
2024-06-07 13:10:01 +02:00
RobertJoonas
2a529b4eaa
Fix with imported switch + warning bubble renders (#4187)
* prioritize other skip_imported_reasons over not requested

* return with_imported_switch info from top stats

* fix the with-imported-switch component

* Refactor ImportedQueryUnsupportedWarning

* do not render warning bubble when imported data not requested

* stop displaying warning bubble before the API response is received

* fix tab switch pills jumping due to bubble height

* drop loading condition of ImportedQueryUnsupportedWarning for Funnels

* add explicit check for realtime when rendering the bubble

* remove unused JS imports

* rename snake_case to camelCase
2024-06-06 11:50:54 +03:00
Adrian Gruntkowski
fd81ef2105
Include imported page paths in pageview goal suggestions (#4193) 2024-06-06 10:40:48 +02:00
Adrian Gruntkowski
a764938928
Fix handling filtering by imported special properties (#4179)
* Handle custom prop queries separately when deciding imported table

* Add Query level tests
2024-06-04 13:06:31 +02:00
hq1
5fd4547382
More verification tweaks (#4177)
* Stop reporting multiple snippets in case manual extension is in use

* Try detecting CookieBot/GTM case
2024-06-03 13:49:04 +02:00
RobertJoonas
1d3b068233
Implement filtering for imported data (#4118)
* move imported.ex to imported subfolder

* move constructing base imported query into a separate module

* Implement imported table deciding and filtering

+ tests for pages, entry_pages, exit_pages and common filter types

* add top stats test with country filter

* add timeseries test

* Drop bounce_rate and time_on_page from imported & page-filtered Top Stats

* rename field returned by top stats

* turn pages into a fn comp

* Move dashboard API results under a results key

...and also return the skip_imported_reason to the frontend to be used
for displaying warnings.

* extend ListReport component with an optional afterFetchData prop

* turn Devices into a fn comp

* add not_requested as a skip_imported_reason

* display warning icons in the dashboard

* Implement filtering suggestions and translate filter fields for imported

* WIP

* Improve and cover filtering suggestions with tests

* Rename imported suggestions query helpers

* fix screen size breakdown with screen size filter

* support filtering by the same suggestion property

* support location filters when fetching location suggestions

* support filtering by multiple props from the same table

* Implement filtering by goals

* Make views per visit metric work for import entry and exit pages

* Get rid of circular dependencies between Stats.Imported and Stats.Imported.Base

* Clean up Query struct manipulation in Breakdown

* Rename helper function for clarity

* Automatically refresh query struct state after modifications

* Shutup credo

* display imported warning bubble in prop breakdown section

* Render warning bubble for funnels whenever imported data is in the view

* Transform any operator on respective goal filters

* Fix percentage and conversion_rate calculation in presence of custom props

* add tests for for combining page and pageview goal filters

* add skip_refresh option to query tweaking functions

* add imported CR support for timeseries

* still show url breakdown when special goal + url in filter

* rename Query.refresh

* use flat_map instead of map and concat

* fix darkmode color

* Handle invalid imported region codes in suggestions gracefully

* Add an entry to CHANGELOG.md

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-06-03 12:29:08 +02:00
ruslandoga
8588776df2
Add newline to csv importer insert sql (#4172)
* add newline to csv importer insert sql

* add test

* changelog

* fix test typo

* fix test typo x2
2024-06-03 09:48:16 +02:00
hq1
bb7b0d9f94
Tidy up verification code a little (#4156) 2024-05-29 08:51:10 +02:00
hq1
9d77e970ef
Redo site verification UI (#4153)
* Redo site verification UI

* format
2024-05-28 13:43:14 +02:00
hq1
d9cf037866
Handle no snippet WordPress case (#4152) 2024-05-28 09:37:01 +02:00
hq1
78c5762761
Verification: prioritize domain mismatch over multiple snippets (#4146) 2024-05-27 10:52:22 +02:00
hq1
0381bc6121
Use codespell (#4145)
* Fix typos

* Add .codespellignore

* Add codespell GH action
2024-05-27 10:52:01 +02:00
hq1
206a6543f6
First pass: extract verification errors (#4138)
* First pass: extract verification errors

We like to make them easier to maintain.
An update fixing documentation URLs will follow.

* Update moduledoc
2024-05-24 16:23:25 +02:00
hq1
a7b69719c1
More fixes to the verification process (#4136)
* More fixes to the verification process:
 - extend the list of known attributes
 - follow redirect chain host to verify data-domain difference
 - display the actual data-domain in error message, not URL

* format

* fix test

* remove redundant pattern matching

* Update lib/plausible/verification/checks/snippet.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-05-24 12:45:31 +02:00
hq1
ee61094023
Verification improvements (#4132)
* Simplify browserless script and catch errors

* Improve fetching body:
    - follow up on max. 4 redirects
    - rely on Req default timeouts (wait much longer to account for slow
      sites)

* Improve installation check:
  - rely on Req default timeouts, wait longer
  - log service errors as warnings
  - use stealth mode to bypass captchas

* Stop cutting off body too large

* Improve diagnostics for known failures

* Another round of diagnostic improvements

* Format

* Add a test for callback status 500
2024-05-24 10:40:59 +02:00
hq1
c81cb16933
Snippet integration verification (#4106)
* Allow running browserless.io locally

* Compile tailwind classes based on extra/ too

* Add browserless runtime configuration

* Ignore verification events on ingestion

* Improve extracting HTML text in tests

* Update dependencies

- Floki will be used on production to parse site contents
- Req will be used to handle redundant stuff like retrying etc.

* Add shuttle SVG to generic components

Later on we'll use it to indicate verification errors

* Connect live socket & allow skipping awaiting the first pageview

* Connect live socket in general settings

* Implement verification checks & diagnostics

* Stub remote services with Req for testing

* Change snippet screen copy

* Update tracker script, so that:

1. headless browsers aren't ignored if `window.__plausible` is defined
2. callback optionally supplies the event response HTTP status

This will be later used to check whether the server acknowledged
the verification event.

* Implement LiveView verification UI

* Embed the verification UIs into settings and onboarding

* Implement browserless puppeteer verification script

It:
 - tries to visit the site
 - defines window.__plausible, so the tracker doesn't ignore test events
 - sends a verification event and instruments the callback
 - awaits the callback to fire and returns the result

* Improve diagnostics for CSP

Only report CSP error if the snippet is already found

* Put verification behind a feature flag/env setting

* Contact Us hint only for Enterprise Edition

* For headless code, use JS context instead of EEx interpolation

* Update diagnostics test with WordPress scenarios

* Shorten exception/throw interception

* Rename test

* Tidy up

* Bust URL always on headless check

* Update moduledoc

* Detect official Plausible WordPress Plugin

and act accordingly on diagnostics interoperation

* Stop using 'rating' in favour of 'interpretation'

* Only report CSP error if no proxy is likely

* Update CHANGELOG

* Allow event-* attributes on snippet elements

* Improve naive GTM detection, not to confuse it with GA4

* Update lib/plausible/verification.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update test/plausible/site/verification/checks_test.exs

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* s/perform_wrapped/perform_safe

* Update lib/plausible/verification/checks/installation.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Remove garbage

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-05-23 15:00:50 +02:00
ruslandoga
9687fa786d
remove Plausible Analytics metions from CE (#4121)
* remove Plausible Analytics metions from CE

* update tests

* still mention Plausible Analytics on landing page
2024-05-23 09:43:01 +02:00
hq1
cd4d1d06bb
Rename Plugins API Token(s) to Plugin Token(s) (#4075)
Deployment needs to be coordinated with WordPress plugin
update. Reasoning: there's too much confusion coming
from the use of the "API" terminology, leading to wrong
keys being generated by users.
2024-05-22 07:45:09 +02:00
Adrian Gruntkowski
2d23db2982
Return warning when imported stats are skipped in Stats API due to unsupported query (#4116) 2024-05-21 11:45:18 +02:00
Adrian Gruntkowski
7876802d05
Improve CSV importer tests and visit duration query for imported data (#4114)
* Tidy up breakdown helper functions in CSV importer tests

* Fix a typo

* Extend city breakdown tests in CSV importer test suite

* Make visit duration computation consistent across native and imported data
2024-05-21 09:24:10 +02:00
Adrian Gruntkowski
756d9c28ce
Improve test coverage and fix stats API and dashboard CSV export (#4110)
* Fix broken hostname property and handle missing imported metrics gracefully

* Add test for CSV export of imported data

* Add extra coverage for property and metric combox which were failing

* Compute visit duration and bounce rate for exit pages in imported data

* Drop support for breaking down by `event:hostname` property for now
2024-05-20 16:11:35 +01:00
RobertJoonas
df178ea2d5
Introduce :skip_imported_reason field into Query struct (#4115)
* introduce new query field

* improve tests
2024-05-20 15:46:25 +01:00
Karl-Aksel Puulmann
cfdd769984
Fix StatsAPI access for new accounts on business tier (#4105)
Currently, business tier users created after business tier launch can't
access Stats API due to faulty grandfathering logic. This change should
fix that.
2024-05-15 11:50:41 +03:00
Adrian Gruntkowski
9374a95cf2
Export and import custom events via CSV (#4096)
* Export and import custom events via CSV

* Add prop support of url for cloaked links and path for 404s in imported queries

* Handle custom events with empty URL and path properties gracefully

* Make events with properties logic DRY and fix missed cloaked link

* Add test for path property breakdown

* Update raw CH data fixture and extend CSV importer tests

* Fix broken query condition after rebase

* Update CHANGELOG.md
2024-05-14 13:50:30 +02:00
Karl-Aksel Puulmann
baa99652f6
Refactor internal Query schema and introduce WhereBuilder (#4082)
* New struct format for query after parsing

* WIP refactoring

* WIP: Validations working

* WIP: tuple to list

* continued refactoring

* WIP: parsing defaults

* Breakdown tests pass

* Window functions fix

* Fix default

* Remove dead argument

* Update filters tests

* Update query_test.exs

* Fix table_decider

* sources tests pass

* Filter suggestions fix

* revenue/goal filter applied refactor

* Update top_stats matching

* Get stats_controller tests passing

* Update neighbor_aggregate_time_on_page

* Refactor Query.remove_event_filters into Query.remove_filters, add new callsites

* Move goal where clause building to new WhereBuilder module

* Move event:name filters

* Move more filters to WhereBuilder

* Update fragment to allow non-static meta columns

* Build where clause for events table using WhereBuilder

* Build sessions table where clause using WhereBuilder

* Move time range filtering and site checking to WhereBuilder

* WhereBuilder.build_condition method

* Remove TODO

* _rest pattern for TableDecider, Query pattern matching

Future-proofing in a tiny way

* Hacky fix to get tests passing for Google API tests

* Typespec fix

* Merge conflict

* refactor special goal filter logic in imported.ex

* Docs feedback

* put_filter

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
2024-05-14 11:58:10 +03:00
Uku Taht
06e8118dab
Filtering Search Console keywords (#4077)
* Apply filters in search console request

* Remove dead code from search console modal

* Remove unimportant information from keyword modal

* Show invalid filters from search console

* Fix tests

* Add/Fix tests

* Fix typo

* Remove unused variable

* Fix typo

* Changelog entry

* Fix Credo

* Display impressions, CTR and position in keyword modal

* Undo change that should not have been committed

* Fix test

* Fix test

* filters -> search_console_filters
2024-05-14 09:56:55 +03:00
ruslandoga
1f4346f4df
Add DATA_DIR (#4100)
* add DATA_DIR

* add test

* changelog

* fix test in CI where PERSISTENT_CACHE_DIR is always set

* consistent fallback
2024-05-13 10:17:56 +02:00
ruslandoga
7af8273702
Handle s3 timeout on settings page (#4036)
* handle s3 timeout

* no-async approach

* fewer changes

* add tests

* make s3 failure test EE-only
2024-05-13 10:17:27 +02:00
RobertJoonas
a6dcd19ccc
Autoconfigure event goals (#4093)
* add new goal suggestions API

* silence credo

* Order suggestions from subqueries explicitly

* allow autoconfiguring goals

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Fix form modal tab switching behavior

* add test

* Remove redundant and invalid action link title

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-05-10 11:48:27 +02:00
Adrian Gruntkowski
6bbc8d69a4
Avoid crashing on empty GA4 HTTP report response (#4094) 2024-05-10 09:12:22 +02:00
Adrian Gruntkowski
4e7e932a75
Add support for imported custom events (#4033)
* Add Ecto schema for imported custom events

* Start importing custom events from GA4

* query imported goals

* make it possible to query events metric from imported

* make it possible to query pageviews in goal breakdown

* make it possible to query conversion rate

* fix rate limiting test

* add CR tests for dashboard API

* implement imported link_url breakdown

* override special custom event names coming from GA4

* allow specific goal filters in imported_q

* update GA4 import tests to use Stats API

* Improve tests slightly

* Update CHANGELOG.md

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
2024-05-09 13:13:19 +01:00
Adrian Gruntkowski
d8435f2e01
Remove imports_exports and csv_imports_exports feature flags (#4089) 2024-05-09 10:09:24 +02:00
ruslandoga
84ed7988a8
Drop local_start_date/1 (#4088) 2024-05-09 08:59:23 +02:00
Karl-Aksel Puulmann
0a883f10e7
Refactor: Use common current_visitors code (#4071)
* Use common module for counting current visitors in external stats controller

* Refactor spike notifier, remove now-dead code
2024-05-07 15:03:37 +03:00
Karl-Aksel Puulmann
64850cd00f
Remove maybe_drop_prop_filter (#4066)
* Fix event props paygate

Previous code wasn't properly omitting event property filters from
queries.

Discovered while refactoring the code. Extracting fix from refactor for easier reviewability.

* a

* Drop function
2024-05-07 15:03:16 +03:00
ruslandoga
02d4709be7
add csv fixture for e2e export/import test (#4037)
* add inline csv fixture

* use new csvs

* cleanup csv reading and site_id replacing

* perform comparisons between native and imported queries

* help help help

* help help

* help

* eh

* fin

* exclude export/import e2e test when experimental_reduced_joins flag is enabled

* adapt to new pageviews

* adapt to experimental_reduced_joins

* credo is formatter

* cleanup

* assert bounce rates equal in city breakdown

* fix rebase against master

* clean-up dataset

* update comment

* fix typo

* apply csv changes to the files

* use sessions timestamp for exports' dates

---------

Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
2024-05-07 08:48:22 +01:00
Adrian Gruntkowski
d7ca8d9600
Revert "Debug queries for super-admins (#4010)" (#4073)
This reverts commit dd493fdad2.
2024-05-06 15:05:17 +02:00
Karl-Aksel Puulmann
17f812443d
Return session in each time bucket its active in for hourly/minute timeseries (#4052)
* Fix typo in test name

* Update test_helper, enable experimental_session_count together with experimental_reduced_joins

* Return session in each time bucket its active in for hourly/minute timeseries

The behavior is behind experimental_session_count flag

This results in more accurate visitor showing compared to previous approach of showing each user only
active the _last_ time they did a pageview.

Were not doing this for monthly/weekly graphs due to query performance cost and it having a small effect there.
See also https://3.basecamp.com/5308029/buckets/35611491/messages/7085680123

* Add tests for new behavior

Note the new behavior mimics the old one precisely, these tests fail if only
experimental_reduced_joins is on, but not experimental_session_count

* Type erasure

* Dead comment remove

* Expected_plot change
2024-05-05 11:44:43 +03:00
ruslandoga
972dd5d150
redirect to s3 url when downloading exports (#4002)
* redirect to s3 url

* use new on_ee macro, reduce wait time for email to five seconds
2024-05-02 19:53:12 +01:00
ruslandoga
8712e91bcb
drop time on page from exports (#4051) 2024-05-02 08:29:24 +01:00
RobertJoonas
bfdadc2eee
Include breakdown property in the Query struct (#4053)
* keep breakdown prop in the query struct

* Explicitly ignore property param in aggregate and timeseries

Since parameter validation depends on the breakdown property, we need to
make sure it doesn't have any unexpected effect in endpoints where it's
not expected.
2024-04-30 18:43:46 +01:00
Adrian Gruntkowski
41fef85d29
Implement resumable GA4 imports to work around rate limiting (#4049)
* Add support for resuming import to GA4 importer

* Handle rate limiting gracefully for all remainig GA4 HTTP requests

* Show notice tooltip for long running imports

* Bump resume job schedule delay to 65 minutes

* Fix tooltip styling
2024-04-30 18:06:18 +02:00
hq1
dd493fdad2
Debug queries for super-admins (#4010)
* Debug queries for super-admins

* Fixup

* Update lib/plausible/clickhouse_repo.ex

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>

* Try again with https://github.com/plausible/analytics/pull/3699

It's still clunky 😅

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
Co-authored-by: ruslandoga <doga.ruslan@gmail.com>

* Move headers injection to a separate plug module

* Add tests

* Update repo test

* Format

* Add moduledoc

* Don't assume order in query_log

* Be patient about query_log maybe?

* huh?

* huh2

* Wait longer

* Guard against \x00 in response header - testing on stage

* Fixup

* fixup

* fixup

* s/debug_label/label

* Include `site_id` and `metadata` in `log_comment`

* Tolerate non-serializable log_comment contents

---------

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
2024-04-30 09:57:28 +02:00
hq1
ad9141a9d0
Display tooltips on plan change when limits exceeded (#4048)
* Reapply "Display upgrade tooltips for exceeded limits (#4032)"

This reverts commit 76e910d45c.

* Switch to alpinejs controlled tooltips

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>

* Remove unused selector

* Refactor plan limits warning and extract tooltip component

* Remove redundant check

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-04-29 11:16:15 +02:00
hq1
b2009aa158
Rely on con_cache telemetry (#4019)
* Rely on con_cache telemetry

Now that https://github.com/sasa1977/con_cache/pull/76
is released, we don't have to use low-level operations
to emit hit/miss events.

This PR also wraps cache processes with
a function returning appropriate child specs lists.

Ideally each cache will have its own supervisor/child specs
going forward. This is an intermediate step in that direction.

* Update lib/plausible/application.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Declare caches without warmers with plain child specs

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-04-29 11:00:53 +02:00
RobertJoonas
726fe2d982
fix test (#4038) 2024-04-29 09:30:30 +01:00