* Always scope import ID by site as well
* Do not schedule new import job if there are any site imports in progress
* Disable import buttons when any import is in progress
* Simplify `schedule_job/4` (h/t @RobertJoonas)
* Give a more semantic name to a function
* Make the LineGraph component thinner
* Move LineGraph into a separate file
* Move interval logic into interval-picker.js
This commit also fixes a bug where the interval name displayed inside
the picker component flickers the default interval when the graph is
loading.
The problem was that we were counting on graphData for returning us the
current interval: `let currentInterval = graphData?.interval`
We should always know the default interval before making the main-graph
request. Sending graphData to IntervalPicker component does not make
sense anyway.
* extract data fetching functions out of VisitorGraph component
* Return graph_metric key from Top Stats API
This commit introduces no behavioral changes - only starts returning an
additional field, allowing us to avoid the following logic in React:
1. Finding the metric names, given a stat display name. E.g.
`Unique visitors (last 30 min) -> visitors`
2. Checking if a metric is graphable or not
* Move metric state into localStorage
This commit gets rid of the internal `metric` state in the VisitorGraph
component and starts using localStorage for that instead.
This commit also chains the main-graph request into the top-stats request
callback - meaning that we'll always fetch new graph data after top stats
are updated. And we do it all in a single function.
Doing so simplifies the loading state significantly, and also helps to
make it clear, that at all times, existing top stats are required before
we can fetch the graph. That's because the metric is determined by which
Top stats are returned (for example, we can't be sure whether revenue
metrics will be returned or not).
* Make sure graph tooltip says "Converted Visitors"
* Extract a StatsExport function component
Again, instead of relying on `graphData?.interval` we can read it from
localStorage, or default to the largest interval available. The export
should not be dependant on the graph.
* Extract SamplingNotice function component
* Extract WithImportedSwitch function component
* Stop "lazy-loading" the graph and top stats
Since the container is always on top on the page, it will be visible on
the first render in any case - no matter the screen size.
* Turn VisitorGraph into a function component
* Display empty container until everything has loaded
* Do not display loading spinner on realtime ticks
* Turn Top Stats into a fn component
* fetch top stats and graph async
* Make sure revenue metrics can remain on the graph
* Add an extra check to canMetricBeGraphed
* fix typo
* remove redundant double negation
* Move experimental_session_count? logic to within query object
* WIP new querying system for deciding what tables to query
* both -> either
* Include sample_percent in both tables
* Remove a hanging TODO
* Allow filtering by visit props on event queries if flag is on
* Make default sessions join more conditional
* Simplify events_join_sessions?
* Add some TODOs
* Fix assignment
* Handle entry/exit page visit props separately from props stored in events table
* Update test which created sessions/events differently from everyone else
* Make query_events private
* Dont filter by session properties on events table if querying sessions and joining in events
* Handle visits, pageviews, events and visitors metrics from other table
* both -> either
* events, pageviews are strictly event metrics
* Add support for (plain) breakdowns deciding which table to use
* Run tests with experimental_reduced_joins as a separate job
Also refactor which tests are run with postgres:15 to reduce number of jobs
* moduledocs for TableDecider
* Fix matrix
* Custom build name
* Move TEST_EXPERIMENTAL_REDUCED_JOINS check
* Handle percentage separately from other metrics
* Remove debug code
* TableDecider tests
* both => sample_percent
* Improve naming
* Simplify code
* Breakdowns retain old behavior if getting metric visitors
* Unify behavior of entry/exit page hostnames with rest
* Fix test naming
* CH Migration: exit/entry hostnames in sessions_v2
* Leave only exit_page_hostname, we already record hostnames
* Use ClickHouse DDL in favour of ecto so that cluster is included
* Compress with ZSTD(3)
* Expose Hostname filter in the dashboard dropdown
* Add `exit_page_hostname` to ClickHouse `sessions_v2` schema
* Start tracking hostname changes in sessions
* Implement hostname filter suggestions
* Enable filtering by `event:hostname`
* Add tests for filtering by hostnames
* Ensure filter suggestions work for exit pages too
* Allow overriding hostnames with `send_pageview` mix task
* Remove `:window_time_on_page` flag
It seems that we can remove it after all?
* Initialize `experimental_hostname_filter` query parameter
* Rewrite cache store behaviour with regards to session hostnames
* Work around inconsistent session merging
So that `populate_stats` can get closer to actual ingestion
* Improve top stats test
* Make it possible to filter sessions by entry/exit hostnames
* Update pages tests
* Expose `experimental_hostname_filtering` temporarily in the UI
* Untested yet: also apply experimental filtering to sources
* Introduce `hostname_filter` feature flag
* Format
* Test top sources with hostname filter + experimental flag
* Check import presence across all imports and not just the first one
Also, simplify imported data toggle rendering to not explicitly
refer to the earliest import source.
* Change imported stats toggle icon in dashboard
* Test `Imported.get_imports_date_range/1`
* Simplify failed UA/GA import email copy
* Implement adjusting imported date range to actual and existing stats
* Drop redundant prefix from import list entries
* Make pageview numbers in imports list formatted for readability
* Test and improve date range cropping
* DRY UA and GA4 stats start and end date API calls
* Extend UA/GA import controller tests and improve error handling
* refactor finding longest open range without existing data
* Fix typo in test description
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
* Rename `open_ranges` to `free_ranges`
---------
Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
* Use sessionStorage for offer e-mail report banner tracking
Keeping it within the cookie is problematic, as the banners don't
expire and overflow the cookie with data when enough new sites
are added.
Ref https://github.com/plausible/analytics/issues/3762
* Update changelog
* Extract a component
* Make is_dbip evaluate to quoted boolean
* Ignore sessions without entry/exit pages when breaking down entry/exit pages
* Update stats controller tests to have more realistic test data (pageview followed by event)
* Add validation for the events metric in main_graph
* Test the already existing events metric support in main-graph
* Put total conversions on the graph
* extract main_graph_csv function (refactor only)
* add total_conversions and conversion_rate to goal-filtered visitors.csv
* update changelog
* add conversion rate to Stats API timeseries
* make sure CR can be queried as the only metric
* add a test asserting zeros are returned
* add tests for filtering by other properties at the same time
* Remove unnecessary validation of params
1. It doesn't make to validate `interval` (and its granularity) in all
endpoints. It's only relevant for the main graph.
2. The plug (renamed to `date_validation_plug`) already makes sure that
the dates are validated. No need to call the same function again in
Top Stats and Funnel endpoints.
* add metric validation to main graph
* Add tests for main graph API
* put conversion rate on the graph
* update changelog
* Add revenue metrics into metrics.ex
* make fn private
* avoid setting graph metric to visitors in goal-filtered view
* Unify GA4 and UA import flow into one
* Clean up property and view data retrieval via Google HTTP APIs
* Turn `Map.get` into `Map.fetch!` in API response processing code
* Bump list account summaries page size limit to max of 200
* Show only views in legacy flow and fix legacy redirect after import start
* Move google analytics import actions tests to a separate module
* Extend Google Analytics controller tests
* DRY up `property?` predicate (h/t @RobertJoonas)
* Revert "Revert "Move conversion_rate logic from elixir to clickhouse (#3887)"…"
This reverts commit 253fb5d67d.
* Fix issue with missing columns
The issue came from refactoring event:goal UNION ALL logic and trying to move
name select from first to last. If any other tables were joined, the incorrect
item would be used as an array index, causing this issue.
Added a relevant test.
* Separate out query building from pagination/execution logic.
* Refactor pageview_goals breakdown query, removing index column from results
* Remove zip_columns logic
* Use common pagination util
* Do everything in a single query for breakdowns for goals
* Order in DB
* Make sure column order is identical
* Calculate CR within the goal breakdown query
* Calculate CR for property breakdowns
* WIP: Calculate group CR
* CR with order_by
* Compatibility fix
* Import Ecto.Query and cleanup
* handle total_visitors the same way as add_percentage
* Handle conversion_rate in aggregate.ex
* Solve rebase fail
* Simplify maybe_add_group_conversion_rate
* Add conversion_rate defaults to 0 test
* Add test for conversion_rate should not be calculated with imported data (failing here and on master)
* Dont include imported data when breakdown by prop or goal
* Remove revenue_nils
* prevent division by 0 in merge_imported queries
* Revert "fix bounce_rate change bug (#3886)"
This reverts commit 6eef32a8ff.
After 02aa0b2, we can keep on assuming that bounce rate is always numeric.
* Implement LV date input using flatpickr
* Implement basics of GA4 import (very dirty WIP)
* Split Google HTTP API into UA and GA4 specific parts
* Add a quick way to record GA4 API responses
* Add first GA4 import fixtures with GA4 Data API responses
* Extract GA4 and UA specific logic form Google API
* Extract UA and GA4 specific actions to distinct controllers
* Add integration test for GA4 importer
* Update GA4 fixtures
* Test GA4 API
* Add debug logging and fix paginating through API results in in GA4 import
* Revert "Implement LV date input using flatpickr"
This reverts commit c696f8ee39d5702f27015c09a4f079ca124cc7bb.
* Fix note
* add metric validation + support in aggregate
* add a test ensuring comparison works
* disallow time_on_page with a goal filter
* Return time_on_page as `nil` from aggregate API
In case time_on_page cannot be calculated, we'll return it as `nil` from
the Stats API.
This is to make the behaviour consistent between breakdown and aggregate
endpoints. As for the UI, we'll still continue to report time_on_page as
0 - not changing any UI behaviour as discussed with Marko.
* add tests for time_on_page in event:page breakdown
* update changelog
* invalidate time_on_page with event:name filter
* add the ability to only query time_on_page in page breakdown
We'll need the visitors metric to get the list of pages to calculate the
time_on_page for.
* Remove `add_percentage`, calculate percentages in clickhouse queries
This simplifies querying logic and avoids doing extra queries and avoids
race conditions.
* Remove special none handling from breakdowns, handling percentages correctly
* Add (failing) test showing expected add_percentage behavior for user making multiple sessions
* Update add_percentage behavior to use separate subqueries
* Refactor: Explicitly add field names to INSERT
This avoids issues when code schema is out of sync with real schema
* Dont write session parameters to events
These would only be stored on first event anyways. Work remains to be done
on tests which have their own helper
* Remove writes to country_code in a test
* Remove old columns from being accessible in elixir code
* Update most tests to use new way of adding session props to events
* Update testing harness
* Update stats controller test
* Update for shield rules
* update breakdown tests
* Fix typing of state for dialyzer
* Drop support for old session attributes code
* Update remaining tests
* cond -> if
* Create a stub of site settings section for imports and exports
* Use legacy site import indication to determine UA import handling
* Add provisional logos for upcoming import sources
* Stub basics of import page
* Add very rudimentary support for multiple UA imports
* Implement imports list as live view
* Add support for opening LV modal from backend and closing from frontend
* Introduce notion of themes to `button` and `button_link` components
* Add confirmation modal on deleting import
* Swap GA4 logo
* Implement disabled state support for `button_link` component
* Disable export and non-implemented import sources
* Use native starts start date for upper boundary of import time range
* Ensure integrations view uses legacy UA import flow
* Remove unnecessary preload in SiteController
* Remove unnecessary exception for legacy imports
* Move API controller stats tests under PlausibleWeb
* Test listing imports
* Add test for explicit listener setup
* Add tests for legacy flag state in UA importer
* Add test for purging legacy import data
* Add tests for `Sites.native_stats_start_date`
* Test forgetting imports
* Add `Stats.Clickhouse.imported_pageview_counts/1` and fix test flakiness
* Show page view counts on imports list
* Add tests for static imports and exports view
* Adjust button look slightly
* Use `case` instead of `cond`
* Make feature flag customisable per site
* Fix buttons and empty state styling
* Add another import to seeds
* Use JS confirm dialog instead of modal for deletion confirmations
* Revert "Add support for opening LV modal from backend and closing from frontend"
This reverts commit 260e6c753032b451542e24be9edc2118790b5a00.
* Default `legacy` to false when inserting new import jobs
* Drop `method` attribute from `button_link` and `unstyled_link` components
* improve test
* add os to os_version breakdown
...and add operating_system_versions.csv to the CSV export
* fix conversion rate for os_version breakdown
* update changelog
* fix existing CSV tests
* use case instead of cond
* Allow e-mail exclusion in team members quota
* Exclude invitee from quota on invitation create
* Enable invitation submission but report errors on quota violation
* Use a single interface for team members quota
* Check the `Keyword.validate/2` result
* Update test/plausible_web/controllers/site/membership_controller_test.exs
Co-authored-by: Uku Taht <Uku.taht@gmail.com>
---------
Co-authored-by: Uku Taht <Uku.taht@gmail.com>
* Revert "Unify percentage change for CR and bounce_rate (#3781)"
This reverts commit a6b1a6ebc7.
* Revert "Bring Stats API up to speed: Add `conversion_rate` to Aggregate and Breakdown (#3739)"
This reverts commit 672d682e95.
* Fix conversion rate change calculation
The change in conversion rate should be calculated similar to bounce rate.
For example, an increase of 25% -> 50% should not be a 100% change, but
a 25% change instead.
* Use the same comparison function in Stats API and dashboard API
This commit fixes a bug where the percentage change reported by the Stats
API is different from the one returned by the internal dashboard API.
* changelog update
* WIP
* Allow `experimetnal_session_count` request serialization
* Extend `Plausible.Stats.Query` with `experimental_session_count` flag
* Add `FunWithFlags` actor implementation for `Site`
* Change the way sessions are retrieved
* Remove redundant test
* Format
* Update the test
---------
Co-authored-by: Uku Taht <uku.taht@gmail.com>
* disable event metric with include_imported in every case
* add missing test for metric validation
* refactor metric validation functions
* implement conversion_rate metric validation
* move calculate_cr function into Stats.Util
* Refactor: Move aggregate CR logic into Stats.aggregate
* define atoms to exist
* Ensure that CR does not depend on visitors being queried
If 'visitors' are already queried, we'll use that value. Otherwise we'll
need to make another query to fetch it.
* confirm Stats API aggregate supports CR (tests only)
* small refactor
This is the only 'event_property' left after pattern matching on all
others in the function clauses defined above.
* Make it possible to optionally query conversion_rate
...in breakdown queries (excluding goal and custom prop breakdown)
* A little refactor asking for revenue metrics
1. The `@revenue_metrics` module attribute is an empty list on full build
anyway
2. We don't need to query for revenue metrics if there are no revenue goals
returned in the given query (even if revenue goals exist in site.goals)
3. Revenue metrics are already dropped in prop breakdown without a goal
filter via (get_revenue_tracking_currency/3)
* Make it possible to optionally query conversion_rate (continuation)
... also from a custom prop and goal breakdown
* Frontend adjustments to the Locations report
* Display conversion rate in Regions and Cities (ListReport view)
* Display total conversions, conversions (visitors), and CR in the
"Details" modals of Countries, Regions, and Cities
* Move the percentage into a separate column in the Countries details table
* confirm Stats API breakdown supports conversion_rate (tests only)
* small refactor: extract maybe_add_time_on_page function
* Make it possible to query cr alone
... (without the visitors metric). Already supported in aggregate, this
commit only implements it for the breakdown API.
* Reuse Stats.Util helper functions from b02db88 for aggregate API
We can follow the same logic as with breakdown for manually adding
`visitors` into the metrics list and taking it out of the response
later on.
That way we don't have to make another query, e.g. in a case where
only pageviews and conversion rate is queried. Also keeps things
consistent.
* changelog update
* fix test after resolving merge conflict
* Use explicit string->atom mapping instead of casting
* alias Util module instead of importing it
* use Enum.empty instead of Enum.any
* improve readability
* rename special_metrics to computed_metrics and explain with a comment
* rename visitors_without_event_filters to total_visitors
* keep a single function for removing unwanted metrics
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Clean up references to no longer active `google_analytics_imports` Oban queue
* Stub CSV importer
* Add SiteImport schema
* Rename `Plausible.Imported` module file to match module name
* Add `import_id` column to `Imported.*` CH schemas
* Implement Importer behavior and manage imports state using new entities
* Implement importer callbacks and maintain site.imported_data for UA
* Keep imports in sync when forgetting all imports
* Scope imported data queries to completed import IDs
* Mark newly imported data with respective import ID
* Clean up Importer implementation a bit
* Test querying legacy and new imported data
* Send Oban notifications on import worker failure too
* Fix checking for forgettable imports and remove redundant function
* Fix UA integration test
* Change site import source to atom enum and add source label
* Add typespecs and reduce repetition in `Plausible.Imported`
* Improve documentation and typespecs
* Add test for purging particular import
* Switch email notification templates depending on import source
* Document running import synchronously
* Fix UA importer args parsing and ensure it's covered by tests
* Clear `site.stats_start_date` on complete import to force recalculation
* Test Oban notifications (h/t @ruslandoga)
* Purge stats on import failure right away to reduce a chance of leaving debris behind
* Fix typos
Co-authored-by: hq1 <hq@mtod.org>
* Fix another typo
* Refactor fetching earliest import and earliest stats start date
* Use `Date.after?` instead of `Timex.after?`
* Cache import data in site virtual fields and limit queried imports to 5
* Ensure always current `stats_start_date` is used
* Work around broken typespec in Timex
* Make `SiteController.forget_imported` action idempotent
* Discard irrecoverably failed import tasks
* Use macros for site import statuses
There's also a fix ensuring only complete imports are considered
where relevant - couldn't isolate it as it was in a common hunk
* Use `import_id` as worker job uniqueness criterion
* Do not load imported stats data in plugins API context
---------
Co-authored-by: hq1 <hq@mtod.org>
* WIP: PropFilterRow
* Get multi-behavior working
* Render multiple prop filters in one
* Modal reads from query string correctly
* Backend support for multiple custom property filters
* Add backend tests for multiple custom property filters
* Disable already selected options in property keys
We can't allow choosing the same property multiple times without changing the request
params, which we decided against
* Allow choosing any property under Behaviors > Custom props even if custom prop filter applied
This was a limitation (I believe) introduced by using ARRAY JOINs to query custom properties
* CHANGELOG.md
* Solve credo warning about too deep nesting
* Update assets/js/dashboard/stats/modals/prop-filter-modal.js
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
* Refactor internal function for clarity
* Add another step -> Add another
* Solve 500 error
* Separate boxes per property filter
* Retain other filters in props table
* removeFilter behavior for props
* matches_member support for custom props
* filter_suggestions for prop keys should account for prop filter
* find over filter
* refactor appliedFilters
* FILTER_TYPES => FILTER_OPERATIONS
* Make add another link not wrap the whole page
* Unique keys
---------
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
* add tests for filtering by goal in timeseries and aggregate
* refactor filter parsing
* stop returning custom props in event:goal breakdown
* test breaking down wildcard pageview goals
* extract filter utils
* parse more goal filter options
* add passing tests for new filter types
* do not allow querying session metrics with a goal filter
* remove unused page_match property
* test that non-configured goals are not returned in breakdown
* enforce filtered goals configured
* update changelog
* Allow simple filtering by revenue goals
This does not mean that revenue metrics are supported. If a revenue goal
is filtered by, we treat it like a simple custom event goal in the API.
* use List.wrap
* Move imported tables schemas to separate modules outside Google ns
* Move buffer for imports to Imported ns
* fix schema newlines
* Extract UA import processing and persistence
* Decouple analytics worker implementation from UA
* Rename env variable for import buffer size
* Preserve old import queue until release
* Allow `matches` operator to work in BE for custom props
Note: No FE support yet, needs further testing
* feat: allow choosing `contains` for property filters in the UI
* no autocomplete on prop values if `contains` for consistency
* CHANGELOG.md
* Fix: Handle (none) property in property breakdowns when using matching
When matching we should always exclude (none)
