* Fix parsing = in jsonurl
This works around a bug in the jsonurl library where the literal causes a syntaxerror.
The urlencoded = however is handled without errors
* Changelog
* move imported.ex to imported subfolder
* move constructing base imported query into a separate module
* Implement imported table deciding and filtering
+ tests for pages, entry_pages, exit_pages and common filter types
* add top stats test with country filter
* add timeseries test
* Drop bounce_rate and time_on_page from imported & page-filtered Top Stats
* rename field returned by top stats
* turn pages into a fn comp
* Move dashboard API results under a results key
...and also return the skip_imported_reason to the frontend to be used
for displaying warnings.
* extend ListReport component with an optional afterFetchData prop
* turn Devices into a fn comp
* add not_requested as a skip_imported_reason
* display warning icons in the dashboard
* Implement filtering suggestions and translate filter fields for imported
* WIP
* Improve and cover filtering suggestions with tests
* Rename imported suggestions query helpers
* fix screen size breakdown with screen size filter
* support filtering by the same suggestion property
* support location filters when fetching location suggestions
* support filtering by multiple props from the same table
* Implement filtering by goals
* Make views per visit metric work for import entry and exit pages
* Get rid of circular dependencies between Stats.Imported and Stats.Imported.Base
* Clean up Query struct manipulation in Breakdown
* Rename helper function for clarity
* Automatically refresh query struct state after modifications
* Shutup credo
* display imported warning bubble in prop breakdown section
* Render warning bubble for funnels whenever imported data is in the view
* Transform any operator on respective goal filters
* Fix percentage and conversion_rate calculation in presence of custom props
* add tests for for combining page and pageview goal filters
* add skip_refresh option to query tweaking functions
* add imported CR support for timeseries
* still show url breakdown when special goal + url in filter
* rename Query.refresh
* use flat_map instead of map and concat
* fix darkmode color
* Handle invalid imported region codes in suggestions gracefully
* Add an entry to CHANGELOG.md
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Allow running browserless.io locally
* Compile tailwind classes based on extra/ too
* Add browserless runtime configuration
* Ignore verification events on ingestion
* Improve extracting HTML text in tests
* Update dependencies
- Floki will be used on production to parse site contents
- Req will be used to handle redundant stuff like retrying etc.
* Add shuttle SVG to generic components
Later on we'll use it to indicate verification errors
* Connect live socket & allow skipping awaiting the first pageview
* Connect live socket in general settings
* Implement verification checks & diagnostics
* Stub remote services with Req for testing
* Change snippet screen copy
* Update tracker script, so that:
1. headless browsers aren't ignored if `window.__plausible` is defined
2. callback optionally supplies the event response HTTP status
This will be later used to check whether the server acknowledged
the verification event.
* Implement LiveView verification UI
* Embed the verification UIs into settings and onboarding
* Implement browserless puppeteer verification script
It:
- tries to visit the site
- defines window.__plausible, so the tracker doesn't ignore test events
- sends a verification event and instruments the callback
- awaits the callback to fire and returns the result
* Improve diagnostics for CSP
Only report CSP error if the snippet is already found
* Put verification behind a feature flag/env setting
* Contact Us hint only for Enterprise Edition
* For headless code, use JS context instead of EEx interpolation
* Update diagnostics test with WordPress scenarios
* Shorten exception/throw interception
* Rename test
* Tidy up
* Bust URL always on headless check
* Update moduledoc
* Detect official Plausible WordPress Plugin
and act accordingly on diagnostics interoperation
* Stop using 'rating' in favour of 'interpretation'
* Only report CSP error if no proxy is likely
* Update CHANGELOG
* Allow event-* attributes on snippet elements
* Improve naive GTM detection, not to confuse it with GA4
* Update lib/plausible/verification.ex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Update test/plausible/site/verification/checks_test.exs
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* s/perform_wrapped/perform_safe
* Update lib/plausible/verification/checks/installation.ex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Remove garbage
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Start refactoring, supporting jsonurl
* WIP: Listing of applied filters works
* WIP: Remove some unneeded hackery
* WIP: Handle country labels better
* WIP: Handle multiple filters better
* WIP: Serializing filters for API queries
* Split modal code into 3
* Merge code paths for prop and other filter modals
* Get suggestions working again
* Display prop filters properly
* Handle re-opening a props modal properly
* Better label handling
* Better linking to filter modals
* Remove unneeded component
* Standardize how we update query more
* Use updatedQuery to remove more usecases of URLSearchParams
* Dont export toFilterQuery
* Custom toString for PlausibleSearchParams
* Fix props suggestions/filtering
* Refactor isFilteringOnFixedValue
* Improved encoding - goals now work again
* fix a typo
* Handle more cases where query.filters[ is used
* Fix locations tab changing behavior
* Fix for `setQuery` not to double up ?
* Handle goal filters properly now
* Delete dead code
* Update special goals handling
* Update <ListReport /> linking
* Show labeled values in list of filters
* Updae Props component handling of storage keys
* re-add special case handling in devices view
* Fix modal-related typo
* Get updatedQuery callsites working
* Update location modals linking
* Update props details model linking logic
* Switch back tab from props when removing goal filter
* Remove query.filters usage from within <Referrers /> component
* Private escapeFilterValue
* Fix sources/index.js
* Legacy redirect logic
* Update comment
* Disabled options in props modal
* Update escaping and is_not operator
* Restore `false` search property handling meaning unset
* changelog
* Fix filtering after clicking on a map
* FilterOperatorSelector
* replaceFilterByPrefix
* Improve naming for filter modals/groups
* Export and import custom events via CSV
* Add prop support of url for cloaked links and path for 404s in imported queries
* Handle custom events with empty URL and path properties gracefully
* Make events with properties logic DRY and fix missed cloaked link
* Add test for path property breakdown
* Update raw CH data fixture and extend CSV importer tests
* Fix broken query condition after rebase
* Update CHANGELOG.md
* Apply filters in search console request
* Remove dead code from search console modal
* Remove unimportant information from keyword modal
* Show invalid filters from search console
* Fix tests
* Add/Fix tests
* Fix typo
* Remove unused variable
* Fix typo
* Changelog entry
* Fix Credo
* Display impressions, CTR and position in keyword modal
* Undo change that should not have been committed
* Fix test
* Fix test
* filters -> search_console_filters
* Add Ecto schema for imported custom events
* Start importing custom events from GA4
* query imported goals
* make it possible to query events metric from imported
* make it possible to query pageviews in goal breakdown
* make it possible to query conversion rate
* fix rate limiting test
* add CR tests for dashboard API
* implement imported link_url breakdown
* override special custom event names coming from GA4
* allow specific goal filters in imported_q
* update GA4 import tests to use Stats API
* Improve tests slightly
* Update CHANGELOG.md
---------
Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
* Add shield hostname rules migration
* Add hostname rule schema
* Initialize hostname rules cache
* Extend Shields context with hostname related functions
* Instrument ingestion pipeline with hostname rule lookups
* Limit hostname suggestions by shield patterns
* Add LiveView for hostname rules management
* Test hostname cache
* Rename feature flag - should be separate from hostname filter
* Remove :shield_pages feature flag
* Update CHANGELOG
* Format
* Update lib/plausible/shield/hostname_rule.ex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Move tests from `lib/` 🤦
* Use plain `assign` where no short-circuit is necessary
* Fine tune the copy a little bit
* Prevent misplaced tests
* Treat a test with common sense
* Fixup another test that hasn't been really run before
* Make the form hint dynamic depending on rules count
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Use sessionStorage for offer e-mail report banner tracking
Keeping it within the cookie is problematic, as the banners don't
expire and overflow the cookie with data when enough new sites
are added.
Ref https://github.com/plausible/analytics/issues/3762
* Update changelog
* Extract a component
* Make is_dbip evaluate to quoted boolean
* Ignore sessions without entry/exit pages when breaking down entry/exit pages
* Update stats controller tests to have more realistic test data (pageview followed by event)
* Add validation for the events metric in main_graph
* Test the already existing events metric support in main-graph
* Put total conversions on the graph
* extract main_graph_csv function (refactor only)
* add total_conversions and conversion_rate to goal-filtered visitors.csv
* update changelog
* add conversion rate to Stats API timeseries
* make sure CR can be queried as the only metric
* add a test asserting zeros are returned
* add tests for filtering by other properties at the same time
* Remove unnecessary validation of params
1. It doesn't make to validate `interval` (and its granularity) in all
endpoints. It's only relevant for the main graph.
2. The plug (renamed to `date_validation_plug`) already makes sure that
the dates are validated. No need to call the same function again in
Top Stats and Funnel endpoints.
* add metric validation to main graph
* Add tests for main graph API
* put conversion rate on the graph
* update changelog
* Add revenue metrics into metrics.ex
* make fn private
* avoid setting graph metric to visitors in goal-filtered view
* UX improvement: don't autofocus on Custom properties on dashboard
Currently when you visit the dashboard with a certain configuration,
it will automatically scroll to the bottom, which is annoying.
To reproduce:
1. localStorage.clear() in console
2. Refresh, open properties at the very bottom
3. Refresh again
Expected behavior:
1. Dashboard shows
Actual behavior:
1. Dashboard scrolls to the bottom
* Changelog entry
* add metric validation + support in aggregate
* add a test ensuring comparison works
* disallow time_on_page with a goal filter
* Return time_on_page as `nil` from aggregate API
In case time_on_page cannot be calculated, we'll return it as `nil` from
the Stats API.
This is to make the behaviour consistent between breakdown and aggregate
endpoints. As for the UI, we'll still continue to report time_on_page as
0 - not changing any UI behaviour as discussed with Marko.
* add tests for time_on_page in event:page breakdown
* update changelog
* invalidate time_on_page with event:name filter
* add the ability to only query time_on_page in page breakdown
We'll need the visitors metric to get the list of pages to calculate the
time_on_page for.
* wip
* more env, setup user before packages to avoid them picking our uid
* make entrypoint.sh executable by all again
* apparently it's a best practice to have executables be owned by root
* make entrypoint executable in COPY
* stop writing to /app, write to /tmp if needed
* fewer changes
* system user
* keep same style for multiline commands
* fewer changes
* add changelog entry
* fix group assignment for plausible user
* use gid=999
* no home
* no home
* add gecos
* add plausible user to nogroup instead of creating a custom one
* eh
* fewer changes
* fewer changes
* fewer changes
* use PERSISTENT_CACHE_DIR instead of STORAGE_DIR
* ignore more
* cleanup
* remove hex timeout env var
* use ERL_FLAGS=+JMsingle true in public builds
* fallback to /tmp and nest under /tzdata_data/ for tzdata
---------
Co-authored-by: Cenk Kücük <cenk@plausible.io>
This migration will noop in staging/production as it already has been run. It also leaves
behind a backup table that initially takes no extra space but will need to be cleaned up
manually
* Migration: add country rules
* Add CountryRule schema
* Implement CountryRule cache
* Add country rules context interface
* Start country rules cache
* Lookup country rules on ingestion
* Remove :shields feature flag from test helpers
* Add nested sidebar menu for Shields
* Fix typo
* IP Rules: hide description on mobile view
* Prepare SiteController to handle multiple shield types
* Seed some country shield
* Implement LV for country rules
* Remove "YOU" indicator from country rules
* Fix small build
* Format
* Update typespecs
* Make docs link point at /countries
* Fix flash on top of modal for Safari
* Build the rule struct with site_id provided up-front
* Clarify why we're messaging the ComboBox component
* Re-open combobox suggestions after pressing Escape
* Update changelog
* Fix font size in country table cells
* Pass `added_by` via rule add options
* Display site's timezone timestamps in rule tooltips
* Display formatted timestamps in site's timezone
And simplify+test Timezone module; an input timestamp converted
to UTC can never be ambiguous.
* Remove no-op atom
* Display the maximum number of rules when reached
* Improve readability of remove button tests
* Credo
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* improve test
* add os to os_version breakdown
...and add operating_system_versions.csv to the CSV export
* fix conversion rate for os_version breakdown
* update changelog
* fix existing CSV tests
* use case instead of cond
* Revert "Unify percentage change for CR and bounce_rate (#3781)"
This reverts commit a6b1a6ebc7.
* Revert "Bring Stats API up to speed: Add `conversion_rate` to Aggregate and Breakdown (#3739)"
This reverts commit 672d682e95.
* Fix conversion rate change calculation
The change in conversion rate should be calculated similar to bounce rate.
For example, an increase of 25% -> 50% should not be a 100% change, but
a 25% change instead.
* Use the same comparison function in Stats API and dashboard API
This commit fixes a bug where the percentage change reported by the Stats
API is different from the one returned by the internal dashboard API.
* changelog update
* disable event metric with include_imported in every case
* add missing test for metric validation
* refactor metric validation functions
* implement conversion_rate metric validation
* move calculate_cr function into Stats.Util
* Refactor: Move aggregate CR logic into Stats.aggregate
* define atoms to exist
* Ensure that CR does not depend on visitors being queried
If 'visitors' are already queried, we'll use that value. Otherwise we'll
need to make another query to fetch it.
* confirm Stats API aggregate supports CR (tests only)
* small refactor
This is the only 'event_property' left after pattern matching on all
others in the function clauses defined above.
* Make it possible to optionally query conversion_rate
...in breakdown queries (excluding goal and custom prop breakdown)
* A little refactor asking for revenue metrics
1. The `@revenue_metrics` module attribute is an empty list on full build
anyway
2. We don't need to query for revenue metrics if there are no revenue goals
returned in the given query (even if revenue goals exist in site.goals)
3. Revenue metrics are already dropped in prop breakdown without a goal
filter via (get_revenue_tracking_currency/3)
* Make it possible to optionally query conversion_rate (continuation)
... also from a custom prop and goal breakdown
* Frontend adjustments to the Locations report
* Display conversion rate in Regions and Cities (ListReport view)
* Display total conversions, conversions (visitors), and CR in the
"Details" modals of Countries, Regions, and Cities
* Move the percentage into a separate column in the Countries details table
* confirm Stats API breakdown supports conversion_rate (tests only)
* small refactor: extract maybe_add_time_on_page function
* Make it possible to query cr alone
... (without the visitors metric). Already supported in aggregate, this
commit only implements it for the breakdown API.
* Reuse Stats.Util helper functions from b02db88 for aggregate API
We can follow the same logic as with breakdown for manually adding
`visitors` into the metrics list and taking it out of the response
later on.
That way we don't have to make another query, e.g. in a case where
only pageviews and conversion rate is queried. Also keeps things
consistent.
* changelog update
* fix test after resolving merge conflict
* Use explicit string->atom mapping instead of casting
* alias Util module instead of importing it
* use Enum.empty instead of Enum.any
* improve readability
* rename special_metrics to computed_metrics and explain with a comment
* rename visitors_without_event_filters to total_visitors
* keep a single function for removing unwanted metrics
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Support using matches/contains for most filters
* Change behavior where we auto-zoom to specific browser/os/source to only do so if filtering on a single value
* No contains filtering on `location`
* Update CHANGELOG.md
* Fix merge conflict
* Add Ecto.Network dependency
* Migration: Add ip block list table
* If Cachex errors out, mark the cache as not ready
* Add IPRule schema
* Seed IPRules
* Add Shields context module
* Implement IPRuleCache
* Start IPRuleCache
* Drop blocklisted IPs on ingestion
* Cosmetic rename
* Add settings sidebar item
* Consider IPRuleCache readiness on health checks
* Fix typo
* Implement IP blocklist live view
* Update moduledocs
* Extend contextual module tests
* Convert IPRules LiveView into LiveComponent
* Keep live flashes on the tabs view
* Update changelog
* Format
* Credo
* Remove garbage
* Update drop reason typespecs
* Update typespecs for cache keys
* Keep track of who added a rule and when
* Test if adding via LV prefills the updated_by tooltip
* Update ecto_network dependency
* s/updated_by/added_by
* s/drop_blocklist_ip/drop_shield_rule_ip
* Add docs link
* s/Updated/Added
* WIP: PropFilterRow
* Get multi-behavior working
* Render multiple prop filters in one
* Modal reads from query string correctly
* Backend support for multiple custom property filters
* Add backend tests for multiple custom property filters
* Disable already selected options in property keys
We can't allow choosing the same property multiple times without changing the request
params, which we decided against
* Allow choosing any property under Behaviors > Custom props even if custom prop filter applied
This was a limitation (I believe) introduced by using ARRAY JOINs to query custom properties
* CHANGELOG.md
* Solve credo warning about too deep nesting
* Update assets/js/dashboard/stats/modals/prop-filter-modal.js
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
* Refactor internal function for clarity
* Add another step -> Add another
* Solve 500 error
* Separate boxes per property filter
* Retain other filters in props table
* removeFilter behavior for props
* matches_member support for custom props
* filter_suggestions for prop keys should account for prop filter
* find over filter
* refactor appliedFilters
* FILTER_TYPES => FILTER_OPERATIONS
* Make add another link not wrap the whole page
* Unique keys
---------
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
* add tests for filtering by goal in timeseries and aggregate
* refactor filter parsing
* stop returning custom props in event:goal breakdown
* test breaking down wildcard pageview goals
* extract filter utils
* parse more goal filter options
* add passing tests for new filter types
* do not allow querying session metrics with a goal filter
* remove unused page_match property
* test that non-configured goals are not returned in breakdown
* enforce filtered goals configured
* update changelog
* Allow simple filtering by revenue goals
This does not mean that revenue metrics are supported. If a revenue goal
is filtered by, we treat it like a simple custom event goal in the API.
* use List.wrap
* Allow `matches` operator to work in BE for custom props
Note: No FE support yet, needs further testing
* feat: allow choosing `contains` for property filters in the UI
* no autocomplete on prop values if `contains` for consistency
* CHANGELOG.md
* Fix: Handle (none) property in property breakdowns when using matching
When matching we should always exclude (none)
* remove unused code
* add e.name == "pageview" condition to pageview goals
This fixes the weird behavior where filtering by a pageview goal would
also return custom events and vice versa.
* update changelog
