* add realtime date_ranges into the private API schema
This commit starts parsing date ranges into a new NaiveDateTimeRange
struct, rather than a simple Date.Range.
* transform realtime labels into negative integers + test
* move schema type argument to last position in helper functions
* allow passing a date param + tests
* Update test/plausible/stats/query_parser_test.exs
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Update test/plausible/stats/query_parser_test.exs
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Update test/plausible/stats/query_parser_test.exs
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Update test/plausible/stats/query_parser_test.exs
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* keep test file structure consistent
* Turn NaiveDateTimeRange into DateTimeRange
* change 'now' field from NaiveDateTime to DateTime in v2 query
* fix minute interval labels + add missing tests
* return query_result.date_range as iso8601 timestamps with timezone
* allow timestamps with tz as date_range arguments in API v2
* delete Plausible.Timezones.to_utc_datetime
* simplify returning comparison periods
* add comment about realtime not supported in comparisons
* pass only now instead of test_opts
* drop redundant else branch
* separate tests
* stick to a single check_date_range function in tests
* fix credo error
---------
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Restore `date` internal parameter, validate via json schema
* Improved error formatting from json schema, get most tests passing
* Handle internal overrides to JSON schema
* Parsing tests all pass
* Remove some repeated code, enforce length/uniqueness in schema
* Explicit separation between internal and public API validation
* Mark file as external_resource
* map_join
* Update query tests
* Update query tests
* Serve schema under an /api/docs/query/schema.json endpoint
* dotify errors
* Order QueryResult in API response
This improves experience in docs when querying interactively
* More utm in seeds
* More improved seeds
* Proper QueryResult.query structure
* Allow docs to query /api/v2/query and sites
The new endpoints use cookie authentication. The docs site uses
these endpoints to provide an interactive docs editor.
* query_result ordering test
* Refresh router
* Test module name
* Set log_comment with request information
* CRMAuthPlug -> SuperAdminOnlyPlug
* Super basic debug view
* Handle clustered setups
* Changelog entry
* Cleanup
* fragment trick to use ecto querying, filtering
* Move clustered_table? function to IngestRepo module
* Format
* More resilient user_id getting in helper
* Add data migration for creating and syncing location_data table and dictionary
* Migration to populate location data
* Daily cron to refresh location dataset if changed
* Add support for visit:country_name, visit:region_name and visit:city_name dimensions
Under the hood this relies on a `location_data` table in clickhouse being regularly synced with
plausible/location repo and dictionary lookups used in ALIAS columns
* Update queue name
* Update documentation
* Explicit structs
* Improve docs further
* Migration comment
* Add queues
* Add error when already loaded
* Test for filtering by new dimensions
* Update deps
* dimension -> select_dimension
* Update a test
* Update Goal schema
* Equip ComboBox with the ability of JS selection callbacks
* Update factory so display_name is always present
* Extend Goals context interface
* Update seeds
Also farming unsuspecting BEAM programmers for better
sample page paths :)
* Update ComboBox test
* Unify error message color class with helpers seen elsewhere
* Use goal.display_name where applicable
* Implement LiveView extensions for editing goals
* Sprinkle display name in external stats controller tests
* Format
* Fix goal list mobile view
* Update lib/plausible_web/live/goal_settings/list.ex
Co-authored-by: Artur Pata <artur.pata@gmail.com>
* Update lib/plausible_web/live/goal_settings/form.ex
Co-authored-by: Artur Pata <artur.pata@gmail.com>
* Update the APIs: plugins and external
* Update test so the intent is clearer
* Format
* Update CHANGELOG
* Simplify form tabs tests
* Revert "Format"
This reverts commit c1647b5307.
* Fixup format commit that went too far
* ComboBox: select the input contents on first focus
* Update lib/plausible/goal/schema.ex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Update lib/plausible/goals/goals.ex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Update lib/plausible_web/live/goal_settings/form.ex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Pass form goal instead of just ID
* Make tab component dumber
* Extract separate render functions for edit and create forms
* Update test to account for extracted forms
* Inline goal get query
* Extract revenue goal settings to a component and avoid computing assigns in flight
* Make LV modal preload optional
* Disable preload for goal settings form modal
* Get rid of phash component ID hack
* For another render after render_submit when testing goal updates
* Fix LV preload option
* Enable preload back for goals modal for now
* Make formatter happy
* Implement support for preopening of LV modal
* Preopen goals modal to avoid feedback gap on loading edited goal
* Remove `console.log` call from modal JS
* Clean up display name input IDs
* Make revenue settings functional on first edit again
* Display names: 2nd stage migration
* Update migration with data backfill
---------
Co-authored-by: Artur Pata <artur.pata@gmail.com>
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Add popular browser icons
* Adjust fallback icon color for dark mode
* Add changelog
* Add MIUI browser logo
* Fix Edge and MacOS in seeds
* Add smaller browser logos
* Prioritize CSP over GTM error in case both detected
* Outline better message for persistent headless timeout
* Bump headless waiting times slightly
* Add generic type of error
For known cases, that we rather not reveal much
details about.
* Add the malformed script tag case
* populate site_imports from sites.imported_data in CE
* not dry_run?
* Wrap data migration using `with_repo` to ensure ClickhouseRepo is running
* Do not create funnels in seeds when running CE
* Remove migrator wrapper
* Explicitly raise on rollback attempt
* Put a warning in data migration scripts which are put in migrations
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Simplify browserless script and catch errors
* Improve fetching body:
- follow up on max. 4 redirects
- rely on Req default timeouts (wait much longer to account for slow
sites)
* Improve installation check:
- rely on Req default timeouts, wait longer
- log service errors as warnings
- use stealth mode to bypass captchas
* Stop cutting off body too large
* Improve diagnostics for known failures
* Another round of diagnostic improvements
* Format
* Add a test for callback status 500
* Allow running browserless.io locally
* Compile tailwind classes based on extra/ too
* Add browserless runtime configuration
* Ignore verification events on ingestion
* Improve extracting HTML text in tests
* Update dependencies
- Floki will be used on production to parse site contents
- Req will be used to handle redundant stuff like retrying etc.
* Add shuttle SVG to generic components
Later on we'll use it to indicate verification errors
* Connect live socket & allow skipping awaiting the first pageview
* Connect live socket in general settings
* Implement verification checks & diagnostics
* Stub remote services with Req for testing
* Change snippet screen copy
* Update tracker script, so that:
1. headless browsers aren't ignored if `window.__plausible` is defined
2. callback optionally supplies the event response HTTP status
This will be later used to check whether the server acknowledged
the verification event.
* Implement LiveView verification UI
* Embed the verification UIs into settings and onboarding
* Implement browserless puppeteer verification script
It:
- tries to visit the site
- defines window.__plausible, so the tracker doesn't ignore test events
- sends a verification event and instruments the callback
- awaits the callback to fire and returns the result
* Improve diagnostics for CSP
Only report CSP error if the snippet is already found
* Put verification behind a feature flag/env setting
* Contact Us hint only for Enterprise Edition
* For headless code, use JS context instead of EEx interpolation
* Update diagnostics test with WordPress scenarios
* Shorten exception/throw interception
* Rename test
* Tidy up
* Bust URL always on headless check
* Update moduledoc
* Detect official Plausible WordPress Plugin
and act accordingly on diagnostics interoperation
* Stop using 'rating' in favour of 'interpretation'
* Only report CSP error if no proxy is likely
* Update CHANGELOG
* Allow event-* attributes on snippet elements
* Improve naive GTM detection, not to confuse it with GA4
* Update lib/plausible/verification.ex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Update test/plausible/site/verification/checks_test.exs
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* s/perform_wrapped/perform_safe
* Update lib/plausible/verification/checks/installation.ex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Remove garbage
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Remove references to `site.imported_data`
* Count pre-existing ID 0 imports when showing pageview count summary for legacy imports
* Fix tests after rebase
* Dry `delete_imported_stats!`
* Clean up remaining imported data references and add notes
* Add runtime config option for enabled/disabling csv imports and exports
* Use the new option to toggle rendering exports UI
* Disable import buttons when at maximum imports or when option disabled for CSV
* Improve forms for GA import flow
* Add test for maximum imports reached
* Remove "Changed your mind?" prefixing back button
* Hide UA imports in Integrations when `imports_exports` flag is enabled
* Implement `csv_imports_exports` feature flag
* Revert "Add runtime config option for enabled/disabling csv imports and exports"
This reverts commit e30f202dd3.
* Send import notification email only to the user who ran the import
* Improve rendering of disabled button state
* Put import status heroicon in front of import label
* Add shield hostname rules migration
* Add hostname rule schema
* Initialize hostname rules cache
* Extend Shields context with hostname related functions
* Instrument ingestion pipeline with hostname rule lookups
* Limit hostname suggestions by shield patterns
* Add LiveView for hostname rules management
* Test hostname cache
* Rename feature flag - should be separate from hostname filter
* Remove :shield_pages feature flag
* Update CHANGELOG
* Format
* Update lib/plausible/shield/hostname_rule.ex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Move tests from `lib/` 🤦
* Use plain `assign` where no short-circuit is necessary
* Fine tune the copy a little bit
* Prevent misplaced tests
* Treat a test with common sense
* Fixup another test that hasn't been really run before
* Make the form hint dynamic depending on rules count
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* CH Migration: exit/entry hostnames in sessions_v2
* Leave only exit_page_hostname, we already record hostnames
* Use ClickHouse DDL in favour of ecto so that cluster is included
* Compress with ZSTD(3)
* Expose Hostname filter in the dashboard dropdown
* Add `exit_page_hostname` to ClickHouse `sessions_v2` schema
* Start tracking hostname changes in sessions
* Implement hostname filter suggestions
* Enable filtering by `event:hostname`
* Add tests for filtering by hostnames
* Ensure filter suggestions work for exit pages too
* Allow overriding hostnames with `send_pageview` mix task
* Remove `:window_time_on_page` flag
It seems that we can remove it after all?
* Initialize `experimental_hostname_filter` query parameter
* Rewrite cache store behaviour with regards to session hostnames
* Work around inconsistent session merging
So that `populate_stats` can get closer to actual ingestion
* Improve top stats test
* Make it possible to filter sessions by entry/exit hostnames
* Update pages tests
* Expose `experimental_hostname_filtering` temporarily in the UI
* Untested yet: also apply experimental filtering to sources
* Introduce `hostname_filter` feature flag
* Format
* Test top sources with hostname filter + experimental flag
* CH Migration: exit/entry hostnames in sessions_v2
* Leave only exit_page_hostname, we already record hostnames
* Use ClickHouse DDL in favour of ecto so that cluster is included
* Compress with ZSTD(3)
* Unify GA4 and UA import flow into one
* Clean up property and view data retrieval via Google HTTP APIs
* Turn `Map.get` into `Map.fetch!` in API response processing code
* Bump list account summaries page size limit to max of 200
* Show only views in legacy flow and fix legacy redirect after import start
* Move google analytics import actions tests to a separate module
* Extend Google Analytics controller tests
* DRY up `property?` predicate (h/t @RobertJoonas)
* WIP mutation to populate event session columns
* Remove duplication
* report errors, allow_nondeterministic_updates
* use right columns
* Update existing columns instead of session_* ones
* Make dialyzer happy
* Fix issue with passing pre-existing params in
* Logger -> IO.puts
* Use IngestRepo.config for connection settings
* Make dictionary options configurable
* Move allow_nondeterministic_mutations to within the migration
* Solve credo warning about too deep nesting
* Missed logger call
* Pattern matching in function head
This migration will noop in staging/production as it already has been run. It also leaves
behind a backup table that initially takes no extra space but will need to be cleaned up
manually
* Create a stub of site settings section for imports and exports
* Use legacy site import indication to determine UA import handling
* Add provisional logos for upcoming import sources
* Stub basics of import page
* Add very rudimentary support for multiple UA imports
* Implement imports list as live view
* Add support for opening LV modal from backend and closing from frontend
* Introduce notion of themes to `button` and `button_link` components
* Add confirmation modal on deleting import
* Swap GA4 logo
* Implement disabled state support for `button_link` component
* Disable export and non-implemented import sources
* Use native starts start date for upper boundary of import time range
* Ensure integrations view uses legacy UA import flow
* Remove unnecessary preload in SiteController
* Remove unnecessary exception for legacy imports
* Move API controller stats tests under PlausibleWeb
* Test listing imports
* Add test for explicit listener setup
* Add tests for legacy flag state in UA importer
* Add test for purging legacy import data
* Add tests for `Sites.native_stats_start_date`
* Test forgetting imports
* Add `Stats.Clickhouse.imported_pageview_counts/1` and fix test flakiness
* Show page view counts on imports list
* Add tests for static imports and exports view
* Adjust button look slightly
* Use `case` instead of `cond`
* Make feature flag customisable per site
* Fix buttons and empty state styling
* Add another import to seeds
* Use JS confirm dialog instead of modal for deletion confirmations
* Revert "Add support for opening LV modal from backend and closing from frontend"
This reverts commit 260e6c753032b451542e24be9edc2118790b5a00.
* Default `legacy` to false when inserting new import jobs
* Drop `method` attribute from `button_link` and `unstyled_link` components
