* Expose a few data migration functions, add quiet option to do_run
* Create functions and test acquisition channel logic in clickhouse
Tests were lifted from test/plausible_web/controllers/api/external_controller_test.exs
* Clean up test code a bit
* Property test for acquisition channels
* Handle empty strings properly in reference implementation
* Fix spelling, minor issues
* Revert "Property test for acquisition channels"
This reverts commit 3fa0e0e4eb.
* Only test clickhouse functions
* Solve minor code issue
* update channels logic
* Revert "Only test clickhouse functions"
This reverts commit e12784031a.
* Add more tests
* Add small result assertion
* Make query options explicit in data migrations
* Move multi-query running logic to within datamigration lib
* Unbreak numeric ids migration
* Named params directly to Clickhouse
* Update reference test implementation
---------
Co-authored-by: Uku Taht <uku.taht@gmail.com>
* Add migration adding team related tables and fields
* Add `team_site_transfers` table to the teams migration
* Remove team_id FK from api_keys table
* Change new FK constraints on existing tables to `nilify_all` on delete
* Ensure unique indexes on invitation_id and transfer_id fields
* WIP: Start refactoring revenue metrics
* Hacks to make things work
* Remove old revenue code, remove revenue metrics if needed
* Update query_optimizer docs
* Minor fixes
* Add tests around average/total revenue when non-revenue goal filtering going on
* Optimize, calculate filters as expected (OR-ing clauses)
* Revenue: Handle cases where revenue metrics should not be returned or nil
* Expose revenue metrics in internal schema, add tests
* Docstring
* Remove TODO
* Typegen
* Solve warnings
* Remove nesting
* ce_test fix
* Tag tests as ee_only
* Fix: When filtering by revenue goal and no conversions, return 0.0 instead of nil
* More straight-forward preloading logic
* Refactor comparisons to a new options format
Prerequisite for APIv2 comparison work
* Experiment with default include deduplication
* WIP
Oops, breaks `include.total_rows`
* WIP
* Refactor breakdown.ex
* Pagination fix: dont paginate split subqueries
* Timeseries tests pass
* Aggregate tests use QueryExecutor
* Simplify QueryExecutor
* Handle legacy time-on-page metric in query_executor.ex
No behavioral changes
* Remove keep_requested_metrics
* Clean up imports
* Refactor aggregate.ex to be more straight-forward in output format building
* top stats: compute comparison via apiv2
* Minor cleanups
* WIP: Pipelines
* WIP: refactor for code cleanliness
* QueryExecutor to QueryRunner
* Make compilable
* Comparisons for timeseries works
Except for comparisons where comparison window is bigger than source query window
* Add special case for timeseries
* JSON schema tests for comparisons
* Test comparisons with the new API
* comparison date range parsing improvement
* Make comparisons api internal-only
* typegen
* credo
* Different schemata
* get_comparison_query
* Add comment on timeseries result format
* comparisons typegen
* Percent change for revenue metrics fix
* Use defstruct for query_runner over map
* Remove preloading atoms
Offset-based pagination is used to make sure Looker integration
is able to work as efficiently as possible. To know how many
requests users need to do `include.total_rows` option was added.
* query.date_range is now in UTC instead of user timezone
This simplifies things down the line and fixes several bugs where
query.date_range is cast to naivedatetime for ecto purposes
Many places still remain broken:
- comparison queries
- `to_date_range` calls
* Make default_for_date_range not care about time zones
* Make timezone parameter mandatory for to_date_range
* Simplify utc_date_range, update legacy query builder
* Fix more cases where query date range is needed
* query.date_range -> query.utc_time_range
* Query.date_range/1 function
* ensure_include_imported update
* Clean up send_email_report
* Rename matches/does_not_match filters internally
These have never been exposed to the frontend/user directly, only via
APIv1 filtering syntax. As such we are free to rename these without
breaking things
* Rename function arguments for consistency, simplify
* Add support for `match`/`not_match` operators for query apiv2
These match the string against a regular expression, as defined in
https://github.com/google/re2/wiki/Syntax
* not_match -> match_not
* does_not_contain -> contains_not
Note that for backwards compatibility:
- Browser handles does_not_contain in URL
- Backend will handle does_not_contain in queries for a day where we will remove it for better autocompletion
* not_matches_wildcard -> matches_wildcard_not
* prettier
* match -> matches
* Fix and test fix for matches_wildcard against prop when prop is missing
* Custom properties support for matches/matches_not
* Restore contains_not
* Test contains and contains_not behavior for custom properties
Previous migration took forever on prod, likely because Map lookups are linear time in complexity.
`transform/3` helps achieve the same functionality with the help of a hash table and updated
WHERE clause allows skipping most rows which dont need updating
Co-authored-by: Uku Taht <Uku.taht@gmail.com>
* add realtime date_ranges into the private API schema
This commit starts parsing date ranges into a new NaiveDateTimeRange
struct, rather than a simple Date.Range.
* transform realtime labels into negative integers + test
* move schema type argument to last position in helper functions
* allow passing a date param + tests
* Update test/plausible/stats/query_parser_test.exs
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Update test/plausible/stats/query_parser_test.exs
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Update test/plausible/stats/query_parser_test.exs
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Update test/plausible/stats/query_parser_test.exs
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* keep test file structure consistent
* Turn NaiveDateTimeRange into DateTimeRange
* change 'now' field from NaiveDateTime to DateTime in v2 query
* fix minute interval labels + add missing tests
* return query_result.date_range as iso8601 timestamps with timezone
* allow timestamps with tz as date_range arguments in API v2
* delete Plausible.Timezones.to_utc_datetime
* simplify returning comparison periods
* add comment about realtime not supported in comparisons
* pass only now instead of test_opts
* drop redundant else branch
* separate tests
* stick to a single check_date_range function in tests
* fix credo error
---------
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Restore `date` internal parameter, validate via json schema
* Improved error formatting from json schema, get most tests passing
* Handle internal overrides to JSON schema
* Parsing tests all pass
* Remove some repeated code, enforce length/uniqueness in schema
* Explicit separation between internal and public API validation
* Mark file as external_resource
* map_join
* Update query tests
* Update query tests
* Serve schema under an /api/docs/query/schema.json endpoint
* dotify errors
* Order QueryResult in API response
This improves experience in docs when querying interactively
* More utm in seeds
* More improved seeds
* Proper QueryResult.query structure
* Allow docs to query /api/v2/query and sites
The new endpoints use cookie authentication. The docs site uses
these endpoints to provide an interactive docs editor.
* query_result ordering test
* Refresh router
* Test module name
* Set log_comment with request information
* CRMAuthPlug -> SuperAdminOnlyPlug
* Super basic debug view
* Handle clustered setups
* Changelog entry
* Cleanup
* fragment trick to use ecto querying, filtering
* Move clustered_table? function to IngestRepo module
* Format
* More resilient user_id getting in helper
* Add data migration for creating and syncing location_data table and dictionary
* Migration to populate location data
* Daily cron to refresh location dataset if changed
* Add support for visit:country_name, visit:region_name and visit:city_name dimensions
Under the hood this relies on a `location_data` table in clickhouse being regularly synced with
plausible/location repo and dictionary lookups used in ALIAS columns
* Update queue name
* Update documentation
* Explicit structs
* Improve docs further
* Migration comment
* Add queues
* Add error when already loaded
* Test for filtering by new dimensions
* Update deps
* dimension -> select_dimension
* Update a test
* Update Goal schema
* Equip ComboBox with the ability of JS selection callbacks
* Update factory so display_name is always present
* Extend Goals context interface
* Update seeds
Also farming unsuspecting BEAM programmers for better
sample page paths :)
* Update ComboBox test
* Unify error message color class with helpers seen elsewhere
* Use goal.display_name where applicable
* Implement LiveView extensions for editing goals
* Sprinkle display name in external stats controller tests
* Format
* Fix goal list mobile view
* Update lib/plausible_web/live/goal_settings/list.ex
Co-authored-by: Artur Pata <artur.pata@gmail.com>
* Update lib/plausible_web/live/goal_settings/form.ex
Co-authored-by: Artur Pata <artur.pata@gmail.com>
* Update the APIs: plugins and external
* Update test so the intent is clearer
* Format
* Update CHANGELOG
* Simplify form tabs tests
* Revert "Format"
This reverts commit c1647b5307.
* Fixup format commit that went too far
* ComboBox: select the input contents on first focus
* Update lib/plausible/goal/schema.ex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Update lib/plausible/goals/goals.ex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Update lib/plausible_web/live/goal_settings/form.ex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Pass form goal instead of just ID
* Make tab component dumber
* Extract separate render functions for edit and create forms
* Update test to account for extracted forms
* Inline goal get query
* Extract revenue goal settings to a component and avoid computing assigns in flight
* Make LV modal preload optional
* Disable preload for goal settings form modal
* Get rid of phash component ID hack
* For another render after render_submit when testing goal updates
* Fix LV preload option
* Enable preload back for goals modal for now
* Make formatter happy
* Implement support for preopening of LV modal
* Preopen goals modal to avoid feedback gap on loading edited goal
* Remove `console.log` call from modal JS
* Clean up display name input IDs
* Make revenue settings functional on first edit again
* Display names: 2nd stage migration
* Update migration with data backfill
---------
Co-authored-by: Artur Pata <artur.pata@gmail.com>
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Add popular browser icons
* Adjust fallback icon color for dark mode
* Add changelog
* Add MIUI browser logo
* Fix Edge and MacOS in seeds
* Add smaller browser logos
* Prioritize CSP over GTM error in case both detected
* Outline better message for persistent headless timeout
* Bump headless waiting times slightly
* Add generic type of error
For known cases, that we rather not reveal much
details about.
* Add the malformed script tag case
* populate site_imports from sites.imported_data in CE
* not dry_run?
* Wrap data migration using `with_repo` to ensure ClickhouseRepo is running
* Do not create funnels in seeds when running CE
* Remove migrator wrapper
* Explicitly raise on rollback attempt
* Put a warning in data migration scripts which are put in migrations
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Simplify browserless script and catch errors
* Improve fetching body:
- follow up on max. 4 redirects
- rely on Req default timeouts (wait much longer to account for slow
sites)
* Improve installation check:
- rely on Req default timeouts, wait longer
- log service errors as warnings
- use stealth mode to bypass captchas
* Stop cutting off body too large
* Improve diagnostics for known failures
* Another round of diagnostic improvements
* Format
* Add a test for callback status 500
