Commit Graph

293 Commits

Author SHA1 Message Date
Adrian Gruntkowski
533bf90329
Create user_sessions table (#4511) 2024-09-03 10:02:43 +02:00
Uku Taht
77248c8800
Add data migration for capitalizing sources (#4418) 2024-09-02 13:59:58 +03:00
RobertJoonas
f04c47f881
Support realtime periods in API v2 (#4469)
* add realtime date_ranges into the private API schema

This commit starts parsing date ranges into a new NaiveDateTimeRange
struct, rather than a simple Date.Range.

* transform realtime labels into negative integers + test

* move schema type argument to last position in helper functions

* allow passing a date param + tests

* Update test/plausible/stats/query_parser_test.exs

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* Update test/plausible/stats/query_parser_test.exs

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* Update test/plausible/stats/query_parser_test.exs

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* Update test/plausible/stats/query_parser_test.exs

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* keep test file structure consistent

* Turn NaiveDateTimeRange into DateTimeRange

* change 'now' field from NaiveDateTime to DateTime in v2 query

* fix minute interval labels + add missing tests

* return query_result.date_range as iso8601 timestamps with timezone

* allow timestamps with tz as date_range arguments in API v2

* delete Plausible.Timezones.to_utc_datetime

* simplify returning comparison periods

* add comment about realtime not supported in comparisons

* pass only now instead of test_opts

* drop redundant else branch

* separate tests

* stick to a single check_date_range function in tests

* fix credo error

---------

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
2024-09-02 12:56:58 +03:00
hq1
05136398cf
Migration: add installation meta (#4486) 2024-08-29 11:05:36 +02:00
Karl-Aksel Puulmann
9c71161eab
APIv2: JSON schema validation, separate internal and public API validation (#4464)
* Restore `date` internal parameter, validate via json schema

* Improved error formatting from json schema, get most tests passing

* Handle internal overrides to JSON schema

* Parsing tests all pass

* Remove some repeated code, enforce length/uniqueness in schema

* Explicit separation between internal and public API validation

* Mark file as external_resource

* map_join

* Update query tests

* Update query tests

* Serve schema under an /api/docs/query/schema.json endpoint

* dotify errors
2024-08-26 14:01:27 +03:00
ruslandoga
b64038af48
Fix alias migration warning (#4465) 2024-08-26 11:16:13 +02:00
Karl-Aksel Puulmann
11acadfde9
APIv2: docs-related changes (#4453)
* Order QueryResult in API response

This improves experience in docs when querying interactively

* More utm in seeds

* More improved seeds

* Proper QueryResult.query structure

* Allow docs to query /api/v2/query and sites

The new endpoints use cookie authentication. The docs site uses
these endpoints to provide an interactive docs editor.

* query_result ordering test

* Refresh router

* Test module name
2024-08-22 10:44:41 +03:00
Karl-Aksel Puulmann
4967960278
Populate log_comment with debug information, /debug/clickhouse route (#4435)
* Set log_comment with request information

* CRMAuthPlug -> SuperAdminOnlyPlug

* Super basic debug view

* Handle clustered setups

* Changelog entry

* Cleanup

* fragment trick to use ecto querying, filtering

* Move clustered_table? function to IngestRepo module

* Format

* More resilient user_id getting in helper
2024-08-14 12:33:36 +03:00
Karl-Aksel Puulmann
b88074bf1b
Fix migration typo (#4437) 2024-08-13 12:07:18 +03:00
Karl-Aksel Puulmann
ee3d1e770e
APIv2: visit:country_name, visit:region_name, visit:city_name dimensions (#4328)
* Add data migration for creating and syncing location_data table and dictionary

* Migration to populate location data

* Daily cron to refresh location dataset if changed

* Add support for visit:country_name, visit:region_name and visit:city_name dimensions

Under the hood this relies on a `location_data` table in clickhouse being regularly synced with
plausible/location repo and dictionary lookups used in ALIAS columns

* Update queue name

* Update documentation

* Explicit structs

* Improve docs further

* Migration comment

* Add queues

* Add error when already loaded

* Test for filtering by new dimensions

* Update deps

* dimension -> select_dimension

* Update a test
2024-08-13 09:44:58 +03:00
hq1
7fb2bfbd29
Migration: turn google auth tokens into text column type (#4428) 2024-08-09 12:17:26 +02:00
hq1
cc769dfb3d
Edit goals with display names (#4415)
* Update Goal schema

* Equip ComboBox with the ability of JS selection callbacks

* Update factory so display_name is always present

* Extend Goals context interface

* Update seeds

Also farming unsuspecting BEAM programmers for better
sample page paths :)

* Update ComboBox test

* Unify error message color class with helpers seen elsewhere

* Use goal.display_name where applicable

* Implement LiveView extensions for editing goals

* Sprinkle display name in external stats controller tests

* Format

* Fix goal list mobile view

* Update lib/plausible_web/live/goal_settings/list.ex

Co-authored-by: Artur Pata <artur.pata@gmail.com>

* Update lib/plausible_web/live/goal_settings/form.ex

Co-authored-by: Artur Pata <artur.pata@gmail.com>

* Update the APIs: plugins and external

* Update test so the intent is clearer

* Format

* Update CHANGELOG

* Simplify form tabs tests

* Revert "Format"

This reverts commit c1647b5307.

* Fixup format commit that went too far

* ComboBox: select the input contents on first focus

* Update lib/plausible/goal/schema.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update lib/plausible/goals/goals.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update lib/plausible_web/live/goal_settings/form.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Pass form goal instead of just ID

* Make tab component dumber

* Extract separate render functions for edit and create forms

* Update test to account for extracted forms

* Inline goal get query

* Extract revenue goal settings to a component and avoid computing assigns in flight

* Make LV modal preload optional

* Disable preload for goal settings form modal

* Get rid of phash component ID hack

* For another render after render_submit when testing goal updates

* Fix LV preload option

* Enable preload back for goals modal for now

* Make formatter happy

* Implement support for preopening of LV modal

* Preopen goals modal to avoid feedback gap on loading edited goal

* Remove `console.log` call from modal JS

* Clean up display name input IDs

* Make revenue settings functional on first edit again

* Display names: 2nd stage migration

* Update migration with data backfill

---------

Co-authored-by: Artur Pata <artur.pata@gmail.com>
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-08-09 11:12:00 +02:00
hq1
2e6ba4ba42
Display names: 1st stage migration (#4425)
* Display names: 1st stage migration

* Trim migration scope for the 1st run
2024-08-09 10:51:31 +02:00
Adrian Gruntkowski
df5e2cba12
Create migration adding help_scout_mappings table (#4395) 2024-07-26 15:22:59 +02:00
hq1
1c39f06ae8
Add curl browser icon (#4372) 2024-07-18 15:30:28 +02:00
Adrian Gruntkowski
ff778436c4
Add migration for table storing HelpScout credentials (#4346) 2024-07-12 11:35:41 +02:00
hq1
76da6b49a1
Migration: add traffic change type (#4325)
* Migration: add traffic change type

* Update schema
2024-07-11 11:01:11 +02:00
Uku Taht
b3669540bd
OS icons (#4254)
* Add popular OS icons

* Show OS name and icon in drilldown as well

* Fix OS version filter links
2024-07-09 13:22:37 +03:00
hq1
c871759fc1
Bugfix: opening details UTM sources modal (#4295) 2024-07-01 12:39:55 +02:00
Uku Taht
009a9da1a7
Update ua_inspector (#4284) 2024-07-01 09:30:09 +02:00
Uku Taht
486ee68ca8
Browser icon improvements (#4246)
* Use png for Safari icon

* Fix spacing for screen sizes

* Add browser icon to version drilldown
2024-06-20 17:32:08 +03:00
Uku Taht
0060954372
Add browser icons (#4239)
* Add popular browser icons

* Adjust fallback icon color for dark mode

* Add changelog

* Add MIUI browser logo

* Fix Edge and MacOS in seeds

* Add smaller browser logos
2024-06-18 12:45:08 +03:00
hq1
1c820ba2af
Verification tweaks (#4234)
* Prioritize CSP over GTM error in case both detected

* Outline better message for persistent headless timeout

* Bump headless waiting times slightly

* Add generic type of error

For known cases, that we rather not reveal much
details about.

* Add the malformed script tag case
2024-06-18 05:58:56 +02:00
ruslandoga
97e0255a2b
Wrap data migration using with_repo to ensure ClickhouseRepo is running (#4190)
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-06-06 09:39:14 +02:00
ruslandoga
dc7243ff2e
Populate site_imports from sites.imported_data in CE (#4155)
* populate site_imports from sites.imported_data in CE

* not dry_run?

* Wrap data migration using `with_repo` to ensure ClickhouseRepo is running

* Do not create funnels in seeds when running CE

* Remove migrator wrapper

* Explicitly raise on rollback attempt

* Put a warning in data migration scripts which are put in migrations

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-05-29 17:37:21 +02:00
ruslandoga
a139e8295c
Fix warning in migration (#4133) 2024-05-24 16:26:25 +02:00
hq1
ee61094023
Verification improvements (#4132)
* Simplify browserless script and catch errors

* Improve fetching body:
    - follow up on max. 4 redirects
    - rely on Req default timeouts (wait much longer to account for slow
      sites)

* Improve installation check:
  - rely on Req default timeouts, wait longer
  - log service errors as warnings
  - use stealth mode to bypass captchas

* Stop cutting off body too large

* Improve diagnostics for known failures

* Another round of diagnostic improvements

* Format

* Add a test for callback status 500
2024-05-24 10:40:59 +02:00
hq1
c81cb16933
Snippet integration verification (#4106)
* Allow running browserless.io locally

* Compile tailwind classes based on extra/ too

* Add browserless runtime configuration

* Ignore verification events on ingestion

* Improve extracting HTML text in tests

* Update dependencies

- Floki will be used on production to parse site contents
- Req will be used to handle redundant stuff like retrying etc.

* Add shuttle SVG to generic components

Later on we'll use it to indicate verification errors

* Connect live socket & allow skipping awaiting the first pageview

* Connect live socket in general settings

* Implement verification checks & diagnostics

* Stub remote services with Req for testing

* Change snippet screen copy

* Update tracker script, so that:

1. headless browsers aren't ignored if `window.__plausible` is defined
2. callback optionally supplies the event response HTTP status

This will be later used to check whether the server acknowledged
the verification event.

* Implement LiveView verification UI

* Embed the verification UIs into settings and onboarding

* Implement browserless puppeteer verification script

It:
 - tries to visit the site
 - defines window.__plausible, so the tracker doesn't ignore test events
 - sends a verification event and instruments the callback
 - awaits the callback to fire and returns the result

* Improve diagnostics for CSP

Only report CSP error if the snippet is already found

* Put verification behind a feature flag/env setting

* Contact Us hint only for Enterprise Edition

* For headless code, use JS context instead of EEx interpolation

* Update diagnostics test with WordPress scenarios

* Shorten exception/throw interception

* Rename test

* Tidy up

* Bust URL always on headless check

* Update moduledoc

* Detect official Plausible WordPress Plugin

and act accordingly on diagnostics interoperation

* Stop using 'rating' in favour of 'interpretation'

* Only report CSP error if no proxy is likely

* Update CHANGELOG

* Allow event-* attributes on snippet elements

* Improve naive GTM detection, not to confuse it with GA4

* Update lib/plausible/verification.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update test/plausible/site/verification/checks_test.exs

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* s/perform_wrapped/perform_safe

* Update lib/plausible/verification/checks/installation.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Remove garbage

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-05-23 15:00:50 +02:00
Adrian Gruntkowski
d8435f2e01
Remove imports_exports and csv_imports_exports feature flags (#4089) 2024-05-09 10:09:24 +02:00
RobertJoonas
c106595be0
Migration: add imported custom events (#4076)
* Add `imported_custom_events` to CH

* remove redundant table setting

* add path column

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-05-07 11:46:27 +01:00
Karl-Aksel Puulmann
850a843d82
Add migration to add ALIAS columns to common session/visit properties (#4058)
This allows for simplifications in the API code
2024-05-05 11:30:39 +03:00
Adrian Gruntkowski
069170eb1d
Add active_visitors column to imported_pages CH table (#4028) 2024-04-22 09:34:01 +02:00
Adrian Gruntkowski
c10580777e
Remove references to site.imported_data (#4006)
* Remove references to `site.imported_data`

* Count pre-existing ID 0 imports when showing pageview count summary for legacy imports

* Fix tests after rebase

* Dry `delete_imported_stats!`

* Clean up remaining imported data references and add notes
2024-04-19 11:15:51 +02:00
hq1
83643450c4
Remove obsolete feature flags (#4026) 2024-04-18 20:18:38 +02:00
Adrian Gruntkowski
9bae3ccce3
Improve UI/UX of imports view and GA import flow (#4017)
* Add runtime config option for enabled/disabling csv imports and exports

* Use the new option to toggle rendering exports UI

* Disable import buttons when at maximum imports or when option disabled for CSV

* Improve forms for GA import flow

* Add test for maximum imports reached

* Remove "Changed your mind?" prefixing back button

* Hide UA imports in Integrations when `imports_exports` flag is enabled

* Implement `csv_imports_exports` feature flag

* Revert "Add runtime config option for enabled/disabling csv imports and exports"

This reverts commit e30f202dd3.

* Send import notification email only to the user who ran the import

* Improve rendering of disabled button state

* Put import status heroicon in front of import label
2024-04-18 12:12:48 +02:00
hq1
f635f0a6d3
Hostnames shield (#3990)
* Add shield hostname rules migration

* Add hostname rule schema

* Initialize hostname rules cache

* Extend Shields context with hostname related functions

* Instrument ingestion pipeline with hostname rule lookups

* Limit hostname suggestions by shield patterns

* Add LiveView for hostname rules management

* Test hostname cache

* Rename feature flag - should be separate from hostname filter

* Remove :shield_pages feature flag

* Update CHANGELOG

* Format

* Update lib/plausible/shield/hostname_rule.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Move tests from `lib/` 🤦

* Use plain `assign` where no short-circuit is necessary

* Fine tune the copy a little bit

* Prevent misplaced tests

* Treat a test with common sense

* Fixup another test that hasn't been really run before

* Make the form hint dynamic depending on rules count

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-04-16 20:30:20 +02:00
hq1
cf61e47a0a
Add shield hostname rules migration (#3992) 2024-04-11 12:00:01 +02:00
Uku Taht
f966419a4a
Update ua_inspector (#3957)
Co-authored-by: hq1 <hq@mtod.org>
2024-04-04 17:20:57 +02:00
hq1
6af80dd246
Filter by hostnames (#3963)
* CH Migration: exit/entry hostnames in sessions_v2

* Leave only exit_page_hostname, we already record hostnames

* Use ClickHouse DDL in favour of ecto so that cluster is included

* Compress with ZSTD(3)

* Expose Hostname filter in the dashboard dropdown

* Add `exit_page_hostname` to ClickHouse `sessions_v2` schema

* Start tracking hostname changes in sessions

* Implement hostname filter suggestions

* Enable filtering by `event:hostname`

* Add tests for filtering by hostnames

* Ensure filter suggestions work for exit pages too

* Allow overriding hostnames with `send_pageview` mix task

* Remove `:window_time_on_page` flag

It seems that we can remove it after all?

* Initialize `experimental_hostname_filter` query parameter

* Rewrite cache store behaviour with regards to session hostnames

* Work around inconsistent session merging

So that `populate_stats` can get closer to actual ingestion

* Improve top stats test

* Make it possible to filter sessions by entry/exit hostnames

* Update pages tests

* Expose `experimental_hostname_filtering` temporarily in the UI

* Untested yet: also apply experimental filtering to sources

* Introduce `hostname_filter` feature flag

* Format

* Test top sources with hostname filter + experimental flag
2024-04-04 10:48:30 +02:00
RobertJoonas
bd73fc8266
CH Migration - add more imported metrics and properties (#3949)
* add migration

* add utm_source to imported_sources

* quickfix

* satisfy credo

* Revert "satisfy credo"

This reverts commit bb0b228164.

* Revert "quickfix"

This reverts commit ab6f70c79e.

---------

Co-authored-by: ruslandoga <67764432+ruslandoga@users.noreply.github.com>
2024-04-04 09:12:53 +01:00
hq1
1f778e0c11
CH Migration: exit page hostname on sessions_v2 (#3953)
* CH Migration: exit/entry hostnames in sessions_v2

* Leave only exit_page_hostname, we already record hostnames

* Use ClickHouse DDL in favour of ecto so that cluster is included

* Compress with ZSTD(3)
2024-04-03 09:42:47 +02:00
hq1
2fae0146a4
Reapply 3918 (#3940)
* Reapply "Pages shield (#3918)"

This reverts commit 33b5c10654.

* Make the FF check work against the site actor
2024-03-25 10:36:22 +01:00
hq1
9989ce6927
Migration for 3918 (#3939)
* Revert "Pages shield (#3918)"

This reverts commit 53f94a9f82.

* Migration: Shield page rules
2024-03-25 10:19:50 +01:00
hq1
53f94a9f82
Pages shield (#3918)
* Migration: Shield page rules

* Add Ecto schema for Page Rules

* Add Page Rule cache

* Fix typo

* BTW: Use already imported function

* Extend Shields context interface + split existing tests

* Ingestion: filter matching patches + refactor shield actions

* Add LV section for adding Page Rules

* Validate max page path length

* Put Pages Shield behind a feature flag

* Update CHANGELOG

* Update docs link anchor

As per https://github.com/plausible/docs/pull/477

* Update lib/plausible/shields.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update lib/plausible_web/live/shields/page_rules.ex

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>

* Update lib/plausible_web/live/shields/page_rules.ex

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
2024-03-25 09:48:56 +01:00
Adrian Gruntkowski
52c226c428
Add label column to site_imports schema (#3926) 2024-03-22 10:51:39 +01:00
Adrian Gruntkowski
d6e81670e4
Unify UA and GA4 import flow into one (#3888)
* Unify GA4 and UA import flow into one

* Clean up property and view data retrieval via Google HTTP APIs

* Turn `Map.get` into `Map.fetch!` in API response processing code

* Bump list account summaries page size limit to max of 200

* Show only views in legacy flow and fix legacy redirect after import start

* Move google analytics import actions tests to a separate module

* Extend Google Analytics controller tests

* DRY up `property?` predicate (h/t @RobertJoonas)
2024-03-21 11:37:10 +01:00
ruslandoga
64f35bf8f6
make robots.txt restrictive by default (#3905)
* make robots.txt restrictive by default

* add note about cloud robots.txt
2024-03-19 12:07:30 +01:00
Karl-Aksel Puulmann
26d41ddbb9
Mutation to populate event session columns (#3844)
* WIP mutation to populate event session columns

* Remove duplication

* report errors, allow_nondeterministic_updates

* use right columns

* Update existing columns instead of session_* ones

* Make dialyzer happy

* Fix issue with passing pre-existing params in

* Logger -> IO.puts

* Use IngestRepo.config for connection settings

* Make dictionary options configurable

* Move allow_nondeterministic_mutations to within the migration

* Solve credo warning about too deep nesting

* Missed logger call

* Pattern matching in function head
2024-03-08 09:27:24 +02:00
Karl-Aksel Puulmann
70d21f1b69
Improve compression of event table session columns (#3864) 2024-03-05 12:45:19 +02:00
Karl-Aksel Puulmann
ea38b45685
Ecto migration to move sessions_v2 to VersionedCollapsingMergeTree (#3809)
This migration will noop in staging/production as it already has been run. It also leaves
behind a backup table that initially takes no extra space but will need to be cleaned up
manually
2024-02-28 11:22:27 +02:00