* Add Ecto schema for imported custom events
* Start importing custom events from GA4
* query imported goals
* make it possible to query events metric from imported
* make it possible to query pageviews in goal breakdown
* make it possible to query conversion rate
* fix rate limiting test
* add CR tests for dashboard API
* implement imported link_url breakdown
* override special custom event names coming from GA4
* allow specific goal filters in imported_q
* update GA4 import tests to use Stats API
* Improve tests slightly
* Update CHANGELOG.md
---------
Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
* Add shield hostname rules migration
* Add hostname rule schema
* Initialize hostname rules cache
* Extend Shields context with hostname related functions
* Instrument ingestion pipeline with hostname rule lookups
* Limit hostname suggestions by shield patterns
* Add LiveView for hostname rules management
* Test hostname cache
* Rename feature flag - should be separate from hostname filter
* Remove :shield_pages feature flag
* Update CHANGELOG
* Format
* Update lib/plausible/shield/hostname_rule.ex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Move tests from `lib/` 🤦
* Use plain `assign` where no short-circuit is necessary
* Fine tune the copy a little bit
* Prevent misplaced tests
* Treat a test with common sense
* Fixup another test that hasn't been really run before
* Make the form hint dynamic depending on rules count
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Use sessionStorage for offer e-mail report banner tracking
Keeping it within the cookie is problematic, as the banners don't
expire and overflow the cookie with data when enough new sites
are added.
Ref https://github.com/plausible/analytics/issues/3762
* Update changelog
* Extract a component
* Make is_dbip evaluate to quoted boolean
* Ignore sessions without entry/exit pages when breaking down entry/exit pages
* Update stats controller tests to have more realistic test data (pageview followed by event)
* Add validation for the events metric in main_graph
* Test the already existing events metric support in main-graph
* Put total conversions on the graph
* extract main_graph_csv function (refactor only)
* add total_conversions and conversion_rate to goal-filtered visitors.csv
* update changelog
* add conversion rate to Stats API timeseries
* make sure CR can be queried as the only metric
* add a test asserting zeros are returned
* add tests for filtering by other properties at the same time
* Remove unnecessary validation of params
1. It doesn't make to validate `interval` (and its granularity) in all
endpoints. It's only relevant for the main graph.
2. The plug (renamed to `date_validation_plug`) already makes sure that
the dates are validated. No need to call the same function again in
Top Stats and Funnel endpoints.
* add metric validation to main graph
* Add tests for main graph API
* put conversion rate on the graph
* update changelog
* Add revenue metrics into metrics.ex
* make fn private
* avoid setting graph metric to visitors in goal-filtered view
* UX improvement: don't autofocus on Custom properties on dashboard
Currently when you visit the dashboard with a certain configuration,
it will automatically scroll to the bottom, which is annoying.
To reproduce:
1. localStorage.clear() in console
2. Refresh, open properties at the very bottom
3. Refresh again
Expected behavior:
1. Dashboard shows
Actual behavior:
1. Dashboard scrolls to the bottom
* Changelog entry
* add metric validation + support in aggregate
* add a test ensuring comparison works
* disallow time_on_page with a goal filter
* Return time_on_page as `nil` from aggregate API
In case time_on_page cannot be calculated, we'll return it as `nil` from
the Stats API.
This is to make the behaviour consistent between breakdown and aggregate
endpoints. As for the UI, we'll still continue to report time_on_page as
0 - not changing any UI behaviour as discussed with Marko.
* add tests for time_on_page in event:page breakdown
* update changelog
* invalidate time_on_page with event:name filter
* add the ability to only query time_on_page in page breakdown
We'll need the visitors metric to get the list of pages to calculate the
time_on_page for.
* wip
* more env, setup user before packages to avoid them picking our uid
* make entrypoint.sh executable by all again
* apparently it's a best practice to have executables be owned by root
* make entrypoint executable in COPY
* stop writing to /app, write to /tmp if needed
* fewer changes
* system user
* keep same style for multiline commands
* fewer changes
* add changelog entry
* fix group assignment for plausible user
* use gid=999
* no home
* no home
* add gecos
* add plausible user to nogroup instead of creating a custom one
* eh
* fewer changes
* fewer changes
* fewer changes
* use PERSISTENT_CACHE_DIR instead of STORAGE_DIR
* ignore more
* cleanup
* remove hex timeout env var
* use ERL_FLAGS=+JMsingle true in public builds
* fallback to /tmp and nest under /tzdata_data/ for tzdata
---------
Co-authored-by: Cenk Kücük <cenk@plausible.io>
This migration will noop in staging/production as it already has been run. It also leaves
behind a backup table that initially takes no extra space but will need to be cleaned up
manually
* Migration: add country rules
* Add CountryRule schema
* Implement CountryRule cache
* Add country rules context interface
* Start country rules cache
* Lookup country rules on ingestion
* Remove :shields feature flag from test helpers
* Add nested sidebar menu for Shields
* Fix typo
* IP Rules: hide description on mobile view
* Prepare SiteController to handle multiple shield types
* Seed some country shield
* Implement LV for country rules
* Remove "YOU" indicator from country rules
* Fix small build
* Format
* Update typespecs
* Make docs link point at /countries
* Fix flash on top of modal for Safari
* Build the rule struct with site_id provided up-front
* Clarify why we're messaging the ComboBox component
* Re-open combobox suggestions after pressing Escape
* Update changelog
* Fix font size in country table cells
* Pass `added_by` via rule add options
* Display site's timezone timestamps in rule tooltips
* Display formatted timestamps in site's timezone
And simplify+test Timezone module; an input timestamp converted
to UTC can never be ambiguous.
* Remove no-op atom
* Display the maximum number of rules when reached
* Improve readability of remove button tests
* Credo
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* improve test
* add os to os_version breakdown
...and add operating_system_versions.csv to the CSV export
* fix conversion rate for os_version breakdown
* update changelog
* fix existing CSV tests
* use case instead of cond
* Revert "Unify percentage change for CR and bounce_rate (#3781)"
This reverts commit a6b1a6ebc7.
* Revert "Bring Stats API up to speed: Add `conversion_rate` to Aggregate and Breakdown (#3739)"
This reverts commit 672d682e95.
* Fix conversion rate change calculation
The change in conversion rate should be calculated similar to bounce rate.
For example, an increase of 25% -> 50% should not be a 100% change, but
a 25% change instead.
* Use the same comparison function in Stats API and dashboard API
This commit fixes a bug where the percentage change reported by the Stats
API is different from the one returned by the internal dashboard API.
* changelog update
* disable event metric with include_imported in every case
* add missing test for metric validation
* refactor metric validation functions
* implement conversion_rate metric validation
* move calculate_cr function into Stats.Util
* Refactor: Move aggregate CR logic into Stats.aggregate
* define atoms to exist
* Ensure that CR does not depend on visitors being queried
If 'visitors' are already queried, we'll use that value. Otherwise we'll
need to make another query to fetch it.
* confirm Stats API aggregate supports CR (tests only)
* small refactor
This is the only 'event_property' left after pattern matching on all
others in the function clauses defined above.
* Make it possible to optionally query conversion_rate
...in breakdown queries (excluding goal and custom prop breakdown)
* A little refactor asking for revenue metrics
1. The `@revenue_metrics` module attribute is an empty list on full build
anyway
2. We don't need to query for revenue metrics if there are no revenue goals
returned in the given query (even if revenue goals exist in site.goals)
3. Revenue metrics are already dropped in prop breakdown without a goal
filter via (get_revenue_tracking_currency/3)
* Make it possible to optionally query conversion_rate (continuation)
... also from a custom prop and goal breakdown
* Frontend adjustments to the Locations report
* Display conversion rate in Regions and Cities (ListReport view)
* Display total conversions, conversions (visitors), and CR in the
"Details" modals of Countries, Regions, and Cities
* Move the percentage into a separate column in the Countries details table
* confirm Stats API breakdown supports conversion_rate (tests only)
* small refactor: extract maybe_add_time_on_page function
* Make it possible to query cr alone
... (without the visitors metric). Already supported in aggregate, this
commit only implements it for the breakdown API.
* Reuse Stats.Util helper functions from b02db88 for aggregate API
We can follow the same logic as with breakdown for manually adding
`visitors` into the metrics list and taking it out of the response
later on.
That way we don't have to make another query, e.g. in a case where
only pageviews and conversion rate is queried. Also keeps things
consistent.
* changelog update
* fix test after resolving merge conflict
* Use explicit string->atom mapping instead of casting
* alias Util module instead of importing it
* use Enum.empty instead of Enum.any
* improve readability
* rename special_metrics to computed_metrics and explain with a comment
* rename visitors_without_event_filters to total_visitors
* keep a single function for removing unwanted metrics
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Support using matches/contains for most filters
* Change behavior where we auto-zoom to specific browser/os/source to only do so if filtering on a single value
* No contains filtering on `location`
* Update CHANGELOG.md
* Fix merge conflict
* Add Ecto.Network dependency
* Migration: Add ip block list table
* If Cachex errors out, mark the cache as not ready
* Add IPRule schema
* Seed IPRules
* Add Shields context module
* Implement IPRuleCache
* Start IPRuleCache
* Drop blocklisted IPs on ingestion
* Cosmetic rename
* Add settings sidebar item
* Consider IPRuleCache readiness on health checks
* Fix typo
* Implement IP blocklist live view
* Update moduledocs
* Extend contextual module tests
* Convert IPRules LiveView into LiveComponent
* Keep live flashes on the tabs view
* Update changelog
* Format
* Credo
* Remove garbage
* Update drop reason typespecs
* Update typespecs for cache keys
* Keep track of who added a rule and when
* Test if adding via LV prefills the updated_by tooltip
* Update ecto_network dependency
* s/updated_by/added_by
* s/drop_blocklist_ip/drop_shield_rule_ip
* Add docs link
* s/Updated/Added
* WIP: PropFilterRow
* Get multi-behavior working
* Render multiple prop filters in one
* Modal reads from query string correctly
* Backend support for multiple custom property filters
* Add backend tests for multiple custom property filters
* Disable already selected options in property keys
We can't allow choosing the same property multiple times without changing the request
params, which we decided against
* Allow choosing any property under Behaviors > Custom props even if custom prop filter applied
This was a limitation (I believe) introduced by using ARRAY JOINs to query custom properties
* CHANGELOG.md
* Solve credo warning about too deep nesting
* Update assets/js/dashboard/stats/modals/prop-filter-modal.js
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
* Refactor internal function for clarity
* Add another step -> Add another
* Solve 500 error
* Separate boxes per property filter
* Retain other filters in props table
* removeFilter behavior for props
* matches_member support for custom props
* filter_suggestions for prop keys should account for prop filter
* find over filter
* refactor appliedFilters
* FILTER_TYPES => FILTER_OPERATIONS
* Make add another link not wrap the whole page
* Unique keys
---------
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
* add tests for filtering by goal in timeseries and aggregate
* refactor filter parsing
* stop returning custom props in event:goal breakdown
* test breaking down wildcard pageview goals
* extract filter utils
* parse more goal filter options
* add passing tests for new filter types
* do not allow querying session metrics with a goal filter
* remove unused page_match property
* test that non-configured goals are not returned in breakdown
* enforce filtered goals configured
* update changelog
* Allow simple filtering by revenue goals
This does not mean that revenue metrics are supported. If a revenue goal
is filtered by, we treat it like a simple custom event goal in the API.
* use List.wrap
* Allow `matches` operator to work in BE for custom props
Note: No FE support yet, needs further testing
* feat: allow choosing `contains` for property filters in the UI
* no autocomplete on prop values if `contains` for consistency
* CHANGELOG.md
* Fix: Handle (none) property in property breakdowns when using matching
When matching we should always exclude (none)
* remove unused code
* add e.name == "pageview" condition to pageview goals
This fixes the weird behavior where filtering by a pageview goal would
also return custom events and vice versa.
* update changelog
* Add 2FA actions to `AuthController`
* Hook up new `AuthController` actions to router
* Add `qr_code` to project dependencies
* Implement generic `qr_code` component rendering SVG QR code from text
* Implement enabled and disabled 2FA setting state in user settings view
* Implement view for initiating 2FA setup
* Implement view for verifying 2FA setup
* Implement view for rendering generated 2FA recovery codes
* Implement view for verifying 2FA code
* Implement view for verifying 2FA recovery code
* Improve `input_with_clipboard` component
* Improve view for initiating 2FA setup
* Improve verify 2FA setup view
* Implement `verify_2fa_input` component
* Improve view for verifying 2FA setup
* Improve view rendering generated 2FA recovery codes
* Use `verify_2fa_input` component in verify 2FA view
* Do not render PA contact on self-hosted instances
* Improve flash message phrasing on generated recovery codes
* Add byline with a warning to disable 2FA modal
* Extract modal to component and move 2FA components to dedicated module
* First pass on loading state for "generate new codes"
* Adjust modal button logic
* Fix button in verify_2fa_input component
* Use button component in activate view
* Implement wait states for recovery code related actions properly
* Apply rate limiting to 2FA verification
* Log failed 2FA code input attempts
* Add ability to trust device and skip 2FA for 30 days
* Improve styling in dark mode
* Fix waiting state under Chrome and Safari
* Delete trust cookie when disabling 2FA
* Put 2FA behind a feature flag
* Extract 2FA cookie deletion
* ff fixup
* Improve session management during 2FA login
* Extract part of 2FA controller logic to a separate module and clean up a bit
* Clear 2FA user session when rate limit hit
* Add id to form in verify 2FA setup view
* Add controller tests for 2FA actions and login action
* Update CHANGELOG.md
* Use `full_build?()` instead of `@is_selfhost` removed after rebase
* Update `Auth.TOTP` moduledoc
* Add TOTP token management and make `TOTP.enable` more test-friendly
* Use TOTP token for device trust feature
* Use zero-deps `eqrcode` instead of deps-heavy `qr_code`
* Improve flash messages copy
Co-authored-by: hq1 <hq@mtod.org>
* Make one more copy improvement
Co-authored-by: hq1 <hq@mtod.org>
* Fix copy in remaining spots
* Change redirect after login to accept URLs from #3560 (h/t @aerosol)
* Add tests checking handling login_dest on login and 2FA verification
* Fix regression in email activation form submit button behavior
* Rename `PlausibleWeb.TwoFactor` -> `PlausibleWeb.TwoFactor.Session`
* Move `qr_code` component under `Components.TwoFactor`
* Set domain and secure options for new cookies
---------
Co-authored-by: hq1 <hq@mtod.org>
* Avoid redirect in site settings
* Fix unicode in SiteController existing tests
* Fix various tests
* Add CHANGELOG
* Make sure test site is example.com
* Use Route helpers in site_controller
* Fix UTF redirect in change domain submit action
* Fix UTF site domain in reset stats action
* Enable JSON log formatting
if `LOG_FORMAT=json` is set, the app will start logging
JSON-formatted messages. By the way, the Repo :loggers configration
was removed since it's been already deprecated by Ecto v3
(https://hexdocs.pm/ecto/changelog.html#deprecations-4).
* Update changelog
* Revert "Remove site pins for now"
This reverts commit 5eccf4eaf6.
* Implement basic site pin schema level logic within user specific preferences
* Add vertical ellipsis menu markup
* Implement basic changesets for user preferences
* Implement pin toggling
* Try to fix pin sorting
* Implement pin toggling in LV
* Adjust moduledocs for new schema(s)
* Remove unnecessary `distinct` from query
* Use `button` for pin/unpin action
* Generalize preference setting
* Rename schema and fields for clarity
* Rename `list_type` -> `entry_type`
* Safeguard setting options
* Test `set_option/4` and `toggle_pin/2`
* Add test for listing pinned sites via `Sites.list`
* Disallow pinning sites outside page explicitly
* Test pinning in LV
* Test conditional rendering of site settings in /sites
* Remove unnecessary TODO comment
* Safeguard `Sites.set_option/4` against invalid user/site combo
* Handle pinned sites in dashboard site picker
* Clear flashes upon (un)pinning sites
* Update CHANGELOG
* Prevent blinking of hamburger menu items on first paint
* Highlight hamburger handle on hover in /sites
* Start showing hotkeys in site picker again
* Sort pinned sites in the order they were pinned
* Update sites list order immediately after pin/unpin toggle
* Refactor and split `Sites.list/3`, extracting `Sites.list_with_invitations/3`
* Cap number of pinned sites at 9 per user
* First pass on visual indication of site cards (dis)appearing
* Apply ellipsis gradient+shadow on card hover
* Fix responsive padding of site cards
* Sort by invitations first, pinned sites second and then the rest
* Revert "Apply ellipsis gradient+shadow on card hover"
This reverts commit 0608796612639030ccbb12df639709f78edc1434.
* Apply more subtle hover effect on the ellipsis menu
* Make error and success flash LV boxes use separate component containers
* Promote `pinned_at` in table migration to a column
* Switch logic to using `pinned_at` as a standard schema field
* Refactor `Sites.list*` getting rid of subquery (h/t @ukutaht)
* Remove migration which is already merged upstream
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Add SSO link with signed JWT token
* Falls back to Nolt URL without SSO if token cannot be generated
* Add profile image (gravatar) to Nolt SSO link
* Improve navbar dropdown
* Add 'contact support' link to nav dropdown
* Add CSS rule to prevent horizontal jumps
* Dark mode styling
* Close dropdown when link is clicked
* Clarify links in dropdown
* Clarify CSS comment
* Use Alpine.data() over window
* Rename suggestions_dropdown -> combo-box
* Mix format
* Make logout link look good on dark mode
* Use proxy for gravatar
* Do not use Gravatar proxy in self-hosted
* Changelog
* Add Github Repo link to nav dropdown
* Make dialyzer happy
* Add proxy for Gravatar
* Update assets/css/app.css
Co-authored-by: hq1 <hq@mtod.org>
* Update lib/plausible_web/controllers/avatar_controller.ex
Co-authored-by: hq1 <hq@mtod.org>
* Fix alpine <> Liveview integration
---------
Co-authored-by: hq1 <hq@mtod.org>
* Implement PoC for email reverification flow on update
* Improve user settings form and email change validation
* Expose `previous_email` in Kaffy CRM
* Improve plugs setup and remove dead action from AuthController
* Fix seeds
* Extract predicate query functions from AuthController
* Add tests
* Update CHANGELOG.md
* Rename `has_any_sites?` to `Memberships.any?` and `has_any_memberships?`
* Improve flash message on cancelling email change
* Cover one more test case for email update
* Add zxcvbn dependency
* Change password length range requirement from 6-64 to 12-128
* Reimplement register form in LV
* Implement server-side check for password strength
* Add rudimentary strength meter
* Make password input with strength a separate component and improve it
* Fix existing tests to provide strong enough password
* Apply formatting
* Replace existing registration form with new one
* Hide built-in label in `.input` component when none provided
* Crop password to first 32 chars for analysis by zxcvbn
* Add tests for new form components
* Integrate hCaptcha into LV
* Fix existing AuthController tests
* Add tests for Live.RegisterForm
* Hide strength meter when password input is empty
* Randomize client IP in headers during tests to avoid hitting rate limit
* Apply auxilliary formatting fixes to AuthController
* Integrate registration from invitation into LV registration logic
* Fix existing password set and reset forms
* Make `password_length_hint` component more customizable
* Optimize `Auth.User.set_password/2`
* Remove unnecessary attribute from registration form
* Move password set and reset forms to LV
* Add tests for SetPasswordForm LV component
* Add tests for password checks in `Auth.User`
* Document code a bit
* Implement simpler approach to hCaptcha integration
* Update CHANGELOG.md
* Improve consistency of color scheme
* Introduce debounce across all text inputs in registration and password forms
* Fix email input background in register form
* Ensure only single error is rendered for empty password confirmation case
* Remove `/password` form entirely in favor of preferred password reset
* Remove unnecessary `router` option from `live_render` calls
* Make expensive assigns in LV with `assign_new` (h/t @aerosol)
* Accept passwords longer than 32 bytes uniformly as very strong
* Avoid displaying blank error side by side with weak password error
* Make register actions handle errors gracefully
* Render only a single piece of feedback to reduce noise
* Make register and password reset forms pw manager friendly (h/t @cnkk)
* Move registration forms to live routes
* Delete no longer used deadviews
* Adjust registration form in accordance to changes in #3290
* Reintroduce dogfood page path for invitation form from #3290
* Use alternative approach to submitting plausible metrics from LV form
* Rename metrics events and extend tests to account for them
