* Update applications
* Clone community config
* Move modules to experimental dir
* Update runtime config
* Apply first set of compile-time conditionals
* Move funnel schemas to experimental
* Make funnel schema-less build compile
* Use experimental/lib for elixir code
* Move JS funnels to experimental
* Clean up conditional rendering
* Tidy up the pipeline
* Make two builds pass tests without warnings
* Reuse existing dotenvs
* Do a bunch of renames
* Clean up naming
* Run secondary CI
* Update router
* Remove RewriteFunnelDupes migration
Tests were disabled already and it was a one-off shot
* Fixup quota mixins
* Add moduledoc
* Change MIX_ENV for seconary test run
* Skip crm on small
* !fixup
* Exclude flags pipeline
* Update lib/plausible_web/controllers/stats_controller.ex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* use windows in 'time on page' query
* add imported time on page back
* join imported_pages instead of extra query
* don't forget aggregated time on page!
* eh?
* no need for separate window clause
* use dynamic_filter_condition in aggregate_time_on_page
* it's avg, not total
* fix dynamic_filter_condition call
* tests pass
* fewer changes
* fewer changes + feature flag
* base quickfix
* fewer changes
* fewer changes?
* fewer changes!
* replace coalesce with if(empty(),etc)
* make window_aggregate_time_on_page return same result as neighbor_
* breakdown as well
* add failing test
* more info
* more info
* format example
* add failing aggregate.ex test
* fix breakdown.ex windowing time_on_page calculation
* fix aggregate.ex windowing time_on_page calculation
* Revert "Remove site pins for now"
This reverts commit 5eccf4eaf6.
* Implement basic site pin schema level logic within user specific preferences
* Add vertical ellipsis menu markup
* Implement basic changesets for user preferences
* Implement pin toggling
* Try to fix pin sorting
* Implement pin toggling in LV
* Adjust moduledocs for new schema(s)
* Remove unnecessary `distinct` from query
* Use `button` for pin/unpin action
* Generalize preference setting
* Rename schema and fields for clarity
* Rename `list_type` -> `entry_type`
* Safeguard setting options
* Test `set_option/4` and `toggle_pin/2`
* Add test for listing pinned sites via `Sites.list`
* Disallow pinning sites outside page explicitly
* Test pinning in LV
* Test conditional rendering of site settings in /sites
* Remove unnecessary TODO comment
* Safeguard `Sites.set_option/4` against invalid user/site combo
* Handle pinned sites in dashboard site picker
* Clear flashes upon (un)pinning sites
* Update CHANGELOG
* Prevent blinking of hamburger menu items on first paint
* Highlight hamburger handle on hover in /sites
* Start showing hotkeys in site picker again
* Sort pinned sites in the order they were pinned
* Update sites list order immediately after pin/unpin toggle
* Refactor and split `Sites.list/3`, extracting `Sites.list_with_invitations/3`
* Cap number of pinned sites at 9 per user
* First pass on visual indication of site cards (dis)appearing
* Apply ellipsis gradient+shadow on card hover
* Fix responsive padding of site cards
* Sort by invitations first, pinned sites second and then the rest
* Revert "Apply ellipsis gradient+shadow on card hover"
This reverts commit 0608796612639030ccbb12df639709f78edc1434.
* Apply more subtle hover effect on the ellipsis menu
* Make error and success flash LV boxes use separate component containers
* Promote `pinned_at` in table migration to a column
* Switch logic to using `pinned_at` as a standard schema field
* Refactor `Sites.list*` getting rid of subquery (h/t @ukutaht)
* Remove migration which is already merged upstream
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Change limits for trials
* Keep legacy trial limits for users that registered before the business tier
* Change private preview notice for release
* Run formatter
* Add countdown to private preview notice
* Bump deps
* Bump stack
* Fix deprecation warnings
* Fix VCR cassettes mismatch due to OTP-18414
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Format & fix flaky tests
* Handle raw IPv4 hostnames; test public suffix TLD
* Configure locus db cache_dir
So that maxmind unavailability doesn't affect
application startup. PERSISTENT_CACHE_DIR env var is used
to point locus at the GeoIP DB file.
* WIP: Remove ExVCR
* Fix test env config
* Fixup exvcr
* Remove exvcr from deps
* Add convert script
* Remove exvcr cassettes
* Remove convert script
* Rename test
* Update moduledoc
* Update dockerfile
* Bump CI cache
* Tag more slow tests, why not?
* Use charlist for locus cache option
* Pin nodejs
* Merge google tests, make them async
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Read feature status from Billing.Feature instead of %Site{}
This commit changes data attributes passed to React. Previously the
controller read feature statuses directly from the %Site{} schema. The
Billing.Feature context is aware of the user plan and the features
available.
* Limit funnels internal API based on site owner plan
* Limit props internal API based on site owner plan
* Use site factory in QueryTest
* Limit custom property filter based on site owner plan
* Limit revenue goals queries based on site owner plan
* Include ApiKey functions in Auth context
* Make feature notice work without %Site{}
Previously the extra feature notice required a %Site{} in order to check
the owner plan. However, not every feature is scoped by site, for
example the Stats API. For features like this, a %User{} is required,
and not a %Site{}.
This commit replaces the `:site` param with `:billable_user`, which is
common to both site and user-scoped features.
* Add stats_api to the list of extra features
* Limit API Key creation based on user plan
This pull request implements limits to funnels, revenue goals and custom props based on the site owner plan. It extends the current "premium feature" notice to account for the new plans, trials and the on-going private preview. Stats API is not in the context of this pull request, but will be implemented likewise.
* Make membership creation and role updates more explicit in terms of changesets
* Extract invitation accept flow logic and refactor it slightly
* Improve acceptation logic
* Update moduledoc
* Improve SiteLocker API and add typespecs
* Stop naming function not returning a boolean like a predicate
* Refactor rest of invitation actions and safeguard against rogue requests
* Update code docs slightly
* Extend `Billing.check_needs_update/1` tests
* Parametrize selfhost flag and toggle SiteLocker logic on it
* Add tests for newly extracted services
* Add test case and a fix for locking site on grace period ended
* Make invitation controller tests async as there's no more env patching
* Add test cases for self-invites and fix one bug
* Add and refactor tests for rejecting and removing invitations
* Prevent issuing ownership transfer to existing owner
* Improve name of the test
* Improve `Billing.check_needs_to_upgrade/1` return value
* Improve `Billing.SiteLocker.update_sites_for/1` and its tests
* Fix typos
Co-authored-by: hq1 <hq@mtod.org>
* Make invitation removal and rejection resilient to races
---------
Co-authored-by: hq1 <hq@mtod.org>
* Add zxcvbn dependency
* Change password length range requirement from 6-64 to 12-128
* Reimplement register form in LV
* Implement server-side check for password strength
* Add rudimentary strength meter
* Make password input with strength a separate component and improve it
* Fix existing tests to provide strong enough password
* Apply formatting
* Replace existing registration form with new one
* Hide built-in label in `.input` component when none provided
* Crop password to first 32 chars for analysis by zxcvbn
* Add tests for new form components
* Integrate hCaptcha into LV
* Fix existing AuthController tests
* Add tests for Live.RegisterForm
* Hide strength meter when password input is empty
* Randomize client IP in headers during tests to avoid hitting rate limit
* Apply auxilliary formatting fixes to AuthController
* Integrate registration from invitation into LV registration logic
* Fix existing password set and reset forms
* Make `password_length_hint` component more customizable
* Optimize `Auth.User.set_password/2`
* Remove unnecessary attribute from registration form
* Move password set and reset forms to LV
* Add tests for SetPasswordForm LV component
* Add tests for password checks in `Auth.User`
* Document code a bit
* Implement simpler approach to hCaptcha integration
* Update CHANGELOG.md
* Improve consistency of color scheme
* Introduce debounce across all text inputs in registration and password forms
* Fix email input background in register form
* Ensure only single error is rendered for empty password confirmation case
* Remove `/password` form entirely in favor of preferred password reset
* Remove unnecessary `router` option from `live_render` calls
* Make expensive assigns in LV with `assign_new` (h/t @aerosol)
* Accept passwords longer than 32 bytes uniformly as very strong
* Avoid displaying blank error side by side with weak password error
* Make register actions handle errors gracefully
* Render only a single piece of feedback to reduce noise
* Make register and password reset forms pw manager friendly (h/t @cnkk)
* Move registration forms to live routes
* Delete no longer used deadviews
* Adjust registration form in accordance to changes in #3290
* Reintroduce dogfood page path for invitation form from #3290
* Use alternative approach to submitting plausible metrics from LV form
* Rename metrics events and extend tests to account for them
* Filter out empty entries when listing stats for UTM props
* Update test fixtures removing noref entries in UTM CSV stat exports
* Update external API tests to account for lack of noref records for UTM stats
* Filter out entries with empty UTM props from imported GA stats
* Remove unreachable GA utm_source dim clause in imported stats logic
This pull request introduces a series of improvements to Plausible.Billing.Plans, including:
* Tag the JSON file with the plan version
* Rename the JSON field limit to monthly_pageview_limit
* Move site_limit function to Billing.Plans
* Refactor subscription_interval, allowance and site_limit functions
* Remove unused AnalyzePlans task
* Remove `Plausible.Sites.set_allowed_event_props/2` function
This commit removes the `Plausible.Sites.set_allowed_event_props/2`
function in favor of `Plausible.Props.allow/2`.
* Allow internal prop keys in queries
This commit allows internal prop keys by default in queries. I decided
not to include internal keys in the database, because they might change
and we'd need to migrate it again.
* remove redundant 'props.' from behaviours/index.js
---------
Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
* extract add_exit_rate function
* Change internal API metric names for /entry-pages & /exit-pages
* `unique_entrances` -> `visitors`,
* `total_entrances` -> `visits`,
* `unique_exits` -> `visitors`,
* `total_exits` -> `visits`,
This is just a consistency improvement - the `visitors` metric always means
one thing and there's no need to call it different for entry pages internally.
This commit does not change any noticable behavior. The UI labels are kept
the same and the column headers in the CSV export will also remain the same.
* Change internal API metric names for /conversions
* `unique_conversions` -> `visitors`,
* `total_conversions` -> `events`,
* return revenue metrics from /custom-prop-values (backend)
* be more explicit about which metric is plotted with Bar
* validate that ListReport input metrics actually exist in the API response
* display revenue metrics in the dashboard Properties section
* limit the number of columns shown on mobile
* add revenue metrics to Properties > Details
* review suggestions
* define hiddenOnMobile per metric instead of keeping the first 3
* rewrite if-else block
---------
Co-authored-by: Vini Brasil <vini@hey.com>
* extract add_exit_rate function
* Change internal API metric names for /entry-pages & /exit-pages
* `unique_entrances` -> `visitors`,
* `total_entrances` -> `visits`,
* `unique_exits` -> `visitors`,
* `total_exits` -> `visits`,
This is just a consistency improvement - the `visitors` metric always means
one thing and there's no need to call it different for entry pages internally.
This commit does not change any noticable behavior. The UI labels are kept
the same and the column headers in the CSV export will also remain the same.
* Change internal API metric names for /conversions
* `unique_conversions` -> `visitors`,
* `total_conversions` -> `events`,
* make (none) value in custom prop breakdown add +1 to pagination limit
This is needed because the (none) value is always added to the
breakdown_results **after** fetching those from ClickHouse.
* only include (none) values on the first page of results
* fix percentage metric calculation for paginated results
Instead of summing up the number of visitors from the breakdown results
to get the total, we have to make a separate query to `Stats.aggregate`.
Otherwise, the percentages for each results page will wrongly add up to 100%.
Since imported data for aggregated visitors and other properties (such as
browsers, OSs, etc) live in different tables, we have to tweak the tests to
also include the same number of visitors in the `imported_visitors` table.
* add details view for props
* changelog
* exclude imported data from total
* move set_allowed_event_props into context module
* extract parse_csv fn
* add custom_props.csv to CSV export (under FF)
* add test for custom prop export with a prop filter
* add test for goal filtered scenario
* mix format
* fix flaky test
* extract breakdown_custom_prop_values function
* Revert "fix flaky test"
This reverts commit 256b9bb2e7. The flaky
test was already fixed in 86557b1878
* make set_allowed_event_props a bang function
* test that all files are included in the CSV export
* refactor SourceList to use ListReport
* refactor SourceList into a fn comp
* change referrer-drilldown API response format and remove dead code
* use ListReport in referrer-list
* fix CI
* fix flaky test
* remove IO.inspect
* copy prop_breakdown tests into a separate file
* add a new endpoint for custom props
Duplicate existing goal prop breakdown as a building base for the
new endpoint
* stick to original metric names + CR definitions
We currently use CR with two different definitions, which is inconsistent
and should be changed. This commit just documents the difference for the
time being.
* basic prop breakdown without goal filter
* increase % metric precision to one decimal place
* add some tests without goal filter
* silence credo for TODO comment
* use events metric instead of pageviews
* review feedback
* inline add_cr_a instead
* Move money cast function to Stats.Util
* Add revenue metrics to main graph API
* Plot revenue metrics on the main graph
* Return plain numbers instead of a map
* Fix Credo issues
* Fix canMetricBeGraphed function
* Revert canMetricBeGraphed function changes
* Add noindex,nofollow to dashboard pages
* Implement NoRobots plug
* Enable NoRobots plug in the router
* Fixup internal route
* Fix double slash in the router
* Add special bot treatment to plausible.io live demo page
* Revert aggressive protection with agent detection
* PoC: evaluate funnel on step selection
* Adjust test to actually _select_ steps
* Throttle and test ephemeral funnel evaluation
* Format compact percentages server-side
At this point it's pointless to duplicate this;
all the primitives exist in the API in case the client
wants a different representation.
* Add Funnel react component
assets/js/dashboard/stats/behaviours/funnel.js - restored from:
98a76cbd Remove console.info calls
d94db99d Convert Funnel class component into a functional one
028036ad Review comments
3067a940 Stop doing maths in react
73407cc3 Fix error handling when local storage gets corrupted
e8c6fc52 Format numbers on funnel labels
c815709f Reorganize component responsibility
7a88fe44 Outline basic error handling
94caed7c Chart styling updates
4514608a Add percentages to funnel
d622c32d Add funnel picker
Co-authored-by: Uku Taht <uku.taht@gmail.com>
* Pass funnels list to react via data-funnels
* Implement Funnels react API
lib/plausible_web/controllers/api/stats_controller.ex - restored from:
f36ad234 Adjust to Plausible.Stats interface
9b532273 Test funnel stats controller
028036ad Review comments
bea3725f Remove IO.inspect
7a88fe44 Outline basic error handling
c8ae3eaf Move Funnels to StatsController and use base query
667cf222 Put private functions at the bottom
* Tweak funnel presentation
* Handle errors at the top
* Do not register DataLabels plugin globally
or else all the existing charts are affected
* Calculate drop-off percentage evaluating funnels
* Tweak dark mode + implement nicer tooltips
* Make currently selected funnel bold in the picker
* Count user_ids not session_ids when evaluating funnels
So if a visitor goes:
1. Start session
2. Complete funnel step 1
3. Inactive for 30 minutes
4. Complete funnel step 2
We would not be able to track this funnel completion because of the session timeout.
We like to o measure this as funnel completion even though the session expired in the middle.
cc @ukutaht
* Add extra properties to the funnels API
cc @ukutaht
* Improve tooltips so that step to data is rendered
* Change tooltip number formatting
* Remove debugging remnants
* Quick & dirty mobile view
* Fix mobile view: tweak dark mode & funnel switching
* Ignore DOMException: the operation was aborted
Otherwise this sometimes flashes the space shuttle
screen when navigating quickly via a keyboard.
* Format percentages on the main chart
* Close missing tag 🙈
* Revert "Close missing tag 🙈"
This reverts commit 9c2f970e22fd7e2980503242b414f42ce8bce1d2.
* Use jsx to render funnel tooltip
To get markup validated via lsp mostly...
* Fixup: s/class/className
* Fix className interpolation
* Add a ruler to the tooltip
* Tweak funnel chart style
* Fix font distortion issue on chart/canvas labels
* s/class/className
* Put "Set up funnels" link behind a feature flag
* Refactor internal selection storage
Getting ready for live funnel evaluation
* Don't try to connect LV socket if there's no CRSF token set up
This is perfectly okay for some of the templates/layouts.
* Fix up funnel creation typespecs
Unfortunately we can't define a type with literal string keys,
hence this must suffice.
* Use uniq over count/distinct
* Revert JSX in tooltips
Ref: https://github.com/plausible/analytics/pull/3066#discussion_r1241891155
* Remove the extra query for counting all visitors
cc @ukutaht
* Add premium notice
---------
Co-authored-by: Uku Taht <uku.taht@gmail.com>
This commit adds revenue data to top stats. Average and total are displayed when filtering by a revenue goal (or many if they have the same currency set).
This commit fixes a bug where performing a breakdown by goal with
revenue metrics would return data for both event goals and pageview
goals. However, revenue data is only applicable for event goals as they
hold currency data.
To fix the bug, this commit removes revenue metrics from the pageview
breakdown query.
This commit fixes a bug where timeseries queries - such as the main
graph - would fail with imported data in realtime mode. Realtime mode
`date` field is not actually a date, but minutes from now. This would
cause the imported tables join to fail with this error:
```
(Ch.Error Code: 53. DB::Exception: Can't infer common type for joined
columns: date: Int64 at left, s1.date: Date at right. There is no
supertype for types Int64, Date because some of them are
Date/Date32/DateTime/DateTime64 and some of them are not.
(TYPE_MISMATCH)
```
This commit removes imported data from realtime queries, as it doesn't
make sense to include it. Imported data does not have time precision,
and would only appear in the first day the data was imported anyways.
* copy relevant files from b2ace16540
* make it work and set site.funnels to empty list
* make Behaviours a functional component
* add UI for a setup hint and always display conversions by default
* cherry-pick migration commit
* update site schema with new fields
* backend: implement disable-feature action
* switch between tabs in the behaviors section
Introduces template components to build props and funnels on. Both
only show a setup notice atm, and both are behind feature flags.
* extend API for disabling props and funnels
* render feature setup note directly from the Behaviours component
* fix UI behavior when features are hidden
* update setup notices
* add conversions feature switch to Site Settings > Goals
* mix format
* remove IO.inspect
* change setup notice - use buttons + popup confirmation
* optimize for light mode
* restrict access to setup notices
* some styling improvements
* allow super-admins to enable/disable features
* only show conversions (last 30min) in realtime mode
* use shorter display names for tabs
* optimize for mobile screens
* note about sending custom events
* changelog update + fix CI
* change HTTP verb for the disable-feature action
* change UI label for show/hide goals
* Add revenue fields to ClickHouse events
This commit adds 4 fields to the ClickHouse events_v2 table:
* `revenue_source_amount` and `revenue_source_currency` store revenue in
the original currency sent during ingestion
* `revenue_reporting_amount` and `revenue_reporting_currency` store
revenue in a common currency to perform calculations, and this
currency is defined by the user when setting up the goal
The type of amount fields is `Nullable(Decimal64(3))`. That covers all
fiat currencies and allows us to store huge amounts. Even though
ClickHouse does not suggest using `Nullable`, this is a good use case,
because otherwise additional work would have to be done to
differentiate missing values from real zeroes.
I ran a benchmark with the data pattern we expect in production, where
we have more missing values than real decimals. I created 100 million
records where 90% of decimals are missing. The difference between the
tables in storage is just 0.4Mb.
* Add revenue parameter to Events API
This commit adds support for sending revenue data in ingestion using the
`revenue` parameter - aliased to `$`.
* Add revenue parameter to mix send_pageview
* Add average and total revenue to breakdown queries
* default to source_query.include_imported before true
This fixes a bug where a Stats API aggregate query was trying
to query current period *without imported data* but previous
period *with imported data*.
* remove __internal_visits at a better time
As soon as we have made the db query, we don't need it anymore
* disallow events metric with imported data
* update changelog
* keep default in a better place
* remove unused option
* update test description
Co-authored-by: Vini Brasil <vini@hey.com>
---------
Co-authored-by: Vini Brasil <vini@hey.com>
* add migration
* add not null constraint
* remove not null constraint and default value
* add site schema change
* filter garbage props for prop_key suggestions
* filter garbage props returned by StatsController.conversions/2
* add crm action to set prop allowlist
* use nullable allowed_event_props field instead
* filter garbage props in db query instead
* implement suggestion dealing with string input
* refactor the allowlist condition in db query
* improve test
* Reject invalid country codes early
* Validate all date parameters early
* Ensure request JSON parses to a map
* Re-arrange function calls
* Extend malicious filter validation
* add entry page filter condition to bounce rate
* Fix select_merge with dynamic
* fix tests and add one test
* generalize page_filter_condition function
* use dynamic_filter_condition for session filters too
* disable views_per_visit with a page filter
* update changelog
* disable credo for complex function
---------
Co-authored-by: Uku Taht <uku.taht@gmail.com>
* Add PropFilterModal (only UI)
* small variable refactor
* allow selecting prop value filter type
* allow selecting only one prop_key
* allow selecting many prop_values only when prop_key selected
* handle submitting filter
* get applied filters from query + remove option
* change prop filter label format
* support member and not_member filter types for pageview props
* show (none) value in filter suggestions
* refactor zip_results/4 and remove unused code
* fix displaying (none) values in goals section prop breakdown
* remove unnecessary functionality
* fix bug: returning prop names for goal :member filter
* fix bug: submitting regular filter modal with Enter key
* bugfix: disallow opening prop filter modal when feature flag disabled
* mix format
* break selected combobox values into multiple lines
* fix useEffect behavior for focusing on prop_key field
* support submitting prop filter with Enter key
* refactor getFormState in PropFilterModal
* separate fetchPropKey and fetchPropValue functions
* Allow querying props for pageview goals
* Make the internal props API only return a list of props (not map)
* Separate function for fetching all props in Stats API goal breakdown (this returns a map as before)
* ditch state for keeping search bar visible
* group by event_name in db query
This pull request removes the comparisons feature flag, making it visible to all users. It also removes percentages from top stats in default view, leaving those for the comparison view only.
This commit fixes a bug where comparisons would not work with imported data. This is because comparisons copies the source query and modifies the dates only, where it should reevaluate if there is imported data for the new comparison date range.
Closes#2870
Co-authored-by: Adam <hq@mtod.org>
