* Enable JSON log formatting
if `LOG_FORMAT=json` is set, the app will start logging
JSON-formatted messages. By the way, the Repo :loggers configration
was removed since it's been already deprecated by Ecto v3
(https://hexdocs.pm/ecto/changelog.html#deprecations-4).
* Update changelog
* Bump deps
* Bump stack
* Fix deprecation warnings
* Fix VCR cassettes mismatch due to OTP-18414
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Format & fix flaky tests
* Handle raw IPv4 hostnames; test public suffix TLD
* Configure locus db cache_dir
So that maxmind unavailability doesn't affect
application startup. PERSISTENT_CACHE_DIR env var is used
to point locus at the GeoIP DB file.
* WIP: Remove ExVCR
* Fix test env config
* Fixup exvcr
* Remove exvcr from deps
* Add convert script
* Remove exvcr cassettes
* Remove convert script
* Rename test
* Update moduledoc
* Update dockerfile
* Bump CI cache
* Tag more slow tests, why not?
* Use charlist for locus cache option
* Pin nodejs
* Merge google tests, make them async
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Add SSO link with signed JWT token
* Falls back to Nolt URL without SSO if token cannot be generated
* Add profile image (gravatar) to Nolt SSO link
* Improve navbar dropdown
* Add 'contact support' link to nav dropdown
* Add CSS rule to prevent horizontal jumps
* Dark mode styling
* Close dropdown when link is clicked
* Clarify links in dropdown
* Clarify CSS comment
* Use Alpine.data() over window
* Rename suggestions_dropdown -> combo-box
* Mix format
* Make logout link look good on dark mode
* Use proxy for gravatar
* Do not use Gravatar proxy in self-hosted
* Changelog
* Add Github Repo link to nav dropdown
* Make dialyzer happy
* Add proxy for Gravatar
* Update assets/css/app.css
Co-authored-by: hq1 <hq@mtod.org>
* Update lib/plausible_web/controllers/avatar_controller.ex
Co-authored-by: hq1 <hq@mtod.org>
* Fix alpine <> Liveview integration
---------
Co-authored-by: hq1 <hq@mtod.org>
* Refactor email verification codes generation to avoid predictability
* Improve `Site.Memberships.any?` slightly
* Update tests
* Fix seeds
* Use `expired?` predicate for checking verification code validity in tests
* Store verification code as string in database to avoid unnecessary int casting
* Reconfigure session cookie
* Allow configure secure cookie (#3277)
* Update config/runtime.exs
Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
* fix runtime conf
* Revert "fix runtime conf"
This reverts commit ff37e479e4.
* Revert "Update config/runtime.exs"
This reverts commit 5d9b310b02.
* Revert "Allow configure secure cookie (#3277)"
This reverts commit 7401a2ad3f.
* Read SECURE_COOKIE with defaults per cloud/selfhost
* Include environment in cookie name
* Remove redundant option
* Format
* s/Map.replace/Map.put
* up
* One more try
* Prevent browser refreshes on socket connection error
We'll keep the log and don't make the page look dumb
even if there's some misconfiguration going on.
---------
Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
This pull request introduces a series of improvements to Plausible.Billing.Plans, including:
* Tag the JSON file with the plan version
* Rename the JSON field limit to monthly_pageview_limit
* Move site_limit function to Billing.Plans
* Refactor subscription_interval, allowance and site_limit functions
* Remove unused AnalyzePlans task
* Dynamically set :domain for the live socket
Ref: https://github.com/phoenixframework/phoenix_live_view/pull/2715
* Make runtime config raise before the user ends up in a reconnect loop
* hall of shame: remove console.info remnant
* Check origin on live websocket
* Get rid of single pipe
* Update formatter config
* Install LiveView JS integration & hooks
* Temporarily update endpoint/session config
* Optionally allow preloading funnels for goals
* Site controller
* Implement funnel settings
lib/plausible_web/live/funnel_settings/combo_box.ex - restored from:
054de6e2 Fix the tab/blur bug again
20da4c89 Rename InputPicker to ComboBox
lib/plausible_web/live/funnel_settings/form.ex - restored from:
9bedda3b Remove potential FIXME
20da4c89 Rename InputPicker to ComboBox
028036ad Review comments
aea4ebc4 Access Funnel min/max steps via the __using__/1 macro
0dde27fd Remove inspect call
eed588a7 Start testing the funnel editor
0e95228b Extract funnel settings test module
7b16ace5 Leverage aplinejs to deal with the tyranny
8dc6a3e7 wip
cf228630 wip
30a43fd1 wip
89f10ecb wip
950a18d9 Dirty funnel save
298a6a53 wip
7690d50f wip
639c6238 fixup
aa59adeb wip
ff75c00b wip
lib/plausible_web/live/funnel_settings/list.ex - restored from:
4eae122c Fix data-confirm attr interpolation
51f0397d Implement deleting funnels
1f6fe25d Add number of steps to funnels list
298a6a53 wip
ff75c00b wip
test/plausible_web/live/funnel_settings/funnel_settings/combo_box_test.exs - restored from:
20da4c89 Rename InputPicker to ComboBox
test/plausible_web/live/funnel_settings/funnel_settings_test.exs - restored from:
34822ff4 Bootstrap InputPicker tests
lib/plausible_web/live/funnel_settings.ex - restored from:
028036ad Review comments
acd9c4f2 Prepare ephemeral funnel definitions so that users can test funnels
51f0397d Implement deleting funnels
0e95228b Extract funnel settings test module
8dc6a3e7 wip
89f10ecb wip
950a18d9 Dirty funnel save
298a6a53 wip
aa59adeb wip
ff75c00b wip
test/plausible_web/controllers/error_report_controller_test.exs - restored from:
34822ff4 Bootstrap InputPicker tests
test/support/html.ex - restored from:
0a53979d Improve InputPicker tests - include AlpineJS assertions
34822ff4 Bootstrap InputPicker tests
lib/plausible_web/views/layout_view.ex - restored from:
b490403b !ifxup
lib/plausible_web/templates/site/settings_funnels.html.eex - restored from:
51f0397d Implement deleting funnels
ea1315f3 Test funnels list in settings
7b16ace5 Leverage aplinejs to deal with the tyranny
ff75c00b wip
4da25c35 Fixup
lib/plausible_web/templates/layout/app.html.eex - restored from:
ff75c00b wip
* Add funnel settings route
* Warn about funnels deletion when deleting goals
lib/plausible_web/templates/site/settings_goals.html.eex - restored from:
fdd9bcd0 Fixup
f1e6364d Merge remote-tracking branch 'origin/master' into funnels-rebase
9d0b7c6d Fix markup error
4a4ddbdc Optionally preload funnels for goals and stub funnel-goal deletion
ebdc4333 Extend the prompt in case of funnel-goal deletion
639c6238 fixup
aa59adeb wip
* Split new JS LiveView additions
* Put funnels behind a feature flag
* Integrate dashboard feature toggle
* Update signing salt for live view
* Update moduledocs
* Update live reloader config
* Use Phoenix.HTML.Safe for goal names
* Workaround to get flashes working in embedded liveview
* Keep feature toggles idempotent, rename property to setting
We'll still retain the ability to flip bools on a lower level.
* Update moduledocs
* Make live flash disappear after 5s
* Tailwind: purge .heex files too
* Update docs link
* Add live components to tailwind purge config
* Update another flaky test
Ref f0bdf872
cc @vinibrsl
* Fix combobox input length w/ WebKit
* Intoduce generic notice component
* Revert "Fix combobox input length w/ WebKit"
This reverts commit 3c653a6d85d5000167631e10ef45a93c13b41ed1.
* Fix combobox input length on webkit
* Make whole combobox item clickable, not only text
* Fix glitch moving Save button on activation
* Tweak dark mode
* Show funnel form without waiting for funnel name input
* Tweak dark mode
* Include static Phoenix components in tailwind purge
* Tune funnels form into a liveview of its own
This is so that ComboBoxes can publish their selections
and unavailable choices can be propagated to other siblings.
* Push less data over websocket
* Undo Lsp/formatter race condition
* Fixup typespecs
* Bust CI cache
* Add revenue fields to ClickHouse events
This commit adds 4 fields to the ClickHouse events_v2 table:
* `revenue_source_amount` and `revenue_source_currency` store revenue in
the original currency sent during ingestion
* `revenue_reporting_amount` and `revenue_reporting_currency` store
revenue in a common currency to perform calculations, and this
currency is defined by the user when setting up the goal
The type of amount fields is `Nullable(Decimal64(3))`. That covers all
fiat currencies and allows us to store huge amounts. Even though
ClickHouse does not suggest using `Nullable`, this is a good use case,
because otherwise additional work would have to be done to
differentiate missing values from real zeroes.
I ran a benchmark with the data pattern we expect in production, where
we have more missing values than real decimals. I created 100 million
records where 90% of decimals are missing. The difference between the
tables in storage is just 0.4Mb.
* Add revenue parameter to Events API
This commit adds support for sending revenue data in ingestion using the
`revenue` parameter - aliased to `$`.
* Add revenue parameter to mix send_pageview
* Add average and total revenue to breakdown queries
* Add revenue goal option to goal creation
This commit adds a currency field to the goals form. Goals that have a
currency set are now revenue goals, and are cached with sites to later
be used during ingestion.
Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
* Enable feature flag in tests
---------
Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
* Simplify Phoenix error template
* Test tracking script is not included in error pages
* Test tracking script is not rendered in error templates
* Rename error layout and remove unnecessary HTML boilerplate
* Add layout setting to errors rendered without exceptions
* Add skip_plausible_tracking option to more pages
* default to v2
* allow N defaults in data migration prompt and custom messages
* join domains lookup
* remove duplicate test runs from ci (both are v2)
* Get rid of PASS_V2_SCHEMA_MIGRATION
* Use in-memory domain lookup + regular table settings
* Remove faulty date arithmetic + prev part calculation
* Set V2_MIGRATION_DONE in Mix.env == :dev
* Mute credo
* Migration (PR: https://github.com/plausible/analytics/pull/2802)
* Implement Site.Domain interface allowing change and expiry
* Fixup seeds so they work with V2_MIGRATION_DONE=1
* Update Sites.Cache so it's capable of multi-keyed lookups
* Implement worker handling domain change expiration
* Implement domain change UI
* Implement transition period for public APIs
* Exclude v2 tests in primary test run
* Update lib/plausible_web/controllers/site_controller.ex
Co-authored-by: Vini Brasil <vini@hey.com>
* Update lib/plausible_web/controllers/site_controller.ex
Co-authored-by: Vini Brasil <vini@hey.com>
* Update moduledoc
* Update changelog
* Remove remnant from previous implementation attempt
* !fixup
* !fixup
* Implement domain change via Sites API
cc @ukutaht
* Update CHANGELOG
* Credo
* !fixup commit missing tests
* Allow continuous domain change within the same site
---------
Co-authored-by: Vini Brasil <vini@hey.com>
* Remove ClickhouseSetup module
This has been an implicit point of contact to many
tests. From now on the goal is for each test to maintain
its own, isolated setup so that no accidental clashes
and implicit assumptions are relied upon.
* Implement v2 schema check
An environment variable V2_MIGRATION_DONE acts like
a feature flag, switching plausible from using old events/sessions
schemas to v2 schemas introduced by NumericIDs migration.
* Run both test suites sequentially
While the code for v1 and v2 schemas must be kept still,
we will from now on run tests against both code paths.
Secondary test run will set V2_MIGRATION_DONE=1 variable,
thus making all `Plausible.v2?()` checks return `true'.
* Remove unused function
This is a remnant from the short period when
we would check for existing events before allowing
creating a new site.
* Update test setups/factories with v2 migration check
* Make GateKeeper return site id along with :allow
* Make Billing module check for v2 schema
* Make ingestion aware of v2 schema
* Disable site transfers for when v2 is live
In a separate changeset we will implement simplified
site transfer for when v2 migration is complete.
The new transfer will only rename the site domain in postgres
and keep track of the original site prior to the transfer
so we keep an ingestion grace period until the customers
redeploy their scripting.
* Make Stats base queries aware of v2 schema switch
* Update breakdown with v2 conditionals
* Update pageview local start with v2 check
* Update current visitoris with v2 check
* Update stats controller with v2 checks
* Update external controller with v2 checks
* Update remaining tests with proper fixtures
* Rewrite redundant assignment
* Remove unused alias
* Mute credo, this is not the right time
* Add test_helper prompt
* Fetch priv dir so it works with a release
* Fetch distinct partitions only
* Don't limit inspect output for partitions
* Ensure SQL is printed to IO
* Remove redundant domain fixture
* Clickhouse migration: add ingest_counters table
* Configure ingest counters per MIX_ENV
* Emit telemetry for ingest events with rich metadata
* Allow building Request.t() with fake now() - for testing purposes
* Use clickhousex branch where session_id is assigned to each connection
* Add helper function for getting site id via cache
* Add Ecto schema for `ingest_counters` table
* Implement metrics buffer
* Implement buffering handler for `Plausible.Ingestion.Event` telemetry
* Implement periodic metrics aggregation
* Update counters docs
* Add toStartOfMinute() to ordering key
* Reset the sync connection state in `after` clause
* Flush counters on app termination
* Use separate Repo with async settings enabled at config level
* Switch to clickhouse_settings repo root config key
* Add AsyncInsertRepo module
* Configure ingest repo access/pool size
If I'm not mistaken 3 is a sane default, the only
inserts we're doing are:
- session buffer dump
- events buffer dump
- GA import dump
And all are serializable within their scopes?
* Add IngestRepo
* Start IngestRepo
* Use IngestRepo for inserts
* Annotate ClickhouseRepo as read_only
So no insert* functions are expanded
* Update moduledoc
* rename alias
* Fix default env var value so it can be casted
* Use IngestRepo for migrations
* Set default ingest pool size from 3 to 5
in case conns are restarting or else...
* Ensure all Repo prometheus metrics are collected
* Move Endpoint errors setup to common config
* Implement naive Sentry link resolver
* Implement error report e-mail
* Delete static sentry script
* Implement user feedback form on server errors
* Re-arrange pipe
* Use Sentry.Config.dsn() where applicable
* Fix typo
* Use Map.replace/3
This PR replaces geolix with locus to simplify self-hosted setup. locus can auto-update maxmind dbs which are recommended for self-hosters if they want city-level geolocation. locus is also a bit faster.
This PR also uses a test mmdb file from https://github.com/maxmind/MaxMind-DB for e2e geolocation tests without stubs.
* Ignore XX and T1 countries
* Add fallback if country_code=nil
* Lookup city overrides directly in CityOverrides module
* Changelog
* Add empty moduledoc
* Remove redundant comment
* Set pg pool size for MIX_ENV=test
* Include slow tests in CI run
* Exclude slow tests by default
* Mark tests slow/async where applicable
* Restructure captcha mocks
* Revert async where env is relied upon
* Add --max-failures=1 to CI run
* Set warnings as errors
* Disable async where various mocks are used
* Revert "Disable async where various mocks are used"
This reverts commit 2446b72a29.
* Disable async for test using vcr
### Changes
This PR adds a fallback to empty build metadata when BUILD_METADATA
contains invalid JSON.
Example `warning` log for `BUILD_METADATA={...}`:
```
20:57:57.872 [warning] failed to parse $BUILD_METADATA, reason: ** (Jason.DecodeError) unexpected byte at position 1: 0x2E (".")
```
Fixes https://github.com/plausible/analytics/issues/2491
### Tests
- [x] This PR does not require tests
### Changelog
- [ ] Entry has been added to changelog
### Documentation
- [x] This change does not need a documentation update
### Dark mode
- [x] This PR does not change the UI
* Update Sites.Cache
So it's now capable of refreshing most recent sites.
Refreshing a single site is no longer wanted.
* Introduce Warmer.RecentlyUpdated
This is Sites Cache warmer that runs only for
most recently updated sites every 30s.
* Validate Request creation early
* Rename RateLimiter to GateKeeper and introduce detailed policies
* Update events API tests - a provisioned site is now required
* Update events ingestion tests
* Make limits visible in CRM Sites index
* Hard-deprecate DOMAIN_BLACKLIST
* Remove unnecessary clause
* Fix typo
* Explicitly delegate Warmer.All
* GateKeeper.allwoance => GateKeeper.check
* Instrument Sites.Cache measurments
* Update send_pageview task to output response headers
* Instrument ingestion pipeline
* Credo
* Make event telemetry test a sync case
* Simplify Request.uri/hostname handling
* Use embedded schema, apply action and rely on get_field
* Implement sites by domain caching interface + warmer
* Add test
* Implement hit rate interface
* Add moduledocs
* Fix up typespec
* s/warmer/warmer_fn
* Extract measure_duration/2
* Fix up typespec
* Log errors and return nil on cache internal errors
* Fix up non-existing cache test
* Retrieve specific db columns when pre-filling the cache
* Reduce the subset of fields retrieved from the DB
See 63f3c6233d (r89871536)
