* Implement FF-driven DB lookup for sites during ingestion
We like to see the impact of doing a simple postgres lookup on each
ingestion event. The percentage-based feature flag `:ingestion_pg_lookup`
must be set in order for lookups to be executed.
* Fix resolving Cachex stats metrics
* Enable PromEx on dev env
This commit removes the current 1% sampling on ingestion to completely
ignore ingestion traces. We are not getting enough value from those, and
we'll keep dashboard, API and background jobs traces only.
Someone mentioned there's no way to cancel from the link we include in this email (https://plausible.io/billing/upgrade) so I'm changing it to the settings page instead https://plausible.io/settings. My own https://plausible.io/billing/upgrade is a bit different than normal subscribers see it so I don't actually know if there's a way to cancel on it. If there is a way there too, feel free to ignore this
People are likely to enter (copy/paste) goals from external sources
which can lead to whitespace characters appended by accident.
That renders the goal unusable and hard to distinct visually.
Normally to fix up existing goals we would use a data migration,
but this should be good enough to check if the problem
with never appearing goals resurfaces.
This commit fixes a bug where Google Analytics import tokens were not
being refreshed properly because the function was not returning the
expected tuple. Thanks to @aerosol we can nicely test this now.
This commit fixes a bug where users clearing one import and trying to
re-import would get invalid import date ranges. This was caused because
the stats_start_date field was not updated to nil after clearing
imported stats, as this field defines the end date of the import.
* Make TestUtils module available in all tests
* Add macros patching the application env in tests
Unfortunately a lot of existing functionality relies on
certain application env setup. This isn't ideal because
the app config is a shared state that prevents us from
running the tests in parallel.
Those macros encapsulate setting up new env for test purposes
and make sure the changes are reverted when the test finishes.
* Allow passing request opts to HTTPClient.post/4
We need this to swap custom request building in
Google Analytics import.
* Unify errors when listing sites
* React: propagate backend error messages if available
* React: catch API errors in Search Terms component
* Propagate google API errors on referrer drilldown
* Handle verified properties errors in SC settings
* Add missing tests for SC settings controller
* Unify errors for fetching search analytics queries (list stats)
* Unify errors refreshing Google Auth Token
* Test fetch_stats/3 errors and replace Double with Mox
* Fixup makrup
* s/class/className
* Simplify Search Terms display in case of errors
* Fix warnings
This commit parses failed GA responses to JSON before reporting to
Sentry. This makes Sentry's PII filtering smarter, redacting only
specific keys from the response, instead of the whole string.
This pull request improves the current OpenTelemetry implementation. Currently only 1% of the spans are sent, due to the high volume of ingestion requests to /api/event. I enabled the 1% sampling to /api/event only, recording 100% of the other traces.
The Google Analytics report request may take some time, especially with big
imports with thousands of pages. To mitigate the issue this commit increases
the timeout to 60s and lowers the page size to 7,500 records per request.
This commit adds an option to override `http://localhost:8000` in
`mix send_event`. This is useful when running the server in another port
or with remote servers such as staging.
* Move clear stats functions to Plausible.Purge
* Delete both native and imported stats when deleting a site
This commit moves the delete site function to the Plausible.Purge
module, and fixes a bug where deleted sites could leave dangling
imported stats.
* Clear sites.stats_start_date after clearing stats
This commit fixes a bug where resetting stats left an invalid state of
the stats_start_date field, used for GA imports, for example.
* Refactor Plausible.Sites.stats_start_date/1 function
This commit adds documentation, typespecs and better pattern matching to
the Plausible.Sites.stats_start_date/1 function.
* Use memoized stats_start_date instead of querying CH
* Prevent domain from being updated from CRM
* Add stats_start_date to CRM site form
This commit adds stats_start_date to CRM site form, and creates a
separate Ecto changeset for CRM changes.
* Alias Plausible.Site calls
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* List all timezones in CRM form
* Require sites.public in CRM changeset
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Create API to check for an active subscription
* Redirect after upgrading only when subscription is created
* Change upgrade page copy
* Add timeout notice to upgrade success page
* Overrides content-type for SVG favicons
* Organize favicon rendering
Make sure the placeholder icon is always requested from
/favicon/sources/placeholder
* Run prettier on site-switcher.js
* Yak Shave: upgrade Heroicons to 2.0
* Use HeroIcons instead of custom svg
* Update lib/plausible_web/plugs/favicon.ex
Co-authored-by: Adam Rutkowski <hq@mtod.org>
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Accept letters from non-Latin alphabets in domain names
* Replace static URLs with Router functions in settings_visibility
* Beautify dashboard URL in visibility tab
* Add IDN support to CHANGELOG
This commit increases the receive_timeout from 15s to 30s for Google
Analytics import requests. Timeouts have been reported, and I could
replay some requests that took 15-20 seconds to complete when fetching
10,000 records in a single request.
* add separate module for filter parsing
* add tests for filter parser
* allow escaping pipe character in filter value
* add documentation and doctests
* do not remove escape chars from wildcard values
* changelog update
* change the parse_filters/1 function argument
This commit fixes an exception when calling `Timex.Timezone.get/2` after
there has been a timezone change. Similar issues have already been reported
in Timex (bitwaker/timex#691, bitwalker/timex#555, bitwalker/timex#625).
This skips timezones from the list when they fail to fetch, and use
`Timex.Timezone.get/2` to get the current offset using `DateTime.utc_now/0`
instead of the `NaiveDateTime` default.
* Return error tuple instead of exception when GA request fails
* Report Google Analytics failed request body to Sentry
* Improve Google Analytics function @doc
* fixup! Report Google Analytics failed request body to Sentry
* fixup! Improve Google Analytics function @doc
* Render 404 when shared link cannot be found
* Add documentation for StatsController and shared link rendering
* Refactor shared_link/2 for more clarity
* Add changelog entry
* Use mermaid graph for sequence diagram
* Use more accurate return value in sequence diagram
* Refactor Ecto query to be more idiomatic
* Remove order dependence in test
* Restore backwards compatibility for older shared links
* Add changelog entry
* Make sure admin can grant the admin role
* Use `site` and `current_user_role` from conn.assigns to avoid extra lookups
* Add some missing documentation
* Restrict owners in the `update_role` action
* Move @moduledoc to appropriate location
* Use Ecto.Enum to get role atoms
* Remove unused functionality
* Ensure that owners cannot lock themselves out
* Add question mark to `can_grant_role`
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Use strict short-circuit operator for booleans
Co-authored-by: Adam Rutkowski <hq@mtod.org>
Co-authored-by: Adam Rutkowski <hq@mtod.org>
This commit fixes a bug where the timezone list was sorted by the
standard offset, not considering timezone changes. The list was working
properly, but sorting was wrong.
* Update Timex version from 3.7.7 to 3.7.8
* Generate timezone list from Tzdata
This commit fixes a bug where timezone changes weren't updating the
timezone list displayed when editing or creating a site.
Timezones were being pulled from a static list. This commit changes it
to generate the list from Tzdata, that uses a timezone database with
updated information on time changes. Additionally it adds more timezones
with aliases and links to the list.
Closes#1340
* Use timezone name from browser to recommend timezone
This commit matches the timezone name instead of offset to recommend a
timezone when creating a new site. The JavaScript Intl.DateTimeFormat
API is widely supported according to the link. In any case, if the
timezone fails to match by name, it fallbacks to the offset strategy.
https://caniuse.com/mdn-javascript_builtins_intl_datetimeformat_resolvedoptions_computed_timezoneCloses#904
* Fixup error reporting (ref #2181)
Fixes two problems:
- `extra` sent to Sentry is always expected to be a map
- no `String.Chars` protocol is implemented for `Finch.Error` struct
* Use explicit paths in http client tests
* Make HTTP Client tests sync
* List all Google Analytics views during import
This commit fixes a bug where different Google Analytics views with the
same name and URI were not shown. This was caused because GA views were
stored as a map, that naturally doesn't support duplicate keys.
This change updates the GA views list to display view IDs, making it
clearer to know what is being imported. The dropdown is now grouped by
website URL.
* Put Google Analytics API URLs in app env
* Add controller test to GA view list
This commit removes some Enum calls to rely on the database for
aggregating data. This improves performance when creating new sites,
especially if the user has multiple sites.
* Replace current tooltip with Popper.js
* Merge tooltip and title for top stats
* Format bounce rate and visit duration numbers in tooltip
* Add 'width=manual' mode for embed
* Add changelog entry
* Use helper function canMetricBeGraphed
This commit handles unexpected Google Analytics responses while
importing, and logs to Sentry and Logger. This will improve
observability when things go wrong with GA imports.
* Replace Ingestion.Request headers with user_agent
* Replace generic Ingestion.Request params with specific fields
* Refactor event building function into small functions
* Move Plausible.Ingestion to Plausible.Ingestion.Event
* Add option to override event fields while building
* Rename Ingestion.Request meta to props
* Replace UTM-specific fields with generic query_params
* Remove Map.from_struct/1 call from ingestion pipeline
* Remove stash options from ingestion
* Remove invalid Jason.decode argument
Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
* Add custom message to Google invalid grant error
Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
* Test invalid_grant while refreshing Google token
Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
* Add Custom telemetry for Plausible.Event.WriteBuffer, Plausible.Event.WriteBuffer and Cachex
Signed-off-by: Manu S Ajith <neo@codingarena.in>
* Rename telemetry.ex to avoid confusion with Phoenix Telemetry supervisor
Signed-off-by: Manu S Ajith <neo@codingarena.in>
* Remove duplicate event
Signed-off-by: Manu S Ajith <neo@codingarena.in>
Signed-off-by: Manu S Ajith <neo@codingarena.in>
This commit fixes a bug where fetching Google Search Console keywords
raised a FunctionClauseError. This was introduced in #2046. This commit
also adds test coverage.
* Create separate module for GA HTTP requests
* Fetch GA data entirely instead of monthly
* Add buffering to GA imports
* Change positional args to maps when serializing from GA
* Create Google Analytics VCR tests
* Fix geolocation subdivision pattern matching
This commit fixes a bug where regions were not being saved. This was
caused because Geolix response was returning an additional
`:geolocation` map key. It also adds a test case for this.
Closes#2033
* Add geolocation database to .gitignore
This commit fixes a bug where location filters were filtering stats but
not the locations list. This was caused by a `Map.put/3` call that
overrides the user filter. This commit rollbacks 5b57143273
changes and removes the `Map.put/3` call.
Closes#1982
* Upgrade geolix
* Remove geolix pool config
* Save unnecessary Task.async_stream roundtrip
Normally the Geolix API accepts `:where` keyword option that designates
the database to look up. In case no parameter is supplied, it'll spawn
a parallel map over all databases available. In this case we have only
one DB anyway, so there is no need for the extra instrumentation.
* Follow up on direct :geolocation lookups
* Introduce Finch for Sentry integration
* Make sure the DummyAgent can be started
* No need to sanitize the dsn, finch takes care of that
* Simplify the dummy child spec
* Annotate redirects clause
* Make use of new `get_int_from_path_or_env`
* Actually use finch in Sentry config
* Configure `excluded_domains` correctly for Sentry
The way sentry is configured currently, when we get an HTTP error it
will be logged twice - once from Sentry.PlugCapture and once from
Sentry.LoggerBackend. The logger backend module does the right thing
by default but for some reason we've been overriding the config
parameter that by default stops double-counting errors. This commit
returns to the default configuration which is better.
* Default to 15s timeout
* Attempt to send twice at most
* Warn in sentry client
* Use warn level in sentry client
Co-authored-by: Adam Rutkowski <hq@mtod.org>
i've tried to make the two footers consistent to include the different links we were missing in the in-app footer (terms, dpa...) but also to display the new job opening for those that are logged in. i don't have a way to test it so hopefully i didn't make any mistakes
* Adds tri-state disable_registration config
* Formatting
* Changes variable back to atom
* Changelog
* Uses atoms correctly :/
* Swaps to a more fitting value
* Formatting
