* Replace Ingestion.Request headers with user_agent
* Replace generic Ingestion.Request params with specific fields
* Refactor event building function into small functions
* Move Plausible.Ingestion to Plausible.Ingestion.Event
* Add option to override event fields while building
* Rename Ingestion.Request meta to props
* Replace UTM-specific fields with generic query_params
* Remove Map.from_struct/1 call from ingestion pipeline
* Remove stash options from ingestion
* Remove invalid Jason.decode argument
Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
* Add custom message to Google invalid grant error
Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
* Test invalid_grant while refreshing Google token
Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
* Add Custom telemetry for Plausible.Event.WriteBuffer, Plausible.Event.WriteBuffer and Cachex
Signed-off-by: Manu S Ajith <neo@codingarena.in>
* Rename telemetry.ex to avoid confusion with Phoenix Telemetry supervisor
Signed-off-by: Manu S Ajith <neo@codingarena.in>
* Remove duplicate event
Signed-off-by: Manu S Ajith <neo@codingarena.in>
Signed-off-by: Manu S Ajith <neo@codingarena.in>
This commit fixes a bug where fetching Google Search Console keywords
raised a FunctionClauseError. This was introduced in #2046. This commit
also adds test coverage.
* Create separate module for GA HTTP requests
* Fetch GA data entirely instead of monthly
* Add buffering to GA imports
* Change positional args to maps when serializing from GA
* Create Google Analytics VCR tests
This commit fixes a bug where location filters were filtering stats but
not the locations list. This was caused by a `Map.put/3` call that
overrides the user filter. This commit rollbacks 5b57143273
changes and removes the `Map.put/3` call.
Closes#1982
* Introduce Finch for Sentry integration
* Make sure the DummyAgent can be started
* No need to sanitize the dsn, finch takes care of that
* Simplify the dummy child spec
* Annotate redirects clause
* Make use of new `get_int_from_path_or_env`
* Actually use finch in Sentry config
* Configure `excluded_domains` correctly for Sentry
The way sentry is configured currently, when we get an HTTP error it
will be logged twice - once from Sentry.PlugCapture and once from
Sentry.LoggerBackend. The logger backend module does the right thing
by default but for some reason we've been overriding the config
parameter that by default stops double-counting errors. This commit
returns to the default configuration which is better.
* Default to 15s timeout
* Attempt to send twice at most
* Warn in sentry client
* Use warn level in sentry client
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Plugin exceptions may be silently covering up database problems such
as failed cron job inserts.
* The previously tracked `:circuit` event is no longer emitted.
* added custom dimension filtering tests for pages
* first filter UI in place
* pages, entry pages and exit pages can be filtered by pageview props
* added tests for expected filtering behaviour
* fix dimension filter for sources + tests
* added is_not filtering functionality
* fixed formatting
* fixed admin_test
* added (none) as filter value + is_not filter type in UI
* added prefilling applied filter values and some UI tweaks
* added fetch options
* Make prop suggestions work with `props` filter
* Fix test
* Track login state internally
* Add CHANGELOG entry
Co-authored-by: Uku Taht <uku.taht@gmail.com>
* small refactors and adds 'contains' to modal
* supports contains filter in the backend
* moves entry and exit page under the page filter
* prettier
* updates the CHANGELOG
* undo package-lock changes
* fixes formatting for elixir
* renames unused parameter to _
* Update changelog
* Use uppercase for constants and update type/prefix lookup
* can now query events metric when breaking down on a visit prop
* updated changelog
* fixed formatting
* fixed a bug - querying for bounce rate does not return visits anymore
* removed the changelog update
* Add has_imported_stats boolean to Site
* Add Google Analytics import panel to general settings
* Get GA profiles to display in import settings panel
* Add import_from_google method as entrypoint to import data
* Add imported_visitors table
* Remove conflicting code from migration
* Import visitors data into clickhouse database
* Pass another dataset to main graph for rendering in red
This adds another entry to the JSON data returned via the main graph API
called `imported_plot`, which is similar to `plot` in form but will be
completed with previously imported data. Currently it simply returns
the values from `plot` / 2. The data is rendered in the main graph in
red without fill, and without an indicator for the present. Rationale:
imported data will not continue to grow so there is no projection
forward, only backwards.
* Hook imported GA data to dashboard timeseries plot
* Add settings option to forget imported data
* Import sources from google analytics
* Merge imported sources when queried
* Merge imported source data native data when querying sources
* Start converting metrics to atoms so they can be subqueried
This changes "visitors" and in some places "sources" to atoms. This does
not change the behaviour of the functions - the tests all pass unchanged
following this commit. This is necessary as joining subqueries requires
that the keys in `select` statements be atoms and not strings.
* Convery GA (direct) source to empty string
* Import utm campaign and utm medium from GA
* format
* Import all data types from GA into new tables
* Handle large amounts of more data more safely
* Fix some mistakes in tables
* Make GA requests in chunks of 5 queries
* Only display imported timeseries when there is no filter
* Correctly show last 30 minutes timeseries when 'realtime'
* Add with_imported key to Query struct
* Account for injected :is_not filter on sources from dashboard
* Also add tentative imported_utm_sources table
This needs a bit more work on the google import side, as GA do not
report sources and utm sources as distinct things.
* Return imported data to dashboard for rest of Sources panel
This extends the merge_imported function definition for sources to
utm_sources, utm_mediums and utm_campaigns too. This appears to be
working on the DB side but something is incomplete on the client side.
* Clear imported stats from all tables when requested
* Merge entry pages and exit pages from imported data into unfiltered dashboard view
This requires converting the `"visits"` and `"visit_duration"` metrics
to atoms so that they can be used in ecto subqueries.
* Display imported devices, browsers and OSs on dashboard
* Display imported country data on dashboard
* Add more metrics to entries/exits for modals
* make sure data is returned via API with correct keys
* Import regions and cities from GA
* Capitalize device upon import to match native data
* Leave query limits/offsets until after possibly joining with imported data
* Also import timeOnPage and pageviews for pages from GA
* imported_countries -> imported_locations
* Get timeOnPage and pageviews for pages from GA
These are needed for the pages modal, and for calculating exit rates for
exit pages.
* Add indicator to dashboard when imported data is being used
* Don't show imported data as separately line on main graph
* "bounce_rate" -> :bounce_rate, so it works in subqueries
* Drop imported browser and OS versions
These are not needed.
* Toggle displaying imported data by clicking indicator
* Parse referrers with RefInspector
- Use 'ga:fullReferrer' instead of 'ga:source'. This provides the actual
referrer host + path, whereas 'ga:source' includes utm_mediums and
other values when relevant.
- 'ga:fullReferror' does however include search engine names directly,
so they are manually checked for as RefInspector won't pick up on
these.
* Keep imported data indicator on dashboard and strikethrough when hidden
* Add unlink google button to import panel
* Rename some GA browsers and OSes to plausible versions
* Get main top pages and exit pages panels working correctly with imported data
* mix format
* Fetch time_on_pages for imported data when needed
* entry pages need to fetch bounces from GA
* "sample_percent" -> :sample_percent as only atoms can be used in subqueries
* Calculate bounce_rate for joined native and imported data for top pages modal
* Flip some query bindings around to be less misleading
* Fixup entry page modal visit durations
* mix format
* Fetch bounces and visit_duration for sources from GA
* add more source metrics used for data in modals
* Make sources modals display correct values
* imported_visitors: bounce_rate -> bounces, avg_visit_duration -> visit_duration
* Merge imported data into aggregate stats
* Reformat top graph side icons
* Ensure sample_percent is yielded from aggregate data
* filter event_props should be strings
* Hide imported data from frontend when using filter
* Fix existing tests
* fix tests
* Fix imported indicator appearing when filtering
* comma needed, lost when rebasing
* Import utm_terms and utm_content from GA
* Merge imported utm_term and utm_content
* Rename imported Countries data as Locations
* Set imported city schema field to int
* Remove utm_terms and utm_content when clearing imported
* Clean locations import from Google Analytics
- Country and region should be set to "" when GA provides "(not set)"
- City should be set to 0 for "unknown", as we cannot reliably import
city data from GA.
* Display imported region and city in dashboard
* os -> operating_system in some parts of code
The inconsistency of using os in some places and operating_system in
others causes trouble with subqueries and joins for the native and
imported data, which would require additional logic to account for. The
simplest solution is the just use a consistent word for all uses. This
doesn't make any user-facing or database changes.
* to_atom -> to_existing_atom
* format
* "events" metric -> :events
* ignore imported data when "events" in metrics
* update "bounce_rate"
* atomise some more metrics from new city and region api
* atomise some more metrics for email handlers
* "conversion_rate" -> :conversion_rate during csv export
* Move imported data stats code to own module
* Move imported timeseries function to Stats.Imported
* Use Timex.parse to import dates from GA
* has_imported_stats -> imported_source
* "time_on_page" -> :time_on_page
* Convert imported GA data to UTC
* Clean up GA request code a bit
There was some weird logic here with two separate lists that really
ought to be together, so this merges those.
* Fail sooner if GA timezone can't be identified
* Link imported tables to site by id
* imported_utm_content -> imported_utm_contents
* Imported GA from all of time
* Reorganise GA data fetch logic
- Fetch data from the start of time (2005)
- Check whether no data was fetched, and if so, inform user and don't
consider data to be imported.
* Clarify removal of "visits" data when it isn't in metrics
* Apply location filters from API
This makes it consistent with the sources etc which filter out 'Direct /
None' on the API side. These filters are used by both the native and
imported data handling code, which would otherwise both duplicate the
filters in their `where` clauses.
* Do not use changeset for setting site.imported_source
* Add all metrics to all dimensions
* Run GA import in the background
* Send email when GA import completes
* Add handler to insert imported data into tests and imported_browsers_factory
* Add remaining import data test factories
* Add imported location data to test
* Test main graph with imported data
* Add imported data to operating systems tests
* Add imported data to pages tests
* Add imported data to entry pages tests
* Add imported data to exit pages tests
* Add imported data to devices tests
* Add imported data to sources tests
* Add imported data to UTM tests
* Add new test module for the data import step
* Test import of sources GA data
* Test import of utm_mediums GA data
* Test import of utm_campaigns GA data
* Add tests for UTM terms
* Add tests for UTM contents
* Add test for importing pages and entry pages data from GA
* Add test for importing exit page data
* Fix module file name typo
* Add test for importing location data from GA
* Add test for importing devices data from GA
* Add test for importing browsers data from GA
* Add test for importing OS data from GA
* Paginate GA requests to download all data
* Bump clickhouse_ecto version
* Move RefInspector wrapper function into module
* Drop timezone transform on import
* Order imported by side_id then date
* More strings -> atoms
Also changes a conditional to be a bit nicer
* Remove parallelisation of data import
* Split sources and UTM sources from fetched GA data
GA has only a "source" dimension and no "UTM source" dimension. Instead
it returns these combined. The logic herein to tease these apart is:
1. "(direct)" -> it's a direct source
2. if the source is a domain -> it's a source
3. "google" -> it's from adwords; let's make this a UTM source "adwords"
4. else -> just a UTM source
* Keep prop names in queries as strings
* fix typo
* Fix import
* Insert data to clickhouse in batches
* Fix link when removing imported data
* Merge source tables
* Import hostname as well as pathname
* Record start and end time of imported data
* Track import progress
* Fix month interval with imported data
* Do not JOIN when imported date range has no overlap
* Fix time on page using exits
Co-authored-by: mcol <mcol@posteo.net>
* Add `utm_content` and `utm_term`.
Support `utm_content` and `utm_term` as requested in #515.
* Add dropdown for UTM options
* Remove utm_content and term from filter modal for now
Co-authored-by: Blender Defender <defenderblender@gmail.com>
* adding api route PUT /api/v1/sites/goals with form fields "goal_type" and "goal_value" with supported types "event" and "page"
* arm64 docker images
* adding api route DELETE /api/v1/sites/goals/:goal_id with form param "site_id"
* revert makfile + package.json
* return statement hotfix in case site could not be found
* adding api route PUT /api/v1/sites/goals/:goal_id with form params "site_id", "goal_type", and "goal_value"
* update the goal api routes to accept event_name or page_path instead of goal_value
* cleaning goals model
* mix format
Co-authored-by: Ahmed Abbas <a.abbas@ixdc.net>
* Merge branch 'plausible_master'
* Add City level details
* Add City level details
* Use ISO codes instead of geoname_id for subdivisions
* Add easier way to configure geolocation database
* Add workflow for dev branch
* Correct clickhouse migration
* Translate subdivision names
* Translate city names
* WIP
* Region and country filters
* Fix region filter
* Remove region_name when removing region filter
* Add modals for regions and cities
* Remove dead code
* WIP
* Revert "WIP"
This reverts commit 3202bf2fe9.
* Feature flag to hide cities when deployed
* Add changelog entry
* Remove unused code
* Remove unused variables
* Fix test
Co-authored-by: AymanTerra <aymanterra@yahoo.com>
* Round time_on_page metric returned by 'pages' API
This rounds the `time_on_page` metric returned as part of the `pages`
API to the nearest second. While this shows no apparent change in the
web UI, it removes the decimal from the exported data in `pages.csv`.
* Tidy up export tests
* Round time_on_page in db query
* Add goal to CSV export tests
* Display CSV export spinner until download is ready
The mechanism used to make the page aware that the download is ready is:
- Client code sets a cookie and requests download.
- Server code handles download and removes cookie when complete.
- Client code polls every 1s to check the cookies, removing spinner when
the export cookie is removed.
* Add details=True to export API parameters
This makes the ZIP export add `%{"details" => "True"}` to the query's
`params` when fetching data internally for packaging in the ZIP.
This adds bounce_rate and time_on_page to the data in pages.csv, and
bounce_rate and visit_duration to sources.csv.
* Make API return data with consistent names
Some of the data types returned via the JSON or CSV API use inconsistent
naming, and some have redundant name changes (i.e. count -> visitors ->
count). This makes these all consistent and removes the redundancy.
This addresses #1426, fixes some of the CSV headers, and unifies the
JSON and CSV return data labels.
* Update changelog
* Test should use Timex.shift, not relative time
* Return full country names in CSV export
This also replaces the " character with ' in two country names, as those
are the characters used in the names, yielding a more predictable and
'correct' output.
* Fetch CSV exported data concurrently
* Use spinner to indicate when export has started
* Use 300 as default number of brekadown entries for export
Higher numbers (e.g. 1000) seem to cause clickhouse errors when there
many pages to request. It is unclear what is causing the error, as
clickhouse returns an "unknown" error code and an empty error message.
* Adds pagination for custom props, fixes errant (none) being returned in the middle of a dataset
* Formatting
* Fixes errant (none) value correctly
* Changelog
* Adds tests
* Invite existing user to a site
* Add invitation flow for non-existing users
* Accept and reject invitations
* Use invitation flow for existing users
* Locking mechanism for sites
* Authorization for site settings
* Show usage based on site ownership
* Add ability to remove members from a site
* Do not show settings link to viewer roles
* Ability to remove invitations
* Remove `Plausible.Sites.count_for/1`
* Fix tests
* Do not show the trial banner after the trial
* Correct trial emails
* Transfer ownership
* Send invitation email to existing user
* Add invitation email flows
* Add plug for role-based authorization
* Rename AuthorizeStatsPlug -> AuthorizeSiteAccess
* Add email flow for ownership transfer
* Fix URLs in emails
* Fix small copy issues
* Make 'People' its own section in site settings
* Notify user via email if their access has been removed
* Check site lock status when invitation is accepted
* Check lock status when user subscribes
* Make sure only admins and owners can create shared links
* Changelog
* Add LockSites to daily cron
* Clean invitations after 48 hours
* Add notices about expiry
* Add invitation expired page
* Add doc link
* First pass on manual filter & path regex/negated filters
Still needs:
- Form structure on filter modal
- Edit filter button
- Filter dropdown UI improvement
- Filter modal mount data collection
- Tests
- Potentially negating other filters
* Second pass - mostly everything user-facing is done
Still needs:
- Tests
- Potentially negating other filters
- Potentially some code cleanup
* Bugfix in realtime view, formatting
* Fixes editing UX on list view
* Adds tests for exclusions and wildcards
* Various UI Improvements
- Makes edit buttons full-length & properly sized
- Adds remove filter button in edit menu
* Changelog
* Makes requested changes, adds different version of filter button
* Makes colorings on top bar elements consistent
Currently, both are hard-coded to relatively small values.
This may cause an unnecessary high load on the Clickhouse instance.
It makes sense to make those configurable.
* First pass
Needs more testing & potentially cleanup
* Fixes tests, error handling
* Formatting
* Removes broken test
* Fixes inconsistent test
This was due to Clickhouse setup not inserting the sessions with the exact same timestamp consistently and making the test inconsistent
* Combines `include` param, asyncs time_on_page and bounce_rate
* Fixes CH error when no pageviews exist in period
* Format
* Changelog
* Increases await timeout to accomodate larger data sets
* Improves handling of timeout behavior
* Fixes session-based filtering on time on page queries
* Formatting
* Removes old forced entry page modal from top pages report
* Add background job to send renewal notification
* Use copy from Marko
* Add expiration email in case the user has cancelled
* Add queue config
* Use correct tag for expiration email
