Commit Graph

442 Commits

Author SHA1 Message Date
ruslandoga
e9ba60c8b4
improve transfer ownership error message (#2651)
* improve transfer ownership error message

* add changelog

* simplify

* revert changeset invitation error message

* more descriptive error message

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2023-02-13 15:25:17 +01:00
Vini Brasil
5159e683cc
First pass at Comparisons (#2650)
This commit adds support for comparing the actual showed period on the main graph with the previous one. This is a first pass and it's hidden under a feature flag because it's not feature complete yet as we want to support other comparison modes.
2023-02-07 10:00:49 -03:00
Adam Rutkowski
482249ffba
Better error message on site creation error (#2644)
* Ensure clear error message on site creation error

* Update colons in API error messages
2023-02-06 12:20:48 +01:00
Adam Rutkowski
f678cfab25
Prevent error reports from being sent with short feedback (#2635) 2023-02-01 05:36:14 -03:00
Adam Rutkowski
8f9f032968
Delay stats deletions (#2632)
* Implement Site removal transaction

* Implement Stats removal Oban worker

* Configure site removal queue

* Call Site.Removal.run() instead of Purge.delete_site!

* Test site/stats removal

* Remove FIXME - filed a ticket

* Over-communicate lenghty deletion process to the users
2023-01-31 16:11:04 -03:00
Adam Rutkowski
ad12e1ef31
Show user feedback form on server errors (#2617)
* Move Endpoint errors setup to common config

* Implement naive Sentry link resolver

* Implement error report e-mail

* Delete static sentry script

* Implement user feedback form on server errors

* Re-arrange pipe

* Use Sentry.Config.dsn() where applicable

* Fix typo

* Use Map.replace/3
2023-01-25 15:15:41 +01:00
Adam Rutkowski
1a4b65c36c
Communicate site creation edge cases to the users (#2605)
* Fingerprint DBConnection.ConnectionError in Sentry

* Check events before creating a site

* Enable sites limit screen

* Remove debugging remnant

* Fix buggy assertions

This wasn't doing what expected:

  iex(1)> Repo.exists?(Plausible.Site, domain: "foo")
  [debug] QUERY OK source="sites" db=0.6ms idle=1906.2ms
  SELECT TRUE FROM "sites" AS s0 LIMIT 1 []

* Encapsulate check to satisfy credo

* Use less technically involved error message

* Bring back e-mail to the limit error message
2023-01-19 15:03:18 +01:00
ruslandoga
166748dcf2
Replace Geolix with Locus (#2362)
This PR replaces geolix with locus to simplify self-hosted setup. locus can auto-update maxmind dbs which are recommended for self-hosters if they want city-level geolocation. locus is also a bit faster.

This PR also uses a test mmdb file from https://github.com/maxmind/MaxMind-DB for e2e geolocation tests without stubs.
2023-01-17 12:05:09 -03:00
Adam Rutkowski
61bf624761
Sentry cleanup (#2590)
* Filter DBConnection logs for clickhouse in Sentry

Removing the never matching clause btw

* Exclude Plug.CSRFProtection.InvalidCSRFTokenError from Sentry

* Turn common Sentry captures into logged warnings

* Exclude fonts from static conf and InvalidPathError from Sentry
2023-01-13 14:04:09 -03:00
RobertJoonas
13a05b4e1e
Change metric labels for realtime with goal filter (#2560)
* change metric labels for realtime with goal filter

* changelog

* fix formatting
2023-01-09 09:31:55 +02:00
Uku Taht
51fc9c0945
Refine parallel application-side joins (#2564)
* Consolidate task timeouts, incrase to 15s

* Use Task.async_stream for parallel Clickhouse queries

* Propagate Opentelemetry context to child Task process
2023-01-05 12:34:01 +01:00
Adam Rutkowski
44afbaa2f9
Wrap paddle actions with DB transactions (#2558)
* Wrap paddle actions with DB transactions

* Bow to credo

* Make the paddle transactions crash when applicable

* s/changeset/details
2023-01-03 16:14:50 +01:00
Uku Taht
a37b3433d7
Always show direct traffic in sources reports (#2531)
* Remove show_noref behaviour

Removes query param show_noref which was used from React to control
whether to show Direct / None traffic or not. The show_noref behaviour
was untested previously.

Closes #2523

* Add changelog entry

* Fix tests

* Removed files I did not mean to check in :)
2022-12-19 13:40:21 +01:00
Vini Brasil
478e0c6990
Handle GA authentication errors (#2505) 2022-12-07 23:32:14 -03:00
Vini Brasil
90786cb3bc
Remove intervals feature flag (#2474)
This commit removes the intervals feature flag, as this feature has
already been released to 100% of the users.
2022-11-29 10:12:36 -03:00
Adam Rutkowski
467317f24e
Fix: application startup, event timestamps (#2479)
### Changes

This PR:
- pushes PromEx to the bottom of supervision stack to avoid Endpoint
instrumentation failure
- ensures the site cache is ready by exposing it through the health
check endpoint
- fixes event timestamps being calculated at compile time, with
regression unit and integration tests

### Tests
- [x] Automated tests have been added
- [ ] This PR does not require tests

### Changelog
- [ ] Entry has been added to changelog
- [x] This PR does not make a user-facing change

### Documentation
- [ ] [Docs](https://github.com/plausible/docs) have been updated
- [x] This change does not need a documentation update

### Dark mode
- [ ] The UI has been tested both in dark and light mode
- [x] This PR does not change the UI
2022-11-29 10:46:49 +01:00
Adam Rutkowski
356575ef78
Gatekeep ingestion pipeline (#2472)
* Update Sites.Cache

So it's now capable of refreshing most recent sites.
Refreshing a single site is no longer wanted.

* Introduce Warmer.RecentlyUpdated

This is Sites Cache warmer that runs only for
most recently updated sites every 30s.

* Validate Request creation early

* Rename RateLimiter to GateKeeper and introduce detailed policies

* Update events API tests - a provisioned site is now required

* Update events ingestion tests

* Make limits visible in CRM Sites index

* Hard-deprecate DOMAIN_BLACKLIST

* Remove unnecessary clause

* Fix typo

* Explicitly delegate Warmer.All

* GateKeeper.allwoance => GateKeeper.check

* Instrument Sites.Cache measurments

* Update send_pageview task to output response headers

* Instrument ingestion pipeline

* Credo

* Make event telemetry test a sync case

* Simplify Request.uri/hostname handling

* Use embedded schema, apply action and rely on get_field
2022-11-28 15:50:55 +01:00
ruslandoga
77cca04417
fix ownership transfer when is_selfhost=true (#2455)
* fix ownership transfer when is_selfhost=true

* add changelog entry
2022-11-24 16:18:29 +02:00
Vini Brasil
994e7d09de
Parse event URL and domain in Plausible.Ingestion.Request (#2351)
* Parse event URL in Plausible.Ingestion.Request

* Parse event domain in Plausible.Ingestion.Request

* Rework ingestion pipeline processing (#2462)

* Rework ingestion pipeline processing

So that Request can have multiple domains and
based on that each event is processed uniformly.
The build_and_buffer/1 function now returns an
accumulator with all the dropped/buffered events
for further inspection.

* Reduce function complexity

* Don't chain struct fields to check for an empty host

* Separate referrer and utm tags

* Fix up `with` clause, credo was right cc @vinibrsl

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2022-11-23 14:05:44 +01:00
Vignesh Joglekar
497a52c10a
Add support for intervals (#1574)
Co-authored-by: Vini Brasil <vini@hey.com>
Co-authored-by: Vignesh Joglekar <hey@vigneshjoglekar.com>
2022-11-22 09:50:58 -03:00
Vini Brasil
9c98a3f2e8
Add API support for intervals (#2417) 2022-11-14 18:41:51 -03:00
ruslandoga
0b7870dc4d
improve first launch experience for self-hosters (#2357)
* first launch

* dynamic children, wait for repo

* remove wait_for_repo and app env manipulations

* don't mention free trial in self-hosted pages

* add changelog

* assigns[:is_selfhost] -> @is_selfhost

* better changelog wording

* rm admin_user, admin_email, admin_pwd from app env

* rm DISABLE_AUTH

* redirect / to /login when not authenticated

* remove TODO

* Update lib/plausible_web/controllers/page_controller.ex

Co-authored-by: Uku Taht <Uku.taht@gmail.com>

* format

Co-authored-by: Uku Taht <Uku.taht@gmail.com>
2022-11-10 12:42:22 +01:00
Uku Taht
8bdbabbdad
Enable Dialyzer and Credo in CI (#2315) 2022-11-04 23:42:03 -03:00
Adam Rutkowski
a44b8ff7f5
Improve test coverage (#2392)
Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
2022-10-27 08:39:34 +02:00
Adam Rutkowski
6ba5e53574
Trim goals when creating and retrieving from the DB (#2382)
People are likely to enter (copy/paste) goals from external sources
which can lead to whitespace characters appended by accident.
That renders the goal unusable and hard to distinct visually.

Normally to fix up existing goals we would use a data migration,
but this should be good enough to check if the problem
with never appearing goals resurfaces.
2022-10-26 09:35:30 +02:00
Adam Rutkowski
d29597ed37
Fix random typos (#2375) 2022-10-25 08:17:09 -03:00
Vinicius Brasil
9a61a10273
Do not report hard bounce e-mail errors to Sentry (#2361) 2022-10-24 07:13:23 -03:00
Adam Rutkowski
0fa6b688af
Google APIs integration improvements (#2358)
* Make TestUtils module available in all tests

* Add macros patching the application env in tests

Unfortunately a lot of existing functionality relies on
certain application env setup. This isn't ideal because
the app config is a shared state that prevents us from
running the tests in parallel.

Those macros encapsulate setting up new env for test purposes
and make sure the changes are reverted when the test finishes.

* Allow passing request opts to HTTPClient.post/4

We need this to swap custom request building in
Google Analytics import.

* Unify errors when listing sites

* React: propagate backend error messages if available

* React: catch API errors in Search Terms component

* Propagate google API errors on referrer drilldown

* Handle verified properties errors in SC settings

* Add missing tests for SC settings controller

* Unify errors for fetching search analytics queries (list stats)

* Unify errors refreshing Google Auth Token

* Test fetch_stats/3 errors and replace Double with Mox

* Fixup makrup

* s/class/className

* Simplify Search Terms display in case of errors

* Fix warnings
2022-10-24 09:34:02 +02:00
Vinicius Brasil
9220d0034d
OpenTelemetry (OTEL) Implementation (#2317)
This pull request improves the current OpenTelemetry implementation. Currently only 1% of the spans are sent, due to the high volume of ingestion requests to /api/event. I enabled the 1% sampling to /api/event only, recording 100% of the other traces.
2022-10-18 12:11:30 -03:00
Adam Rutkowski
ae4ae5d0a0
Handle duplicate invitations gracefully (#2323) 2022-10-11 11:40:20 -03:00
Adam Rutkowski
ec90a264b4
Improve user input validation (#2291)
* Make pagination params parsing ignore bad input

* Remove unused binding

* Don't crash on filter parse error

* Sanitize input date on internal stats API

* Revert Query module changes (ref 55645734)

* Implement simplistic input date validation in stats controller

* Mute bad request logging
2022-10-11 14:42:14 +02:00
Vinicius Brasil
0733efa89e
Delete stats improvements (#2318)
* Move clear stats functions to Plausible.Purge

* Delete both native and imported stats when deleting a site

This commit moves the delete site function to the Plausible.Purge
module, and fixes a bug where deleted sites could leave dangling
imported stats.

* Clear sites.stats_start_date after clearing stats

This commit fixes a bug where resetting stats left an invalid state of
the stats_start_date field, used for GA imports, for example.
2022-10-10 08:55:58 -03:00
Vinicius Brasil
4a2e0af337
Override stats_start_date in CRM (#2310)
* Refactor Plausible.Sites.stats_start_date/1 function

This commit adds documentation, typespecs and better pattern matching to
the Plausible.Sites.stats_start_date/1 function.

* Use memoized stats_start_date instead of querying CH

* Prevent domain from being updated from CRM

* Add stats_start_date to CRM site form

This commit adds stats_start_date to CRM site form, and creates a
separate Ecto changeset for CRM changes.

* Alias Plausible.Site calls

Co-authored-by: Adam Rutkowski <hq@mtod.org>

* List all timezones in CRM form

* Require sites.public in CRM changeset

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2022-10-06 08:08:22 -03:00
Vinicius Brasil
beabef7269
Redirect user after upgrading only when subscription is created (#2285)
* Create API to check for an active subscription

* Redirect after upgrading only when subscription is created

* Change upgrade page copy

* Add timeout notice to upgrade success page
2022-10-04 15:54:54 +03:00
Adam Rutkowski
bf2b6880c5
Block Stats API for locked sites (#2302)
* Extract Sites.locked? predicate

* Lock Stats API when dashboard is locked

* Tidy tests

* Don't pollute application env from tests

* Add changelog entry

* Revert "Add changelog entry"

This reverts commit 76346074f9.
2022-10-04 15:34:45 +03:00
Adam Rutkowski
c8ffcc413f
Prevent sensitive information from being logged (#2300) 2022-10-04 10:46:01 +03:00
Uku Taht
65f0ce3061
[Sentry APP-5B] Ensure form errors can be rendered on /settings (#2278)
* Ensure form errors can be rendered on /settings

* Changelog
2022-09-28 08:56:07 -03:00
Vinicius Brasil
52b4ee6287
fixup! Refresh Google Analytics token before import (#2254) (#2273) 2022-09-26 16:21:19 -03:00
Vinicius Brasil
1adda42a75
Refresh Google Analytics token before import (#2254)
* Capture refresh and expires from GA callback

* Pass GA refresh token to import worker

* Refresh GA token before import
2022-09-26 12:29:56 +03:00
Uku Taht
e16e357dd2
Fix shared link with bad auth (#2225)
* Render 404 when shared link cannot be found

* Add documentation for StatsController and shared link rendering

* Refactor shared_link/2 for more clarity

* Add changelog entry

* Use mermaid graph for sequence diagram

* Use more accurate return value in sequence diagram

* Refactor Ecto query to be more idiomatic

* Remove order dependence in test

* Restore backwards compatibility for older shared links

* Add changelog entry
2022-09-20 15:37:18 +03:00
Uku Taht
5d4918b66b
Allow admins to grant admin rights (#2216)
* Make sure admin can grant the admin role

* Use `site` and `current_user_role` from conn.assigns to avoid extra lookups

* Add some missing documentation

* Restrict owners in the `update_role` action

* Move @moduledoc to appropriate location

* Use Ecto.Enum to get role atoms

* Remove unused functionality

* Ensure that owners cannot lock themselves out

* Add question mark to `can_grant_role`

Co-authored-by: Adam Rutkowski <hq@mtod.org>

* Use strict short-circuit operator for booleans

Co-authored-by: Adam Rutkowski <hq@mtod.org>

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2022-09-16 11:03:09 +03:00
RobertJoonas
0194d57b38
add upper bound to limit parameter (#2226)
* add upper bound to limit parameter

* add more strict validation for limit parameter + 2 tests

* update changelog
2022-09-16 10:21:44 +03:00
Vinicius Brasil
d31db86b49
List all Google Analytics views during import (#2184)
* List all Google Analytics views during import

This commit fixes a bug where different Google Analytics views with the
same name and URI were not shown. This was caused because GA views were
stored as a map, that naturally doesn't support duplicate keys.

This change updates the GA views list to display view IDs, making it
clearer to know what is being imported. The dropdown is now grouped by
website URL.

* Put Google Analytics API URLs in app env

* Add controller test to GA view list
2022-09-08 21:02:17 +03:00
Adam Rutkowski
1a294854e3
Fix improper pattern matching on Finch responses (#2181) 2022-09-06 21:27:25 +02:00
Adam Rutkowski
2275cd8e22 Respond with 400 when no auth supplied to shared link request 2022-09-05 11:15:09 +02:00
Vinicius Brasil
e417c82a26
Perform calculations in SQL instead of app when creating sites (#2156)
This commit removes some Enum calls to rely on the database for
aggregating data. This improves performance when creating new sites,
especially if the user has multiple sites.
2022-09-01 17:09:28 +03:00
Vinicius Brasil
a10d44a0d7
Refactor event struct creation function (#2098)
* Replace Ingestion.Request headers with user_agent

* Replace generic Ingestion.Request params with specific fields

* Refactor event building function into small functions

* Move Plausible.Ingestion to Plausible.Ingestion.Event

* Add option to override event fields while building

* Rename Ingestion.Request meta to props

* Replace UTM-specific fields with generic query_params

* Remove Map.from_struct/1 call from ingestion pipeline

* Remove stash options from ingestion
2022-08-16 14:43:10 +03:00
Uku Taht
a5757bc61e Allow ownership transfers 2022-08-11 13:16:46 +03:00
ruslandoga
ce461b5192 Auth updates 2022-08-10 10:37:37 +03:00
Vinicius Brasil
d41fd68e99
Create struct for event requests (#2084)
* Create struct for saving ingestion request

* Create separate function to buffer events
2022-08-10 10:36:40 +03:00