Commit Graph

79 Commits

Author SHA1 Message Date
Vini Brasil
48b02c1396
Avoid suggesting plan for trials switching to enterprise plans (#3282)
This commit fixes a bug where the SendTrialNotifications job tried to
suggest a plan for users switching to enterprise plans, resulting in the
exception below:

```
UndefinedFunctionError: function :enterprise.volume/0 is undefined (module :enterprise is not available)
  Module "enterprise", in :enterprise.volume/0
  File "lib/plausible_web/templates/email/trial_upgrade_email.html.eex", line 5, in PlausibleWeb.EmailView."trial_upgrade_email.html"/1
  File "lib/phoenix_view.ex", line 381, in Phoenix.View.render_within/3
  File "lib/phoenix_view.ex", line 557, in Phoenix.View.render_to_iodata/3
  File "lib/phoenix_view.ex", line 564, in Phoenix.View.render_to_string/3
  File "lib/bamboo_phoenix.ex", line 291, in Bamboo.Phoenix.render_text_or_html_email/1
  File "lib/workers/send_trial_notifications.ex", line 67, in Plausible.Workers.SendTrialNotifications.send_today_reminder/1
  File "lib/workers/send_trial_notifications.ex", line 36, in anonymous fn/2 in Plausible.Workers.SendTrialNotifications.perform/1
```
2023-08-21 09:01:50 -03:00
hq1
9f0e7d54f8
Fix flaky log assertion (#3199) 2023-07-27 12:51:54 +02:00
hq1
e6996ccfda
Fix weekly/monthly report e-mails (MJML) (#3150)
* Render weekly/monthly reports with MJML

* Update changelog

* Fix APP-2NA (#3119)
2023-07-17 16:59:01 +02:00
hq1
9debcdee24
Fix APP-2NA (#3119) 2023-07-11 10:47:14 +02:00
Vini Brasil
f0bdf872b5
Add eventually/1 call to flaky test (#3054) 2023-06-19 10:32:34 +01:00
hq1
71ef0bd043
Clean up after V2 migration (#2868)
* Clean up after V2 migration

This PR removes all the leftovers and alternative code
branching after v2 migration.

The self-hosted release is being drafted at:

https://github.com/plausible/hosting/issues/68

Refs:
  - https://github.com/plausible/analytics/pull/2865
  - https://github.com/plausible/analytics/pull/2825
  - https://github.com/plausible/analytics/pull/2780

* !fixup
2023-04-24 12:17:57 +02:00
hq1
1d01328287
Allow domain change (#2803)
* Migration (PR: https://github.com/plausible/analytics/pull/2802)

* Implement Site.Domain interface allowing change and expiry

* Fixup seeds so they work with V2_MIGRATION_DONE=1

* Update Sites.Cache so it's capable of multi-keyed lookups

* Implement worker handling domain change expiration

* Implement domain change UI

* Implement transition period for public APIs

* Exclude v2 tests in primary test run

* Update lib/plausible_web/controllers/site_controller.ex

Co-authored-by: Vini Brasil <vini@hey.com>

* Update lib/plausible_web/controllers/site_controller.ex

Co-authored-by: Vini Brasil <vini@hey.com>

* Update moduledoc

* Update changelog

* Remove remnant from previous implementation attempt

* !fixup

* !fixup

* Implement domain change via Sites API

cc @ukutaht

* Update CHANGELOG

* Credo

* !fixup commit missing tests

* Allow continuous domain change within the same site

---------

Co-authored-by: Vini Brasil <vini@hey.com>
2023-04-04 10:55:12 +02:00
hq1
d2f2c69387
Conditionally support switching between v1 and v2 clickhouse schemas (#2780)
* Remove ClickhouseSetup module

This has been an implicit point of contact to many
tests. From now on the goal is for each test to maintain
its own, isolated setup so that no accidental clashes
and implicit assumptions are relied upon.

* Implement v2 schema check

An environment variable V2_MIGRATION_DONE acts like
a feature flag, switching plausible from using old events/sessions
schemas to v2 schemas introduced by NumericIDs migration.

* Run both test suites sequentially

While the code for v1 and v2 schemas must be kept still,
we will from now on run tests against both code paths.
Secondary test run will set V2_MIGRATION_DONE=1 variable,
thus making all `Plausible.v2?()` checks return `true'.

* Remove unused function

This is a remnant from the short period when
we would check for existing events before allowing
creating a new site.

* Update test setups/factories with v2 migration check

* Make GateKeeper return site id along with :allow

* Make Billing module check for v2 schema

* Make ingestion aware of v2 schema

* Disable site transfers for when v2 is live

In a separate changeset we will implement simplified
site transfer for when v2 migration is complete.
The new transfer will only rename the site domain in postgres
and keep track of the original site prior to the transfer
so we keep an ingestion grace period until the customers
redeploy their scripting.

* Make Stats base queries aware of v2 schema switch

* Update breakdown with v2 conditionals

* Update pageview local start with v2 check

* Update current visitoris with v2 check

* Update stats controller with v2 checks

* Update external controller with v2 checks

* Update remaining tests with proper fixtures

* Rewrite redundant assignment

* Remove unused alias

* Mute credo, this is not the right time

* Add test_helper prompt

* Fetch priv dir so it works with a release

* Fetch distinct partitions only

* Don't limit inspect output for partitions

* Ensure SQL is printed to IO

* Remove redundant domain fixture
2023-03-27 13:52:42 +02:00
Adam
8f86036e57
Keep track of native stats start timestamp when retrieving data (#2715)
* Stats boundary/PoC?

* Delete stats removal

* Drop events check on site creation

* Update seeds script

* Use native_stats_start_at

* Don't rely on native stats pointer in imported stats queries

* Reset site

* Export reset/1

* Remove unnecessary inserted_at settings

* Update seeds

* Remove unnecessary inserted_at setting
2023-03-01 13:11:31 +01:00
Adam Rutkowski
ef894b0546
Fail the CI if tests emit log errors/warnings (#2694)
* Make CI fail if tests emit warning/error logs

* Swallow log output in GA test case
2023-02-21 10:38:22 +01:00
Robert
fbe3a4e6f1 add a test and fix rendering the dashboard link 2022-11-03 11:31:32 +02:00
Vinicius Brasil
c9945e09cf
Suggest enterprise plan when usage is greater than 10M pageviews (#2345) 2022-10-25 13:16:44 +02:00
Adam Rutkowski
0fa6b688af
Google APIs integration improvements (#2358)
* Make TestUtils module available in all tests

* Add macros patching the application env in tests

Unfortunately a lot of existing functionality relies on
certain application env setup. This isn't ideal because
the app config is a shared state that prevents us from
running the tests in parallel.

Those macros encapsulate setting up new env for test purposes
and make sure the changes are reverted when the test finishes.

* Allow passing request opts to HTTPClient.post/4

We need this to swap custom request building in
Google Analytics import.

* Unify errors when listing sites

* React: propagate backend error messages if available

* React: catch API errors in Search Terms component

* Propagate google API errors on referrer drilldown

* Handle verified properties errors in SC settings

* Add missing tests for SC settings controller

* Unify errors for fetching search analytics queries (list stats)

* Unify errors refreshing Google Auth Token

* Test fetch_stats/3 errors and replace Double with Mox

* Fixup makrup

* s/class/className

* Simplify Search Terms display in case of errors

* Fix warnings
2022-10-24 09:34:02 +02:00
Vinicius Brasil
40182f6d6e
Manually lock outgrown enterprise accounts (#2197)
* Rename enterprise_over_limit e-mail template

* Change suggested_plan/2 function to suggest enterprise plans

* Start grace period for enterprise accounts

* Create separate module for GracePeriod schema

* Enable manual locking of enterprise accounts

* Update outgrown account alert for enterprise accounts

* Document account locking feature

* fixup! Enable manual locking of enterprise accounts
2022-09-20 11:46:28 +03:00
Uku Taht
948de2b4c9 Mark more tests as async 2022-05-06 10:30:38 +03:00
Uku Taht
6f3399bb7c Fix annual notification loop 2022-05-03 17:00:19 +03:00
Uku Taht
d2dc06ec96 Fix trial notifications 2022-04-18 11:05:08 +03:00
Kian-Meng Ang
70cb3eed54
Fix typos (#1829) 2022-04-11 20:42:40 +03:00
Uku Taht
06b165eb6d Run GA import in monthly batches 2022-04-08 08:43:07 +03:00
Uku Taht
333de87ceb Add stats_start_date field 2022-04-06 10:10:53 +03:00
Uku Taht
51cd24bcaf Allow user to cancel ongoing import 2022-03-23 11:58:36 +02:00
Uku Taht
203f87520b
Ga import improvements (#1784)
* Do not link Google account for import

* Record start date

* Fix tests
2022-03-22 16:09:45 +02:00
Uku Taht
59acff02f8 Fix tests 2022-03-21 12:55:20 +02:00
Uku Taht
e27734ed79
[Continued] Google Analytics import (#1753)
* Add has_imported_stats boolean to Site

* Add Google Analytics import panel to general settings

* Get GA profiles to display in import settings panel

* Add import_from_google method as entrypoint to import data

* Add imported_visitors table

* Remove conflicting code from migration

* Import visitors data into clickhouse database

* Pass another dataset to main graph for rendering in red

This adds another entry to the JSON data returned via the main graph API
called `imported_plot`, which is similar to `plot` in form but will be
completed with previously imported data.  Currently it simply returns
the values from `plot` / 2. The data is rendered in the main graph in
red without fill, and without an indicator for the present. Rationale:
imported data will not continue to grow so there is no projection
forward, only backwards.

* Hook imported GA data to dashboard timeseries plot

* Add settings option to forget imported data

* Import sources from google analytics

* Merge imported sources when queried

* Merge imported source data native data when querying sources

* Start converting metrics to atoms so they can be subqueried

This changes "visitors" and in some places "sources" to atoms. This does
not change the behaviour of the functions - the tests all pass unchanged
following this commit. This is necessary as joining subqueries requires
that the keys in `select` statements be atoms and not strings.

* Convery GA (direct) source to empty string

* Import utm campaign and utm medium from GA

* format

* Import all data types from GA into new tables

* Handle large amounts of more data more safely

* Fix some mistakes in tables

* Make GA requests in chunks of 5 queries

* Only display imported timeseries when there is no filter

* Correctly show last 30 minutes timeseries when 'realtime'

* Add with_imported key to Query struct

* Account for injected :is_not filter on sources from dashboard

* Also add tentative imported_utm_sources table

This needs a bit more work on the google import side, as GA do not
report sources and utm sources as distinct things.

* Return imported data to dashboard for rest of Sources panel

This extends the merge_imported function definition for sources to
utm_sources, utm_mediums and utm_campaigns too. This appears to be
working on the DB side but something is incomplete on the client side.

* Clear imported stats from all tables when requested

* Merge entry pages and exit pages from imported data into unfiltered dashboard view

This requires converting the `"visits"` and `"visit_duration"` metrics
to atoms so that they can be used in ecto subqueries.

* Display imported devices, browsers and OSs on dashboard

* Display imported country data on dashboard

* Add more metrics to entries/exits for modals

* make sure data is returned via API with correct keys

* Import regions and cities from GA

* Capitalize device upon import to match native data

* Leave query limits/offsets until after possibly joining with imported data

* Also import timeOnPage and pageviews for pages from GA

* imported_countries -> imported_locations

* Get timeOnPage and pageviews for pages from GA

These are needed for the pages modal, and for calculating exit rates for
exit pages.

* Add indicator to dashboard when imported data is being used

* Don't show imported data as separately line on main graph

* "bounce_rate" -> :bounce_rate, so it works in subqueries

* Drop imported browser and OS versions

These are not needed.

* Toggle displaying imported data by clicking indicator

* Parse referrers with RefInspector

- Use 'ga:fullReferrer' instead of 'ga:source'. This provides the actual
  referrer host + path, whereas 'ga:source' includes utm_mediums and
  other values when relevant.
- 'ga:fullReferror' does however include search engine names directly,
  so they are manually checked for as RefInspector won't pick up on
  these.

* Keep imported data indicator on dashboard and strikethrough when hidden

* Add unlink google button to import panel

* Rename some GA browsers and OSes to plausible versions

* Get main top pages and exit pages panels working correctly with imported data

* mix format

* Fetch time_on_pages for imported data when needed

* entry pages need to fetch bounces from GA

* "sample_percent" -> :sample_percent as only atoms can be used in subqueries

* Calculate bounce_rate for joined native and imported data for top pages modal

* Flip some query bindings around to be less misleading

* Fixup entry page modal visit durations

* mix format

* Fetch bounces and visit_duration for sources from GA

* add more source metrics used for data in modals

* Make sources modals display correct values

* imported_visitors: bounce_rate -> bounces, avg_visit_duration -> visit_duration

* Merge imported data into aggregate stats

* Reformat top graph side icons

* Ensure sample_percent is yielded from aggregate data

* filter event_props should be strings

* Hide imported data from frontend when using filter

* Fix existing tests

* fix tests

* Fix imported indicator appearing when filtering

* comma needed, lost when rebasing

* Import utm_terms and utm_content from GA

* Merge imported utm_term and utm_content

* Rename imported Countries data as Locations

* Set imported city schema field to int

* Remove utm_terms and utm_content when clearing imported

* Clean locations import from Google Analytics

- Country and region should be set to "" when GA provides "(not set)"
- City should be set to 0 for "unknown", as we cannot reliably import
  city data from GA.

* Display imported region and city in dashboard

* os -> operating_system in some parts of code

The inconsistency of using os in some places and operating_system in
others causes trouble with subqueries and joins for the native and
imported data, which would require additional logic to account for. The
simplest solution is the just use a consistent word for all uses. This
doesn't make any user-facing or database changes.

* to_atom -> to_existing_atom

* format

* "events" metric -> :events

* ignore imported data when "events" in metrics

* update "bounce_rate"

* atomise some more metrics from new city and region api

* atomise some more metrics for email handlers

* "conversion_rate" -> :conversion_rate during csv export

* Move imported data stats code to own module

* Move imported timeseries function to Stats.Imported

* Use Timex.parse to import dates from GA

* has_imported_stats -> imported_source

* "time_on_page" -> :time_on_page

* Convert imported GA data to UTC

* Clean up GA request code a bit

There was some weird logic here with two separate lists that really
ought to be together, so this merges those.

* Fail sooner if GA timezone can't be identified

* Link imported tables to site by id

* imported_utm_content -> imported_utm_contents

* Imported GA from all of time

* Reorganise GA data fetch logic

- Fetch data from the start of time (2005)
- Check whether no data was fetched, and if so, inform user and don't
  consider data to be imported.

* Clarify removal of "visits" data when it isn't in metrics

* Apply location filters from API

This makes it consistent with the sources etc which filter out 'Direct /
None' on the API side. These filters are used by both the native and
imported data handling code, which would otherwise both duplicate the
filters in their `where` clauses.

* Do not use changeset for setting site.imported_source

* Add all metrics to all dimensions

* Run GA import in the background

* Send email when GA import completes

* Add handler to insert imported data into tests and imported_browsers_factory

* Add remaining import data test factories

* Add imported location data to test

* Test main graph with imported data

* Add imported data to operating systems tests

* Add imported data to pages tests

* Add imported data to entry pages tests

* Add imported data to exit pages tests

* Add imported data to devices tests

* Add imported data to sources tests

* Add imported data to UTM tests

* Add new test module for the data import step

* Test import of sources GA data

* Test import of utm_mediums GA data

* Test import of utm_campaigns GA data

* Add tests for UTM terms

* Add tests for UTM contents

* Add test for importing pages and entry pages data from GA

* Add test for importing exit page data

* Fix module file name typo

* Add test for importing location data from GA

* Add test for importing devices data from GA

* Add test for importing browsers data from GA

* Add test for importing OS data from GA

* Paginate GA requests to download all data

* Bump clickhouse_ecto version

* Move RefInspector wrapper function into module

* Drop timezone transform on import

* Order imported by side_id then date

* More strings -> atoms

Also changes a conditional to be a bit nicer

* Remove parallelisation of data import

* Split sources and UTM sources from fetched GA data

GA has only a "source" dimension and no "UTM source" dimension. Instead
it returns these combined. The logic herein to tease these apart is:

1. "(direct)" -> it's a direct source
2. if the source is a domain -> it's a source
3. "google" -> it's from adwords; let's make this a UTM source "adwords"
4. else -> just a UTM source

* Keep prop names in queries as strings

* fix typo

* Fix import

* Insert data to clickhouse in batches

* Fix link when removing imported data

* Merge source tables

* Import hostname as well as pathname

* Record start and end time of imported data

* Track import progress

* Fix month interval with imported data

* Do not JOIN when imported date range has no overlap

* Fix time on page using exits

Co-authored-by: mcol <mcol@posteo.net>
2022-03-10 15:04:59 -06:00
Uku Taht
60f116facd Fix tests 2022-01-03 11:55:24 +02:00
Uku Taht
0b253421ce Fix tests 2022-01-03 11:34:59 +02:00
Uku Taht
711cbdfca2 Remove unused custom domain code 2021-12-20 11:49:50 +02:00
Uku Taht
01706b7590 Remove dead code 2021-12-03 11:19:39 +02:00
Uku Taht
4d0bc61ffd Remove Twitter stuff 2021-12-02 11:53:29 +02:00
Uku Taht
e7e36b994d Skip usage checking for users who are on grace period 2021-12-02 11:42:34 +02:00
Uku Taht
06d0d0eafa Fix over limit email reccommendation 2021-11-23 11:21:55 +02:00
Uku Taht
d3311898c5 Show message when grace period is over 2021-11-16 10:14:24 +02:00
Uku Taht
e9cb8eb4e2 Remove grace period if user upgrades 2021-11-16 10:14:24 +02:00
Uku Taht
ec2a4a1a65 Add [Action required] to over limit email 2021-11-16 10:14:24 +02:00
Uku Taht
29cb7462e6 Add grace period to upgrade 2021-11-16 10:14:23 +02:00
Uku Taht
e66e9bd1b7 Check site limit for enterprise customers 2021-10-22 11:26:07 +02:00
Uku Taht
6a5b383e2b Add enterprise plans 2021-10-20 16:49:11 +02:00
Uku Taht
01412d090f Fix sampling in JOIN queries 2021-08-31 11:13:09 +03:00
Uku Taht
0de89bad82
Add filter groups (#1167)
* Add filter groups

* New flow for filters

* Visual consistency

* Mobile improvements to dropdown

* Do not let filters wrap on a new line

* Fix country filter

* Add mix format to pre-commit configuration

* Make eslint happy

* Fix formatting

* Use transition from headlessUI
2021-07-21 16:50:26 +03:00
Uku Taht
fef5337a64 Make sure only the current subcription is used in expiry notifications 2021-07-19 15:03:41 +03:00
Uku Taht
883841f789 Make sure the latest subscription is always used 2021-06-18 11:13:54 +03:00
Uku Taht
27f99da8ad Do not send email reports for locked sites 2021-06-17 10:20:54 +03:00
Uku Taht
e71de6dc1f
Invitations (#1122)
* Invite existing user to a site

* Add invitation flow for non-existing users

* Accept and reject invitations

* Use invitation flow for existing users

* Locking mechanism for sites

* Authorization for site settings

* Show usage based on site ownership

* Add ability to remove members from a site

* Do not show settings link to viewer roles

* Ability to remove invitations

* Remove `Plausible.Sites.count_for/1`

* Fix tests

* Do not show the trial banner after the trial

* Correct trial emails

* Transfer ownership

* Send invitation email to existing user

* Add invitation email flows

* Add plug for role-based authorization

* Rename AuthorizeStatsPlug -> AuthorizeSiteAccess

* Add email flow for ownership transfer

* Fix URLs in emails

* Fix small copy issues

* Make 'People' its own section in site settings

* Notify user via email if their access has been removed

* Check site lock status when invitation is accepted

* Check lock status when user subscribes

* Make sure only admins and owners can create shared links

* Changelog

* Add LockSites to daily cron

* Clean invitations after 48 hours

* Add notices about expiry

* Add invitation expired page

* Add doc link
2021-06-16 15:00:07 +03:00
Uku Taht
f16666e374 Localize billing screens 2021-05-13 12:42:01 +03:00
Uku Taht
793957ed90 fix check usage test 2021-05-12 11:28:44 +03:00
Uku Taht
d33fe1cf97 Use new pricing in trial email test 2021-05-11 12:05:17 +03:00
Uku Taht
7c520e16d2 Make discount notices more generic 2021-05-11 11:34:25 +03:00
Uku Taht
1b91909021 Grandfather old plans 2021-05-11 11:32:29 +03:00
Uku Taht
885b1d271a Refactor current plan data format to make changing it easier 2021-05-11 11:32:29 +03:00
Uku Taht
07653079e8 Don't fail oban job when provision_ssl_certificates fails 2021-04-28 15:25:30 +03:00