Commit Graph

168 Commits

Author SHA1 Message Date
Uku Taht
f18a211dcc
Ingest throughput improvement test setup (#1867)
* Add OTEL and test Cachex for sessions

* Move load test

* Start apps in the appropriate order
2022-04-28 12:24:29 +03:00
Uku Taht
7c1d64458e Add fun with flags library 2022-04-21 10:54:08 +03:00
Uku Taht
680bd98bd1 Fix logic 2022-04-13 10:40:51 +03:00
Uku Taht
a282478838 Update cron config 2022-04-11 20:20:05 +03:00
Uku Taht
83c407c016
Upgrade Oban & configure Stager plugin (#1822) 2022-04-08 11:05:21 +03:00
Uku Taht
06b165eb6d Run GA import in monthly batches 2022-04-08 08:43:07 +03:00
Uku Taht
ae78444830 Add notice about feature preview 2022-03-25 11:22:02 +02:00
Uku Taht
4cc4e0d61b Add config flag for import testers 2022-03-25 10:46:43 +02:00
Uku Taht
a9879de1f4 Remove more OTEL stuff 2022-03-21 13:05:34 +02:00
Uku Taht
e27734ed79
[Continued] Google Analytics import (#1753)
* Add has_imported_stats boolean to Site

* Add Google Analytics import panel to general settings

* Get GA profiles to display in import settings panel

* Add import_from_google method as entrypoint to import data

* Add imported_visitors table

* Remove conflicting code from migration

* Import visitors data into clickhouse database

* Pass another dataset to main graph for rendering in red

This adds another entry to the JSON data returned via the main graph API
called `imported_plot`, which is similar to `plot` in form but will be
completed with previously imported data.  Currently it simply returns
the values from `plot` / 2. The data is rendered in the main graph in
red without fill, and without an indicator for the present. Rationale:
imported data will not continue to grow so there is no projection
forward, only backwards.

* Hook imported GA data to dashboard timeseries plot

* Add settings option to forget imported data

* Import sources from google analytics

* Merge imported sources when queried

* Merge imported source data native data when querying sources

* Start converting metrics to atoms so they can be subqueried

This changes "visitors" and in some places "sources" to atoms. This does
not change the behaviour of the functions - the tests all pass unchanged
following this commit. This is necessary as joining subqueries requires
that the keys in `select` statements be atoms and not strings.

* Convery GA (direct) source to empty string

* Import utm campaign and utm medium from GA

* format

* Import all data types from GA into new tables

* Handle large amounts of more data more safely

* Fix some mistakes in tables

* Make GA requests in chunks of 5 queries

* Only display imported timeseries when there is no filter

* Correctly show last 30 minutes timeseries when 'realtime'

* Add with_imported key to Query struct

* Account for injected :is_not filter on sources from dashboard

* Also add tentative imported_utm_sources table

This needs a bit more work on the google import side, as GA do not
report sources and utm sources as distinct things.

* Return imported data to dashboard for rest of Sources panel

This extends the merge_imported function definition for sources to
utm_sources, utm_mediums and utm_campaigns too. This appears to be
working on the DB side but something is incomplete on the client side.

* Clear imported stats from all tables when requested

* Merge entry pages and exit pages from imported data into unfiltered dashboard view

This requires converting the `"visits"` and `"visit_duration"` metrics
to atoms so that they can be used in ecto subqueries.

* Display imported devices, browsers and OSs on dashboard

* Display imported country data on dashboard

* Add more metrics to entries/exits for modals

* make sure data is returned via API with correct keys

* Import regions and cities from GA

* Capitalize device upon import to match native data

* Leave query limits/offsets until after possibly joining with imported data

* Also import timeOnPage and pageviews for pages from GA

* imported_countries -> imported_locations

* Get timeOnPage and pageviews for pages from GA

These are needed for the pages modal, and for calculating exit rates for
exit pages.

* Add indicator to dashboard when imported data is being used

* Don't show imported data as separately line on main graph

* "bounce_rate" -> :bounce_rate, so it works in subqueries

* Drop imported browser and OS versions

These are not needed.

* Toggle displaying imported data by clicking indicator

* Parse referrers with RefInspector

- Use 'ga:fullReferrer' instead of 'ga:source'. This provides the actual
  referrer host + path, whereas 'ga:source' includes utm_mediums and
  other values when relevant.
- 'ga:fullReferror' does however include search engine names directly,
  so they are manually checked for as RefInspector won't pick up on
  these.

* Keep imported data indicator on dashboard and strikethrough when hidden

* Add unlink google button to import panel

* Rename some GA browsers and OSes to plausible versions

* Get main top pages and exit pages panels working correctly with imported data

* mix format

* Fetch time_on_pages for imported data when needed

* entry pages need to fetch bounces from GA

* "sample_percent" -> :sample_percent as only atoms can be used in subqueries

* Calculate bounce_rate for joined native and imported data for top pages modal

* Flip some query bindings around to be less misleading

* Fixup entry page modal visit durations

* mix format

* Fetch bounces and visit_duration for sources from GA

* add more source metrics used for data in modals

* Make sources modals display correct values

* imported_visitors: bounce_rate -> bounces, avg_visit_duration -> visit_duration

* Merge imported data into aggregate stats

* Reformat top graph side icons

* Ensure sample_percent is yielded from aggregate data

* filter event_props should be strings

* Hide imported data from frontend when using filter

* Fix existing tests

* fix tests

* Fix imported indicator appearing when filtering

* comma needed, lost when rebasing

* Import utm_terms and utm_content from GA

* Merge imported utm_term and utm_content

* Rename imported Countries data as Locations

* Set imported city schema field to int

* Remove utm_terms and utm_content when clearing imported

* Clean locations import from Google Analytics

- Country and region should be set to "" when GA provides "(not set)"
- City should be set to 0 for "unknown", as we cannot reliably import
  city data from GA.

* Display imported region and city in dashboard

* os -> operating_system in some parts of code

The inconsistency of using os in some places and operating_system in
others causes trouble with subqueries and joins for the native and
imported data, which would require additional logic to account for. The
simplest solution is the just use a consistent word for all uses. This
doesn't make any user-facing or database changes.

* to_atom -> to_existing_atom

* format

* "events" metric -> :events

* ignore imported data when "events" in metrics

* update "bounce_rate"

* atomise some more metrics from new city and region api

* atomise some more metrics for email handlers

* "conversion_rate" -> :conversion_rate during csv export

* Move imported data stats code to own module

* Move imported timeseries function to Stats.Imported

* Use Timex.parse to import dates from GA

* has_imported_stats -> imported_source

* "time_on_page" -> :time_on_page

* Convert imported GA data to UTC

* Clean up GA request code a bit

There was some weird logic here with two separate lists that really
ought to be together, so this merges those.

* Fail sooner if GA timezone can't be identified

* Link imported tables to site by id

* imported_utm_content -> imported_utm_contents

* Imported GA from all of time

* Reorganise GA data fetch logic

- Fetch data from the start of time (2005)
- Check whether no data was fetched, and if so, inform user and don't
  consider data to be imported.

* Clarify removal of "visits" data when it isn't in metrics

* Apply location filters from API

This makes it consistent with the sources etc which filter out 'Direct /
None' on the API side. These filters are used by both the native and
imported data handling code, which would otherwise both duplicate the
filters in their `where` clauses.

* Do not use changeset for setting site.imported_source

* Add all metrics to all dimensions

* Run GA import in the background

* Send email when GA import completes

* Add handler to insert imported data into tests and imported_browsers_factory

* Add remaining import data test factories

* Add imported location data to test

* Test main graph with imported data

* Add imported data to operating systems tests

* Add imported data to pages tests

* Add imported data to entry pages tests

* Add imported data to exit pages tests

* Add imported data to devices tests

* Add imported data to sources tests

* Add imported data to UTM tests

* Add new test module for the data import step

* Test import of sources GA data

* Test import of utm_mediums GA data

* Test import of utm_campaigns GA data

* Add tests for UTM terms

* Add tests for UTM contents

* Add test for importing pages and entry pages data from GA

* Add test for importing exit page data

* Fix module file name typo

* Add test for importing location data from GA

* Add test for importing devices data from GA

* Add test for importing browsers data from GA

* Add test for importing OS data from GA

* Paginate GA requests to download all data

* Bump clickhouse_ecto version

* Move RefInspector wrapper function into module

* Drop timezone transform on import

* Order imported by side_id then date

* More strings -> atoms

Also changes a conditional to be a bit nicer

* Remove parallelisation of data import

* Split sources and UTM sources from fetched GA data

GA has only a "source" dimension and no "UTM source" dimension. Instead
it returns these combined. The logic herein to tease these apart is:

1. "(direct)" -> it's a direct source
2. if the source is a domain -> it's a source
3. "google" -> it's from adwords; let's make this a UTM source "adwords"
4. else -> just a UTM source

* Keep prop names in queries as strings

* fix typo

* Fix import

* Insert data to clickhouse in batches

* Fix link when removing imported data

* Merge source tables

* Import hostname as well as pathname

* Record start and end time of imported data

* Track import progress

* Fix month interval with imported data

* Do not JOIN when imported date range has no overlap

* Fix time on page using exits

Co-authored-by: mcol <mcol@posteo.net>
2022-03-10 15:04:59 -06:00
RobertJoonas
6b22806e1f
Allow admin access to locked dashboards (#1710)
* added super-admin access to locked dashboards

* fixed formatting
2022-02-23 13:48:33 -06:00
Eric Sullivan
0ac26cb0fc
optional ipv6 for repo (#1661)
* optional ipv6 for repo

* changelog
2022-02-16 21:26:26 -06:00
Uku Taht
69576aa253 Default to 127.0.0.1 listen_ip on non-docker use-cases 2022-01-21 14:23:09 -06:00
Niklas Hambüchen
1337b46e52
Allow user to specify listen interface via LISTEN_IP. (#1190)
This enables safer deployments to allow localhost-only
or VPN-interface-only listening.
2022-01-21 14:04:29 -06:00
Uku Taht
880ea45cd9 Remove cities feature flag 2021-12-31 12:16:25 +02:00
Uku Taht
711cbdfca2 Remove unused custom domain code 2021-12-20 11:49:50 +02:00
Uku Taht
1dba113e2f
[Draft] Improve location translations (#1526)
* WIP

* Use location library for search suggestions

* Remove unused code

* Remove Countries completely

* Fix tests
2021-12-13 12:03:27 +02:00
RobertJoonas
9022234aa6
added invoices section (#1511)
* added invoices section

* fixed formatting

* UI and code style improvements, paddle sandbox setup only on dev environment (fixed)
2021-12-09 15:49:57 +02:00
Uku Taht
4d0bc61ffd Remove Twitter stuff 2021-12-02 11:53:29 +02:00
Uku Taht
05bf43c1be
City level location data (#1449)
* Merge branch 'plausible_master'

* Add City level details

* Add City level details

* Use ISO codes instead of geoname_id for subdivisions

* Add easier way to configure geolocation database

* Add workflow for dev branch

* Correct clickhouse migration

* Translate subdivision names

* Translate city names

* WIP

* Region and country filters

* Fix region filter

* Remove region_name when removing region filter

* Add modals for regions and cities

* Remove dead code

* WIP

* Revert "WIP"

This reverts commit 3202bf2fe9.

* Feature flag to hide cities when deployed

* Add changelog entry

* Remove unused code

* Remove unused variables

* Fix test

Co-authored-by: AymanTerra <aymanterra@yahoo.com>
2021-11-23 11:39:09 +02:00
Uku Taht
76f6a00bec Add honeycomb 2021-11-05 14:58:57 +02:00
Lorenz Brun
a4389c99c7
Don't enforce longer SECRET_KEY_BASE than Phoenix (#1441)
Phoenix only requires SECRET_KEY_BASE to be longer than 32 characters. 32 characters even with hex encoding is 128 bits of entropy which is more than enough. Some secret generation tools only generate 32 characters/128 bits by default which makes this relatively arbitrary limit annoying. Let's change it to what Phoenix requires.
2021-11-04 14:16:37 +02:00
Uku Taht
6a5b383e2b Add enterprise plans 2021-10-20 16:49:11 +02:00
Uku Taht
950bd012d5 Simplify geolix database config 2021-10-18 15:49:55 +02:00
Uku Taht
2ca3baeafd Disable email verification by default 2021-10-18 12:01:54 +02:00
Uku Taht
85fcdbdbd6 Respect the path component of BASE_URL 2021-10-18 11:16:56 +02:00
Uku Taht
8c6f4b7d85 Revert "Configure database SSL"
This reverts commit 46ca66805a.
2021-10-09 15:33:20 +02:00
Uku Taht
46ca66805a Configure database SSL 2021-10-09 15:28:25 +02:00
Uku Taht
d8144436e1 Use unlimited maximum connections 2021-10-09 15:18:34 +02:00
Uku Taht
015dcfa051 HOTFIX: add domain blacklist 2021-09-30 09:45:52 +02:00
Uku Taht
c68b88c24b
Upgrade otp (#1313)
* update tool versions to latest

* Upgrades erlang and elixir, format project

Co-authored-by: happysalada <raphael@megzari.com>
2021-09-09 11:17:24 +03:00
Uku Taht
369f63bc58 Adds an option to rename the script file 2021-07-23 15:17:32 +03:00
Uku Taht
c68d90c8b0 Make default site limit 50 2021-06-29 16:38:10 +03:00
Uku Taht
6eb982f7fd Fix issue with config loading 2021-06-29 15:26:42 +03:00
Uku Taht
0ef0fd2048 Fix config option in test 2021-06-16 15:43:26 +03:00
Uku Taht
18ddf9dd9d Admin access 2021-06-16 15:33:37 +03:00
Uku Taht
e71de6dc1f
Invitations (#1122)
* Invite existing user to a site

* Add invitation flow for non-existing users

* Accept and reject invitations

* Use invitation flow for existing users

* Locking mechanism for sites

* Authorization for site settings

* Show usage based on site ownership

* Add ability to remove members from a site

* Do not show settings link to viewer roles

* Ability to remove invitations

* Remove `Plausible.Sites.count_for/1`

* Fix tests

* Do not show the trial banner after the trial

* Correct trial emails

* Transfer ownership

* Send invitation email to existing user

* Add invitation email flows

* Add plug for role-based authorization

* Rename AuthorizeStatsPlug -> AuthorizeSiteAccess

* Add email flow for ownership transfer

* Fix URLs in emails

* Fix small copy issues

* Make 'People' its own section in site settings

* Notify user via email if their access has been removed

* Check site lock status when invitation is accepted

* Check lock status when user subscribes

* Make sure only admins and owners can create shared links

* Changelog

* Add LockSites to daily cron

* Clean invitations after 48 hours

* Add notices about expiry

* Add invitation expired page

* Add doc link
2021-06-16 15:00:07 +03:00
Uku Taht
7e229140ed Move config files to config directory 2021-06-16 12:16:11 +03:00
Uku Taht
3fe8bd6681
Pr/1117 formatted (#1129)
* runtime.exs: get var from file

* Format pr/1117

Co-authored-by: happysalada <raphael@megzari.com>
2021-06-16 12:03:01 +03:00
Uku Taht
0054c9fc96 Add api keys to CRM 2021-06-10 10:25:19 +03:00
Uku Taht
d9cc21fc06 Add site limit exempt config 2021-06-04 14:31:48 +03:00
Raphael Megzari
a0a0c3a8db
add tzdata storage dir (#1104) 2021-06-03 10:55:48 +03:00
Maximilian Bosch
4ff25f68c8
Postgres socket authentication (#1052)
* Allow to connect to postgres via socket auth

* Add changelog entry for new env-vars

Co-authored-by: Uku Taht <Uku.taht@gmail.com>
2021-05-26 15:40:56 +03:00
Andrey Meshkov
90cfed035f
Added parameters for Clickhouse flush interval and max buffer size (#1073)
Currently, both are hard-coded to relatively small values.
This may cause an unnecessary high load on the Clickhouse instance.
It makes sense to make those configurable.
2021-05-25 11:37:52 +03:00
Uku Taht
6228995d4f Rate limit login attempts to 5 per minute 2021-05-25 11:32:54 +03:00
Uku Taht
22172074d8 Remove json logger 2021-05-24 11:07:19 +03:00
Uku Taht
23196b97ec Increase clickhouse pool checkout target 2021-05-06 18:03:42 +03:00
Uku Taht
503916980e Remove no_limit_ids 2021-05-05 10:57:35 +03:00
Uku Taht
c3b24342c6 Allow exemption list to be provided in config 2021-05-05 10:45:01 +03:00
Uku Taht
119b9514b2 Add limit of 20 sites 2021-05-05 10:30:05 +03:00
Uku Taht
8d1039b8ce
Upgrades Oban to v2.6.1 (#967)
* Upgrades Oban to v2.6.1

* Use correct clickhouse_ecto
2021-04-26 11:32:18 +03:00
Uku Taht
7982c236c1 Better Sentry tracking 2021-04-23 11:56:41 +03:00
Uku Taht
fbe6ab9b61 Remove AppSignal 2021-04-23 11:42:05 +03:00
Uku Taht
6ca7425dc9
Send renewal notification for annual subscriptions (#949)
* Add background job to send renewal notification

* Use copy from Marko

* Add expiration email in case the user has cancelled

* Add queue config

* Use correct tag for expiration email
2021-04-21 15:57:38 +03:00
Uku Taht
4fd5cdff58 Add download step to docker build 2021-04-13 12:36:47 +03:00
Uku Taht
8ba21d2470 Use correct pruning syntax 2021-04-13 11:35:32 +03:00
Uku Taht
be26662f36 Crash early if secret_key_base is not configured correctly 2021-04-13 11:19:24 +03:00
Uku Taht
0d71ec3acd Keep longer history of oban jobs 2021-04-12 14:51:22 +03:00
Uku Taht
83a759c60a
Use db-ip lite database for countries (#906)
* Use DBIP database for countries

* Add fake data for geolocation tests
2021-04-12 12:21:07 +03:00
Uku Taht
4cbebe8ac9 Remove old Sentry code 2021-03-31 11:38:14 +03:00
Uku Taht
3b8fad4942 Upgrade sentry to 8.0 2021-03-30 15:51:05 +03:00
Uku Taht
2bab36a438 Add usage checking to cron 2021-02-19 15:15:56 +02:00
Uku Taht
2d8a03e861 Add JSON logger 2021-01-29 10:24:20 +02:00
Uku Taht
99a3582883 Add option to completely disable cron 2021-01-18 11:28:18 +02:00
Uku Taht
e873d790e5
Self hosted improvements (#603)
* Introduce SELFHOST config. Enable cron by default for self hosters

* Fix cron, disable custom domain for selfhost

* Add github links

* Add notice about google search console on self hosted

* Enfore BASE_URL

* Set all selfhost users' trial expiry to 100 years from now

* Fix admin user creation
2021-01-15 10:12:00 +02:00
Uku Taht
563c8d99f4 Fix new configuration 2021-01-13 17:04:01 +02:00
Uku Taht
7d8951435e Add test configuration 2021-01-13 16:15:03 +02:00
Uku Taht
6c8ae24adc
New config (#595)
* Use Config instead of Mix.Config

* Use new config/runtime.exs to reduce duplication
2021-01-13 15:41:16 +02:00