Commit Graph

319 Commits

Author SHA1 Message Date
Uku Taht
e27734ed79
[Continued] Google Analytics import (#1753)
* Add has_imported_stats boolean to Site

* Add Google Analytics import panel to general settings

* Get GA profiles to display in import settings panel

* Add import_from_google method as entrypoint to import data

* Add imported_visitors table

* Remove conflicting code from migration

* Import visitors data into clickhouse database

* Pass another dataset to main graph for rendering in red

This adds another entry to the JSON data returned via the main graph API
called `imported_plot`, which is similar to `plot` in form but will be
completed with previously imported data.  Currently it simply returns
the values from `plot` / 2. The data is rendered in the main graph in
red without fill, and without an indicator for the present. Rationale:
imported data will not continue to grow so there is no projection
forward, only backwards.

* Hook imported GA data to dashboard timeseries plot

* Add settings option to forget imported data

* Import sources from google analytics

* Merge imported sources when queried

* Merge imported source data native data when querying sources

* Start converting metrics to atoms so they can be subqueried

This changes "visitors" and in some places "sources" to atoms. This does
not change the behaviour of the functions - the tests all pass unchanged
following this commit. This is necessary as joining subqueries requires
that the keys in `select` statements be atoms and not strings.

* Convery GA (direct) source to empty string

* Import utm campaign and utm medium from GA

* format

* Import all data types from GA into new tables

* Handle large amounts of more data more safely

* Fix some mistakes in tables

* Make GA requests in chunks of 5 queries

* Only display imported timeseries when there is no filter

* Correctly show last 30 minutes timeseries when 'realtime'

* Add with_imported key to Query struct

* Account for injected :is_not filter on sources from dashboard

* Also add tentative imported_utm_sources table

This needs a bit more work on the google import side, as GA do not
report sources and utm sources as distinct things.

* Return imported data to dashboard for rest of Sources panel

This extends the merge_imported function definition for sources to
utm_sources, utm_mediums and utm_campaigns too. This appears to be
working on the DB side but something is incomplete on the client side.

* Clear imported stats from all tables when requested

* Merge entry pages and exit pages from imported data into unfiltered dashboard view

This requires converting the `"visits"` and `"visit_duration"` metrics
to atoms so that they can be used in ecto subqueries.

* Display imported devices, browsers and OSs on dashboard

* Display imported country data on dashboard

* Add more metrics to entries/exits for modals

* make sure data is returned via API with correct keys

* Import regions and cities from GA

* Capitalize device upon import to match native data

* Leave query limits/offsets until after possibly joining with imported data

* Also import timeOnPage and pageviews for pages from GA

* imported_countries -> imported_locations

* Get timeOnPage and pageviews for pages from GA

These are needed for the pages modal, and for calculating exit rates for
exit pages.

* Add indicator to dashboard when imported data is being used

* Don't show imported data as separately line on main graph

* "bounce_rate" -> :bounce_rate, so it works in subqueries

* Drop imported browser and OS versions

These are not needed.

* Toggle displaying imported data by clicking indicator

* Parse referrers with RefInspector

- Use 'ga:fullReferrer' instead of 'ga:source'. This provides the actual
  referrer host + path, whereas 'ga:source' includes utm_mediums and
  other values when relevant.
- 'ga:fullReferror' does however include search engine names directly,
  so they are manually checked for as RefInspector won't pick up on
  these.

* Keep imported data indicator on dashboard and strikethrough when hidden

* Add unlink google button to import panel

* Rename some GA browsers and OSes to plausible versions

* Get main top pages and exit pages panels working correctly with imported data

* mix format

* Fetch time_on_pages for imported data when needed

* entry pages need to fetch bounces from GA

* "sample_percent" -> :sample_percent as only atoms can be used in subqueries

* Calculate bounce_rate for joined native and imported data for top pages modal

* Flip some query bindings around to be less misleading

* Fixup entry page modal visit durations

* mix format

* Fetch bounces and visit_duration for sources from GA

* add more source metrics used for data in modals

* Make sources modals display correct values

* imported_visitors: bounce_rate -> bounces, avg_visit_duration -> visit_duration

* Merge imported data into aggregate stats

* Reformat top graph side icons

* Ensure sample_percent is yielded from aggregate data

* filter event_props should be strings

* Hide imported data from frontend when using filter

* Fix existing tests

* fix tests

* Fix imported indicator appearing when filtering

* comma needed, lost when rebasing

* Import utm_terms and utm_content from GA

* Merge imported utm_term and utm_content

* Rename imported Countries data as Locations

* Set imported city schema field to int

* Remove utm_terms and utm_content when clearing imported

* Clean locations import from Google Analytics

- Country and region should be set to "" when GA provides "(not set)"
- City should be set to 0 for "unknown", as we cannot reliably import
  city data from GA.

* Display imported region and city in dashboard

* os -> operating_system in some parts of code

The inconsistency of using os in some places and operating_system in
others causes trouble with subqueries and joins for the native and
imported data, which would require additional logic to account for. The
simplest solution is the just use a consistent word for all uses. This
doesn't make any user-facing or database changes.

* to_atom -> to_existing_atom

* format

* "events" metric -> :events

* ignore imported data when "events" in metrics

* update "bounce_rate"

* atomise some more metrics from new city and region api

* atomise some more metrics for email handlers

* "conversion_rate" -> :conversion_rate during csv export

* Move imported data stats code to own module

* Move imported timeseries function to Stats.Imported

* Use Timex.parse to import dates from GA

* has_imported_stats -> imported_source

* "time_on_page" -> :time_on_page

* Convert imported GA data to UTC

* Clean up GA request code a bit

There was some weird logic here with two separate lists that really
ought to be together, so this merges those.

* Fail sooner if GA timezone can't be identified

* Link imported tables to site by id

* imported_utm_content -> imported_utm_contents

* Imported GA from all of time

* Reorganise GA data fetch logic

- Fetch data from the start of time (2005)
- Check whether no data was fetched, and if so, inform user and don't
  consider data to be imported.

* Clarify removal of "visits" data when it isn't in metrics

* Apply location filters from API

This makes it consistent with the sources etc which filter out 'Direct /
None' on the API side. These filters are used by both the native and
imported data handling code, which would otherwise both duplicate the
filters in their `where` clauses.

* Do not use changeset for setting site.imported_source

* Add all metrics to all dimensions

* Run GA import in the background

* Send email when GA import completes

* Add handler to insert imported data into tests and imported_browsers_factory

* Add remaining import data test factories

* Add imported location data to test

* Test main graph with imported data

* Add imported data to operating systems tests

* Add imported data to pages tests

* Add imported data to entry pages tests

* Add imported data to exit pages tests

* Add imported data to devices tests

* Add imported data to sources tests

* Add imported data to UTM tests

* Add new test module for the data import step

* Test import of sources GA data

* Test import of utm_mediums GA data

* Test import of utm_campaigns GA data

* Add tests for UTM terms

* Add tests for UTM contents

* Add test for importing pages and entry pages data from GA

* Add test for importing exit page data

* Fix module file name typo

* Add test for importing location data from GA

* Add test for importing devices data from GA

* Add test for importing browsers data from GA

* Add test for importing OS data from GA

* Paginate GA requests to download all data

* Bump clickhouse_ecto version

* Move RefInspector wrapper function into module

* Drop timezone transform on import

* Order imported by side_id then date

* More strings -> atoms

Also changes a conditional to be a bit nicer

* Remove parallelisation of data import

* Split sources and UTM sources from fetched GA data

GA has only a "source" dimension and no "UTM source" dimension. Instead
it returns these combined. The logic herein to tease these apart is:

1. "(direct)" -> it's a direct source
2. if the source is a domain -> it's a source
3. "google" -> it's from adwords; let's make this a UTM source "adwords"
4. else -> just a UTM source

* Keep prop names in queries as strings

* fix typo

* Fix import

* Insert data to clickhouse in batches

* Fix link when removing imported data

* Merge source tables

* Import hostname as well as pathname

* Record start and end time of imported data

* Track import progress

* Fix month interval with imported data

* Do not JOIN when imported date range has no overlap

* Fix time on page using exits

Co-authored-by: mcol <mcol@posteo.net>
2022-03-10 15:04:59 -06:00
Uku Taht
d4e7d27df6 Merge branch 'new-plans' 2021-12-30 11:09:13 +02:00
Uku Taht
e4b99dbad6
Fix Security error with localstorage access (#1568)
* Fix Security error with localstorage access

* Changelog
2021-12-29 11:08:11 +02:00
Uku Taht
0bf1da09ce New plans 2021-12-28 12:26:53 +02:00
Uku Taht
48ad7485c8
PR 1393 continued (#1542)
* Add `utm_content` and `utm_term`.
Support `utm_content` and `utm_term` as requested in #515.

* Add dropdown for UTM options

* Remove utm_content and term from filter modal for now

Co-authored-by: Blender Defender <defenderblender@gmail.com>
2021-12-16 11:02:09 +02:00
Uku Taht
1dba113e2f
[Draft] Improve location translations (#1526)
* WIP

* Use location library for search suggestions

* Remove unused code

* Remove Countries completely

* Fix tests
2021-12-13 12:03:27 +02:00
Uku Taht
4d0bc61ffd Remove Twitter stuff 2021-12-02 11:53:29 +02:00
Uku Taht
05bf43c1be
City level location data (#1449)
* Merge branch 'plausible_master'

* Add City level details

* Add City level details

* Use ISO codes instead of geoname_id for subdivisions

* Add easier way to configure geolocation database

* Add workflow for dev branch

* Correct clickhouse migration

* Translate subdivision names

* Translate city names

* WIP

* Region and country filters

* Fix region filter

* Remove region_name when removing region filter

* Add modals for regions and cities

* Remove dead code

* WIP

* Revert "WIP"

This reverts commit 3202bf2fe9.

* Feature flag to hide cities when deployed

* Add changelog entry

* Remove unused code

* Remove unused variables

* Fix test

Co-authored-by: AymanTerra <aymanterra@yahoo.com>
2021-11-23 11:39:09 +02:00
Émile Perron
fbd6e37767
Add option to specify a URL in the manual extension (#1479)
* Add option to specify a URL in the manual extension

* Fix error when option parameter isn't provided
The uglify-js dependency had to be updated to support the optional chaining operator

* Remove the possibility to pass a function for the custom URL
This added weight unnecessarily, as the users can simply call the function themselves before triggering the event.

* Change optional chaining to ternary for improved browser support

* Revert package-lock.json to version 3.9.4 of uglify-js
It was previously updated to support optional chaining; reverting it as a reminder that this is the current browser support level that is accepted by Plausible.
2021-11-23 10:50:06 +02:00
Uku Taht
e85f0f13cc Improve Clickhouse DDL 2021-11-22 16:06:21 +02:00
Uku Taht
e9cb8eb4e2 Remove grace period if user upgrades 2021-11-16 10:14:24 +02:00
Uku Taht
29cb7462e6 Add grace period to upgrade 2021-11-16 10:14:23 +02:00
Janosh Riebesell
e7593ce94f
Additional pre-commit hooks (#1428)
* add 1st-party langauge agnnostic pre-commit hooks

* white space fixes from pre-commit run -av

* fix CI test/plausible_web/controllers/stats_controller_test.exs#L153

* Revert "fix CI test/plausible_web/controllers/stats_controller_test.exs#L153"

This reverts commit 85c9450efe.

* git checkout 94f476eb~1 -- priv/ref_inspector priv/ua_inspector
2021-11-03 15:46:12 +01:00
Uku Taht
e66e9bd1b7 Check site limit for enterprise customers 2021-10-22 11:26:07 +02:00
Uku Taht
086d4de74e Remove window.phantom from automated browser detection
Fixes #1403
2021-10-21 13:57:30 +02:00
Uku Taht
6a5b383e2b Add enterprise plans 2021-10-20 16:49:11 +02:00
Uku Taht
23a4f37ab4 Add 'manual' script variant 2021-10-10 13:07:37 +02:00
Uku Taht
a7eab3f17b Add null check to localstorage access
Fixes #1225
2021-09-23 11:16:19 +02:00
Uku Taht
700a65c98a
Remove trial banner for admins & viewers (#1308)
* Start trial only when the user creates a site

* End trial when ownership is transfered
2021-09-08 15:15:37 +03:00
Uku Taht
33ba5d79b6
Memoize has_stats? (#1302)
* Memoize has_stats?

* Remove unused alias
2021-09-06 13:54:51 +03:00
Uku Taht
a84429ba0f Update ref and ua databases 2021-08-23 11:41:43 +03:00
Uku Taht
62c5bfcbf8 Use unless instead of if 2021-08-13 17:21:37 +03:00
Uku Taht
7e78157c4b
Merge branch 'master' into stats-module 2021-08-04 16:11:40 +03:00
Uku Taht
6758931431 Add plug for source favicons 2021-08-04 12:01:50 +03:00
Uku Taht
13d0dce901 Make invitations email case insensitive 2021-07-26 12:08:35 +03:00
Uku Taht
0b4c430bb8 Cascade site deletion to invitations 2021-06-29 15:45:24 +03:00
Uku Taht
7c9ca12a3e Don't use URL in compat mode 2021-06-17 15:58:54 +03:00
Uku Taht
a378935aac Add IE11 compatibility mode 2021-06-17 10:55:45 +03:00
Uku Taht
e71de6dc1f
Invitations (#1122)
* Invite existing user to a site

* Add invitation flow for non-existing users

* Accept and reject invitations

* Use invitation flow for existing users

* Locking mechanism for sites

* Authorization for site settings

* Show usage based on site ownership

* Add ability to remove members from a site

* Do not show settings link to viewer roles

* Ability to remove invitations

* Remove `Plausible.Sites.count_for/1`

* Fix tests

* Do not show the trial banner after the trial

* Correct trial emails

* Transfer ownership

* Send invitation email to existing user

* Add invitation email flows

* Add plug for role-based authorization

* Rename AuthorizeStatsPlug -> AuthorizeSiteAccess

* Add email flow for ownership transfer

* Fix URLs in emails

* Fix small copy issues

* Make 'People' its own section in site settings

* Notify user via email if their access has been removed

* Check site lock status when invitation is accepted

* Check lock status when user subscribes

* Make sure only admins and owners can create shared links

* Changelog

* Add LockSites to daily cron

* Clean invitations after 48 hours

* Add notices about expiry

* Add invitation expired page

* Add doc link
2021-06-16 15:00:07 +03:00
Uku Taht
2be99db54d Fix p.js 2021-05-28 13:15:00 +03:00
Uku Taht
1a93542cd7 Add rate limit to API requests 2021-05-25 11:58:49 +03:00
Uku Taht
e80fc7bb8e
Tracker script improvements (#1048)
* Use `document.currentScript` to get script host

* Add data-api
2021-05-18 15:24:45 +03:00
Vignesh Joglekar
7f3e55418b
Changes from static tracker script name assignments to dynamic (#786)
Co-authored-by: Uku Taht <Uku.taht@gmail.com>
2021-05-18 11:36:12 +03:00
Uku Taht
f16666e374 Localize billing screens 2021-05-13 12:42:01 +03:00
Uku Taht
88efccef2f Localize the /upgrade page 2021-05-13 11:59:33 +03:00
Uku Taht
4935a62b22 Use JSON 2021-05-11 11:32:29 +03:00
Uku Taht
885b1d271a Refactor current plan data format to make changing it easier 2021-05-11 11:32:29 +03:00
Uku Taht
8d1039b8ce
Upgrades Oban to v2.6.1 (#967)
* Upgrades Oban to v2.6.1

* Use correct clickhouse_ecto
2021-04-26 11:32:18 +03:00
Uku Taht
6ca7425dc9
Send renewal notification for annual subscriptions (#949)
* Add background job to send renewal notification

* Use copy from Marko

* Add expiration email in case the user has cancelled

* Add queue config

* Use correct tag for expiration email
2021-04-21 15:57:38 +03:00
Uku Taht
d473accf40 Add API key scopes 2021-04-14 11:45:45 +03:00
Uku Taht
2fc136777c Add unique index to shared link name 2021-04-14 11:45:45 +03:00
Uku Taht
58cff47b6f
Add name to shared links (#910)
* Add name to shared links

* Add changelog entry
2021-04-06 14:32:38 +03:00
Uku Taht
3eb2f6d7a8
Add support for sampling (#875)
* Add support for sampling

* Use correct mix package

* Add notice about data sampling
2021-03-24 11:19:10 +02:00
Vignesh Joglekar
96633f46b1
Fixes exclude regex eating one character before single * (#861)
* Swaps to Negative Lookbehind for exclude Regex

* Updates to not use negative lookbehind
2021-03-22 15:44:53 +02:00
Uku Taht
a8cb187e8d
Track billing cycles (#697)
* Add dummy check usage worker

* Add and persist last_bill_date

* Remove name clash

* Correct timing and suggestions for over limit emails

* Fix tests for trial upgrade notifications
2021-02-12 10:17:53 +02:00
Uku Taht
5acb5b7039
Stats API (#679)
* WIP

* Add ability to filter by anything

* Add API keys

* Add version to api endpoint

* Fix API test route

* Fix API tests

* Allow 'date' parameter in '6mo' and '12mo'

* Rename session -> visit in API filters

* Filter expressions in the API

* Implement filters in aggregate call

* Add `compare` option to aggregate call

* Add way to manage API keys through the UI

* Authenticate with API key

* Use API key in tests
2021-02-05 11:23:30 +02:00
Uku Taht
bc44879bf8 Cascade site_membership deletion 2021-01-28 10:37:44 +02:00
Vignesh Joglekar
fb7a3fef89
Adds data-exclude support and localStorage.plausible_ignore support (#489)
* Adds data-exclude support and plausible_ignore support

* Splits exclusion into separate script option

* localStorage parsing upgrades

* Additional script type additions

I'm unsure about the formatting decision in tracker.ex - lmk.

* Adds new compiled files

This will certainly have conflicts with my other PRs related to the tracking scripts right now, I'll make one extra PR after both are done to ensure they're consolidated into the compiled scripts.

* Moves localStorage blocker out of special script

* Changelog

* Second thoughts on localStorage exclusion

* Updates `*` to not match `/` or whitespace

* Fix formatting

* Removes zero-length asterisks

* Adds support for double glob, zero-length replacements

* Update to reduce size+allow localStorage exclude

Co-authored-by: Uku Taht <Uku.taht@gmail.com>
2021-01-22 13:32:40 +02:00
Uku Taht
a4bf74a84c Do not allow duplicate spike notification to be created 2021-01-19 11:41:15 +02:00
Uku Taht
b7a2b3b423 Cascade deletion of site to spike notifications 2021-01-15 11:24:28 +02:00
Uku Taht
9bbaa756b8 Fix user deletion 2020-12-30 11:00:37 +02:00
Uku Taht
81c12884cd
Add elixir action (#526)
* Add elixir action

* Format the codebase

* Add postgresql

* Postgres config

* Run postgres on localhost

* Add clickhouse to CI
2020-12-29 15:17:27 +02:00
Vignesh Joglekar
425975efec
Adds dark mode to entire dashboard (#467)
* Adds New Dark Mode Assets

* Moves triangle for dropdown to a reasonable position

* Majority .eex dark implementation

* Fixes Logo Positioning

* Adds theme flag to user schema, uses it

* Uses correct variables for theme applicator script

* Minor missed theme changes/fallbacks

* Individual Component Support + Theme Context

* Sources Tab Support

This was a pain to test D:

* Partial Stats Sections Support

* More of stats modules supported

* Modal +table support

* Improves some Flatpickr in light theme, supports dark theme

* Fixes missed settings tab colors

* Finishes Devices module support

* Fixes bar graph colors

* Better colorizes maps module

* Undoes colorized bars

(they looked bad, on second thought)

* Fixes loading indicator

* Finishes conversions module

* Adds changelog entry

The PR number could be wrong, will double check

* Fixes missed header color

* Fixes naming of migration and removes static alter

* Does migration correctly

As I said, my Elixir is pretty weak heh

* Adds support for spike notifications setting

* Improves contrast and visibility for email settings

* Resolves @ukutaht's comments on #467

* Fixes missing dark style

* Found one more missed dark element (shared links)

* Formatting fixes
2020-12-16 11:57:28 +02:00
Uku Taht
aa7ae87811
Onboarding UX improvements (#441)
* WIP

* Actually activate the user

* Send email verification codes

* Send activation code with email

* Only show onboarding steps during first site creation

* Add worker to config

* Consistent form styles

* Send welcome email when user activates account

* Add changelog entry

* Use https in new site form

* Correct spelling in email
2020-12-15 11:30:45 +02:00
Uku Taht
54f70717c4 Add basic spike notifications
Co-authored-by: Jerod Santo <jerod@changelog.com>
2020-12-11 17:03:25 +02:00
Uku Taht
b724def948 Only register outbound link click if it has a host property
Fixes #398
2020-11-18 15:46:20 +02:00
Uku Taht
392b0b33ee Make outbound link tracking dynamic 2020-11-18 15:43:45 +02:00
Uku Taht
0a530a0f66 Do not use pushState and popState in hash mode 2020-11-18 10:50:59 +02:00
Uku Taht
1b21a26e78 Ignore hash clicks 2020-11-17 15:06:53 +02:00
Uku Taht
d0d7b823f8
Browser and OS version (#397)
* Collect browser and OS version

* Display browser version

* Show operating system versions

* Device categorization

* Treat headless chrome like a bot

* Ignore events from automated browsers

* Only take major and minor of the version

* Add tests

* Add CHANGELOG entry

* Add changelog entry for bots

* Store empty value as browser when unknown
2020-11-10 15:18:59 +02:00
Uku Taht
f0cbf33d7c
Add tracker feature to automatically track outbound links (#389)
* Add tracker feature to automatically track outbound links

* Precompute templates and allow aliases

* Fix /js/analytics.js

* Remove change to tracking code

* Update CHANGELOG
2020-11-03 11:09:50 +02:00
Uku Taht
0b6e645b44
Rename metadata -> properties (#390) 2020-10-30 10:49:41 +02:00
Uku Taht
40900c7653
Ability to add event metadata (#381)
* Ability to add event metadata

* Close Dropdown on outside click

* Show (none) value in metadata breakdown

* Allow filtering for metadata key/val pairs

* Use correct clickhouse_ecto

* Better naming for meta filter

* Add tests

* Add changelog entry

* Remove change made for testing
2020-10-28 11:09:04 +02:00
Uku Taht
6e8ccdd473 Download updated user-agent files
Closes #309
2020-10-14 12:26:43 +03:00
Uku Taht
122dd94327
Store utm_medium, utm_source, and utm_campaign (#318) 2020-09-18 11:37:32 +03:00
Uku Taht
4b36bb7138
Use clickhouse_ecto for db connection (#317)
* Use clickhouse-ecto for stats

* Use clickhouse ecto instead of low-level clickhousex

* Remove defaults from event schema

* Remove all references to Clickhousex

* Document configuration change

* Ensure createdb and migrations can be run in a release

* Remove config added for debug

* Update plausible_variables.sample.env
2020-09-17 16:36:01 +03:00
Uku Taht
3a99b8a4fb
Handle page visibility change when preloading (#311) 2020-09-09 10:15:34 +03:00
Uku Taht
4aa4dfdcaf
Hash mode (#299)
* Build tracker with hash mode

* Extract hash fragment in hash mode

* Serve new hash-based tracker
2020-08-25 10:56:36 +03:00
Uku Taht
d30f17c807
Extract utm_source on the server side (#298)
* Extract utm_source on the server side

* Build tracker scripts
2020-08-24 16:32:24 +03:00
Uku Taht
1519359bbf
Makes the tracker and its payload smaller (#296) 2020-08-24 15:16:07 +03:00
Uku Taht
4d55c4657a
Rotate salts on a daily basis (#224)
* Add daily salt rotation for user ids

* Rotate salts on a cron schedule daily

* Keep old salt fo 48 hours

* Double write for plausible.io to test salt rotation

* Rotate salts for every site
2020-07-15 11:47:24 +03:00
Uku Taht
bb36bae15c
Inject base url to tracker at runtime (#78)
* Inject base url to tracker at runtime

* Cache tracker script for 1 hour
2020-06-11 09:30:00 +03:00
Chandra Tungathurthi
b4b7532f07
Formatting only changes - No code change (#75)
* first commit with test and compile job

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding 'prepare' stage

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated ci script to include "test" compile phase

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding environment variables for connecting to postgresql

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated ci config for postgres

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* using non-alpine version of elixir

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* re-using the 'compile' artifacts and added explict env variables for testing

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removing redundant deps fetching from common code

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* formatting using mix.format -- beware no-code changes!

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* added release config

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding consistent env variable for Database

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* more cleaning up of environment variables

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding releases config for enabling releases

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* cleaning up env configs

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Cleaned up config and prepared config for releases

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated CI script with new config for test

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added Dockerfile for creating production docker image

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding "docker" build job yay!

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* using non-slim version of debian and installing webpack

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding overlays for migrations on releases

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* restricting the docker built to master branch only

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* typo fix

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding "Hosting.md" to explain hosting instructions

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removed the default comments

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added documentation related to env variables

* updated documentation and fixed typo

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated documentation

* Bumping up elixir version as `overlays` are only supported in latest version

read release notes: https://github.com/elixir-lang/elixir/releases/tag/v1.10.0

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding tarball assembly during release

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated HOSTING.md

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added support for db migration

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* minor corrections

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* initializing admin user

Admin user has been added in the "migration" phase. A default user is automatically created in the process. One can provide the related env variables, else a new one will be automatically created for you.

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Initial base domain update - phase#1

These changes are only meant for correct operating it under self-hosting. There are many other cosmetic changes, that require updates to email, site and other places where the original website and author is used.

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Using dedicated config variable `base_domain` instead

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding base_domain to releases config

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removing the dedicated config "base_domain", relying on endpoint host

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Removed the usage of "Mix" in code!

It is bad practice to use "mix" module inside the code as in actual release this module is unavailable. Replacing this with a config environment variable

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added support for SMTP via Bamboo Smtp Adapter

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Capturing SMTP errors via Sentry

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Minor updates

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding junit formatter -- useful for generating test reports

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding documentation for default user

* Resolve "Gitlab Adoption: Add supported services in "Security & Compliance""

* bumping up the debian version to fix issues

fixing some vulnerabilities identified by the scanning tools

* More updates for self-hosting

Changes in most of the places to suit self-hosting. Although, there are some which have been left-off.

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* quick-dirty-fix!

* bumping up the db connect timeout

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* bumping up the db connect timeout

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* bumping up the db connect timeout

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* bumping up timeout - skipping MRs :-/

* removing restrictions on watching for changes

this stuff isn't working

* Update HOSTING.md

* renamed the module name

* reverting formatting-whitespace changes

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* reverting the name to release

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding docker-compose.yml and related instructions

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* using `plausible_url` instead of assuming `https`

this is because, it is much to test in local dev machines and in most cases there's already a layer above which is capable for `https` termination and http -> https upgrade

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* WIP: merging changes from upstream

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* wip: more changes

* Pushing in changes from upstream

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* changes to ci for testing

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* cleaning up and finishing clickhouse integration

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updating readme with hosting details

* removing deleted files from upstream

* minor config adjustments

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* formatting changes

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>
2020-06-08 10:35:13 +03:00
Uku Taht
46f94df421 Remove fingerprint sessions table 2020-06-05 17:27:59 +03:00
Uku Taht
6eeeb52f03 Remove events and sessions table from postgres 2020-06-05 16:47:23 +03:00
Uku Taht
933cff6fe0
Remove stats from postgres (#74)
* Remove stats code related to postgres events and sessions

* Use Clickhouse events in fetch tweets worker
2020-06-05 16:14:17 +03:00
Uku Taht
25d8f2f3f9
Schedule regular jobs with Oban (#69)
* Use oban for work

* Remove mix task for email reports

* Convert remaining tasks to Oban workers

* Add crontab for scheduled jobs

* Only enable cron if configured to do so
2020-06-02 13:37:38 +03:00
Chandra Tungathurthi
5eb8929929
Support for docker based self-hosting (#64)
* first commit with test and compile job

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding 'prepare' stage

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated ci script to include "test" compile phase

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding environment variables for connecting to postgresql

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated ci config for postgres

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* using non-alpine version of elixir

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* re-using the 'compile' artifacts and added explict env variables for testing

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removing redundant deps fetching from common code

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* formatting using mix.format -- beware no-code changes!

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* added release config

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding consistent env variable for Database

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* more cleaning up of environment variables

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding releases config for enabling releases

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* cleaning up env configs

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Cleaned up config and prepared config for releases

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated CI script with new config for test

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added Dockerfile for creating production docker image

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding "docker" build job yay!

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* using non-slim version of debian and installing webpack

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding overlays for migrations on releases

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* restricting the docker built to master branch only

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* typo fix

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding "Hosting.md" to explain hosting instructions

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removed the default comments

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added documentation related to env variables

* updated documentation and fixed typo

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated documentation

* Bumping up elixir version as `overlays` are only supported in latest version

read release notes: https://github.com/elixir-lang/elixir/releases/tag/v1.10.0

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding tarball assembly during release

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated HOSTING.md

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added support for db migration

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* minor corrections

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* initializing admin user

Admin user has been added in the "migration" phase. A default user is automatically created in the process. One can provide the related env variables, else a new one will be automatically created for you.

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Initial base domain update - phase#1

These changes are only meant for correct operating it under self-hosting. There are many other cosmetic changes, that require updates to email, site and other places where the original website and author is used.

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Using dedicated config variable `base_domain` instead

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding base_domain to releases config

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removing the dedicated config "base_domain", relying on endpoint host

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Removed the usage of "Mix" in code!

It is bad practice to use "mix" module inside the code as in actual release this module is unavailable. Replacing this with a config environment variable

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added support for SMTP via Bamboo Smtp Adapter

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Capturing SMTP errors via Sentry

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Minor updates

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding junit formatter -- useful for generating test reports

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding documentation for default user

* Resolve "Gitlab Adoption: Add supported services in "Security & Compliance""

* bumping up the debian version to fix issues

fixing some vulnerabilities identified by the scanning tools

* More updates for self-hosting

Changes in most of the places to suit self-hosting. Although, there are some which have been left-off.

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* quick-dirty-fix!

* bumping up the db connect timeout

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* bumping up the db connect timeout

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* bumping up the db connect timeout

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* bumping up timeout - skipping MRs :-/

* removing restrictions on watching for changes

this stuff isn't working

* Update HOSTING.md

* renamed the module name

* reverting formatting-whitespace changes

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* reverting the name to release

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding docker-compose.yml and related instructions

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* using `plausible_url` instead of assuming `https`

this is because, it is much to test in local dev machines and in most cases there's already a layer above which is capable for `https` termination and http -> https upgrade

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* WIP: merging changes from upstream

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* wip: more changes

* Pushing in changes from upstream

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* changes to ci for testing

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* cleaning up and finishing clickhouse integration

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updating readme with hosting details
2020-05-26 16:09:34 +03:00
Uku Taht
4ff528d246 Cascade setup success emails deletion 2020-04-08 15:25:35 +03:00
Uku Taht
ddf08b97e8 Cascade custom domain deletion 2020-04-06 14:53:31 +03:00
Uku Taht
7755b8b188 Make sure pageviews can be inserted 2020-03-24 15:29:35 +02:00
Uku Taht
594b085467
Analytics without using cookies (#44)
* Use fingerprints instead of user_id

* Fix tests
2020-03-24 10:50:16 +02:00
Uku Taht
ee92ed5213
Email flows (#46)
* Update activation email

* Update trial upgrade emails

* Remove feedback emails

* Remove feedback email test

* Update welcome email (and send it earlier)

* Site setup emails

* Add create site emails for trialers

* Add check stats email

* Remove dead code

* Adjust site setup emails for users who are not on trial

* BCC myself on all new emails

* Use old phrasing in the welcome email
2020-03-23 11:34:25 +02:00
Uku Taht
3fc17a51d5
Add field to track trial expiry date (#45) 2020-03-18 16:27:46 +02:00
Uku Taht
a79a4c8076 Make fingerprint referrer flexible 2020-03-02 12:57:51 +02:00
Uku Taht
aaa9c67e47
Add fingerprint sessions (#36) 2020-02-27 11:46:48 +02:00
Uku Taht
79b9f72b52
Custom domains (#34)
* UI to create custom domains

* Only call ssh once per domain

* Update copy for custom domain setup

* Use correct user for ssh
2020-02-26 10:54:21 +02:00
Uku Taht
ed3110a6ea
Introduce initial referrer and initial referrer source (#32)
* Introduce initial referrer and initial referrer source

* Add test
2020-02-12 11:11:02 +02:00
Uku Taht
7bb5d98956 Remove raw fingerprint 2020-02-11 11:02:44 +02:00
Uku Taht
16b2dbf023 Store raw fingerprint for testing 2020-02-11 10:10:53 +02:00
Uku Taht
0875d57b59 Track fingerprint 2020-02-10 16:00:19 +02:00
Uku Taht
7dbbc8ba22
Configurable site id (#30)
* Use site id instead of hostname for events

* Use site id in domain status check

* Revert change to tracking module

* Catch more places where link generation needed updating

* Rename site_id to domain

* Drop hostname index from events
2020-02-04 15:44:13 +02:00
Uku Taht
5f62025176
Shared links (#29)
* Create shared links UI

* Show shared links in a list on settings page

* Show dropdown for each shared link

* Show icon actions for shared links

* Log user in when they click non-password-protected shared link

* Actually authenticate using a password

* Delete shared links
2020-01-29 11:29:11 +02:00
Uku Taht
c96f364ad8
Add and remove recipients for email reports (#28)
* Add and remove recipients for email reports

* Remove unused google_settings controller action

* Background job sends email reports to multiple recipients

* Add a way to unsubscribe for recipients who cannot log in

* Fix view on plausible link

* Include bounce rate in email report
2020-01-22 11:16:53 +02:00
Uku Taht
4d2f2cc1bd Change session referrer to text 2020-01-20 11:12:13 +02:00
Uku Taht
b02cb74181
Fetch and display tweets (#27) 2020-01-16 13:39:47 +02:00
Uku Taht
94a20fb0a2
Add exit page to sessions (#25) 2020-01-15 11:00:42 +02:00
Uku Taht
29ae3a2c21 Show bounce rate for referrers and pages 2020-01-07 14:53:04 +02:00
Uku Taht
438a1c3d19 Cascade google auth deletion 2020-01-06 11:08:36 +02:00
Uku Taht
b7815bfe25 Revert "Cascade site google auth deletion"
This reverts commit fd8917d4f3.
2020-01-06 11:04:28 +02:00
Uku Taht
fd8917d4f3 Cascade site google auth deletion 2020-01-06 11:01:20 +02:00
Uku Taht
3548c57710 Capture session start 2019-12-20 12:30:29 +08:00
Uku Taht
66e9d197eb Correct migration order 2019-12-18 16:07:50 +08:00
Uku Taht
f7d752988a Add sessions 2019-12-18 15:23:53 +08:00
Uku Taht
4b29735c0c Add unique index to monthly reports site id 2019-12-16 14:53:28 +08:00
Uku Taht
10840f9e10 Create free subscriptions 2019-11-18 16:13:54 +08:00
Uku Taht
0b96ed3b6c Remove goal name 2019-10-31 14:36:16 +08:00
Uku Taht
c7ed0d54ce Add goals and conversions 2019-10-31 13:39:51 +08:00
Uku Taht
edca2a57df Remove historical migrations that stop test database from being created 2019-10-25 14:34:54 +08:00
Uku Taht
3ea64de42b Add name to events 2019-10-25 14:06:11 +08:00
Uku Taht
0c453be08c Rename pageviews to events 2019-10-24 14:58:17 +08:00
Uku Taht
7c7bdc6ef6 Proper timestamp 2019-10-15 15:37:55 +08:00
Uku Taht
71edf2c86d Remove unused fields from pageviews 2019-10-15 15:34:41 +08:00
Uku Taht
731407f7b2 Configure Google Search Console property 2019-10-10 12:12:15 +08:00
Uku Taht
17e4dc96ef Rename email settings to weekly report 2019-09-09 12:19:21 +01:00
Uku Taht
7b8b48982b Customize email address for email reports 2019-09-10 16:18:17 +01:00
Uku Taht
cc75974821 Allow user to change email settings from settings page 2019-09-07 15:01:37 +01:00
Uku Taht
f3dd5d696e Add weekly email report template 2019-09-02 13:11:37 +01:00
Uku Taht
779d64e19a Initial commit 2019-09-02 12:29:19 +01:00