Commit Graph

352 Commits

Author SHA1 Message Date
Uku Taht
6eeeb52f03 Remove events and sessions table from postgres 2020-06-05 16:47:23 +03:00
Uku Taht
389f10d254 Remove CH hydration task 2020-06-05 16:47:14 +03:00
Uku Taht
933cff6fe0
Remove stats from postgres (#74)
* Remove stats code related to postgres events and sessions

* Use Clickhouse events in fetch tweets worker
2020-06-05 16:14:17 +03:00
Uku Taht
390a4577b5 Fetch config options at runtime 2020-06-05 11:15:26 +03:00
Uku Taht
6b8cfbe676 Use password for sshh 2020-06-04 20:25:13 +03:00
Uku Taht
1efc68fa59 Fix ssh test 2020-06-04 18:47:32 +03:00
Uku Taht
4793893eaf Use ssh in app directory 2020-06-04 16:51:12 +03:00
Uku Taht
06b70f765d Add user_dir to ssl command 2020-06-04 16:39:04 +03:00
Uku Taht
34d7d12229 Use SSH in Elixir for custom domains 2020-06-04 14:47:30 +03:00
Uku Taht
2a7e77ab10 Add Oban config to releases 2020-06-04 11:23:30 +03:00
Uku Taht
db7e2f1b76 Enable sentry in staging and prod 2020-06-04 11:00:07 +03:00
Uku Taht
de771897f9 Buld and push docker image to dockerhub 2020-06-03 14:18:14 +03:00
Uku Taht
69e9842786 Remove devDependencies 2020-06-03 10:15:53 +03:00
Uku Taht
baa5738ae4 Use npm bin for executables 2020-06-03 10:04:05 +03:00
Uku Taht
a817ba5270 Update npm deps 2020-06-03 09:25:16 +03:00
Uku Taht
761072d285 Fix bug with has_pageviews? 2020-06-02 16:13:12 +03:00
Uku Taht
8e9c938a9e Fix URL encoding in devices report 2020-06-02 14:39:21 +03:00
Uku Taht
8f0e52e3e5 Set Oban queues to execute 2020-06-02 14:26:58 +03:00
Uku Taht
ad1a39dbde Fix shared link 2020-06-02 14:17:28 +03:00
Uku Taht
29a6c49922 Fix crontab 2020-06-02 14:04:01 +03:00
Uku Taht
c7ddb7d339 Simplify environment variable in SEntry config 2020-06-02 13:53:34 +03:00
Uku Taht
e7f3455f2d Fix environment config variablt 2020-06-02 13:50:28 +03:00
Uku Taht
25d8f2f3f9
Schedule regular jobs with Oban (#69)
* Use oban for work

* Remove mix task for email reports

* Convert remaining tasks to Oban workers

* Add crontab for scheduled jobs

* Only enable cron if configured to do so
2020-06-02 13:37:38 +03:00
Liz
262e74c0a9
Fixed small typo. (#70) 2020-05-31 13:36:56 +03:00
Uku Taht
ad83c99647 Update elixir version in Travis 2020-05-29 13:20:24 +03:00
Uku Taht
4bd12f03fc
Reduce script size (#68)
* Remove cookie code

* Compile tracker separetly from website assets

* Install uglify

* Help minifier reduce varaibles
2020-05-29 09:51:16 +03:00
Uku Taht
01bc5911d1 Remove APP_ENV completely, use ENVIRONMENT instead 2020-05-28 21:31:54 +03:00
Uku Taht
9f446c83fd Use ENVIRONMENT for sentry in runtime config 2020-05-28 21:31:38 +03:00
Uku Taht
5d0fec2016 Use Bamboo local adapter by default 2020-05-28 10:50:40 +03:00
Uku Taht
7916e6f44e URL generation and configuration 2020-05-28 10:41:09 +03:00
Uku Taht
3b3d4d41b0 Configure SSL for database 2020-05-27 16:03:30 +03:00
Uku Taht
ecd30e2892 Bump elixir version 2020-05-27 14:11:42 +03:00
Chandra Tungathurthi
5eb8929929
Support for docker based self-hosting (#64)
* first commit with test and compile job

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding 'prepare' stage

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated ci script to include "test" compile phase

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding environment variables for connecting to postgresql

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated ci config for postgres

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* using non-alpine version of elixir

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* re-using the 'compile' artifacts and added explict env variables for testing

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removing redundant deps fetching from common code

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* formatting using mix.format -- beware no-code changes!

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* added release config

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding consistent env variable for Database

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* more cleaning up of environment variables

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding releases config for enabling releases

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* cleaning up env configs

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Cleaned up config and prepared config for releases

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated CI script with new config for test

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added Dockerfile for creating production docker image

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding "docker" build job yay!

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* using non-slim version of debian and installing webpack

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding overlays for migrations on releases

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* restricting the docker built to master branch only

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* typo fix

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding "Hosting.md" to explain hosting instructions

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removed the default comments

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added documentation related to env variables

* updated documentation and fixed typo

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated documentation

* Bumping up elixir version as `overlays` are only supported in latest version

read release notes: https://github.com/elixir-lang/elixir/releases/tag/v1.10.0

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding tarball assembly during release

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updated HOSTING.md

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added support for db migration

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* minor corrections

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* initializing admin user

Admin user has been added in the "migration" phase. A default user is automatically created in the process. One can provide the related env variables, else a new one will be automatically created for you.

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Initial base domain update - phase#1

These changes are only meant for correct operating it under self-hosting. There are many other cosmetic changes, that require updates to email, site and other places where the original website and author is used.

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Using dedicated config variable `base_domain` instead

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding base_domain to releases config

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* removing the dedicated config "base_domain", relying on endpoint host

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Removed the usage of "Mix" in code!

It is bad practice to use "mix" module inside the code as in actual release this module is unavailable. Replacing this with a config environment variable

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Added support for SMTP via Bamboo Smtp Adapter

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Capturing SMTP errors via Sentry

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Minor updates

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* Adding junit formatter -- useful for generating test reports

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding documentation for default user

* Resolve "Gitlab Adoption: Add supported services in "Security & Compliance""

* bumping up the debian version to fix issues

fixing some vulnerabilities identified by the scanning tools

* More updates for self-hosting

Changes in most of the places to suit self-hosting. Although, there are some which have been left-off.

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* quick-dirty-fix!

* bumping up the db connect timeout

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* bumping up the db connect timeout

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* bumping up the db connect timeout

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* bumping up timeout - skipping MRs :-/

* removing restrictions on watching for changes

this stuff isn't working

* Update HOSTING.md

* renamed the module name

* reverting formatting-whitespace changes

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* reverting the name to release

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* adding docker-compose.yml and related instructions

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* using `plausible_url` instead of assuming `https`

this is because, it is much to test in local dev machines and in most cases there's already a layer above which is capable for `https` termination and http -> https upgrade

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* WIP: merging changes from upstream

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* wip: more changes

* Pushing in changes from upstream

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* changes to ci for testing

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* cleaning up and finishing clickhouse integration

Signed-off-by: Chandra Tungathurthi <tckb@tgrthi.me>

* updating readme with hosting details
2020-05-26 16:09:34 +03:00
Uku Taht
89e807b469 Handle errors better 2020-05-26 13:40:49 +03:00
Uku Taht
34de4d5d2e Log any error from clickhouse 2020-05-26 12:35:03 +03:00
Uku Taht
43b46e7607 Show last touch referrer for drilldown 2020-05-25 14:52:20 +03:00
Uku Taht
bda08df7b5 Get goal referrers from sessions table 2020-05-25 14:29:41 +03:00
Uku Taht
a228c6162f Default to empty pathname 2020-05-25 12:25:41 +03:00
Uku Taht
0524ac92c2 Fix error loggin in Clickhouse 2020-05-25 11:57:11 +03:00
Uku Taht
b31c0114c5 Use Clickhouse everywhere 2020-05-22 12:33:17 +03:00
Uku Taht
439a4a80c7 Escape quote in event name 2020-05-21 19:14:45 +03:00
Uku Taht
ee4f604712 Update hydrate command 2020-05-21 19:08:39 +03:00
Uku Taht
578523170f Query via clickhouse 2020-05-21 18:59:07 +03:00
Uku Taht
4800cc6cf7 Escape everything 2020-05-21 16:19:28 +03:00
Uku Taht
1087742fea Clean after itself 2020-05-21 15:55:46 +03:00
Uku Taht
d83ec7b6a1 Report progress 2020-05-21 14:53:09 +03:00
Uku Taht
a8c1fa2ea6 Escape quote in sessions 2020-05-21 14:30:04 +03:00
Uku Taht
ea53daeaac Hydrate historical data 2020-05-21 14:06:14 +03:00
Uku Taht
0ccb893c04 Ingest with new CH schema 2020-05-21 13:36:53 +03:00
Uku Taht
d94a8dbc7e
Clickhouse (#66)
* Get stats from clickhosue

* Pull stats from clickhouse

* Use correct Query namespace

* Use Clickhouse in unit tests

* Use Clickhouse in stats controller tests

* Use fixtures for unit tests

* Add Clickhouse to travis

* Use Clickhouse session store for sessions

* Add garbage collection to session store

* Reload session state from Clickhouse on server restart

* Query from sessions table

* Trap exits in event write buffer

* Run hydration without starting the whole app

* Make session length 30 minutes

* Revert changes to fingerprint schema

* Remove clickhouse from fingerprint sessions

* Flush buffers before shutdown

* Use old stats when merging

* Remove old session schema

* Fix tests with CH sessions

* Add has_pageviews? to Stats

* Use CH in staging

* Update schema

* Fix test setup
2020-05-21 13:03:39 +03:00