Commit Graph

193 Commits

Author SHA1 Message Date
Adrian Gruntkowski
51c1138d0d
Implement better user password validation (#3344)
* Add zxcvbn dependency

* Change password length range requirement from 6-64 to 12-128

* Reimplement register form in LV

* Implement server-side check for password strength

* Add rudimentary strength meter

* Make password input with strength a separate component and improve it

* Fix existing tests to provide strong enough password

* Apply formatting

* Replace existing registration form with new one

* Hide built-in label in `.input` component when none provided

* Crop password to first 32 chars for analysis by zxcvbn

* Add tests for new form components

* Integrate hCaptcha into LV

* Fix existing AuthController tests

* Add tests for Live.RegisterForm

* Hide strength meter when password input is empty

* Randomize client IP in headers during tests to avoid hitting rate limit

* Apply auxilliary formatting fixes to AuthController

* Integrate registration from invitation into LV registration logic

* Fix existing password set and reset forms

* Make `password_length_hint` component more customizable

* Optimize `Auth.User.set_password/2`

* Remove unnecessary attribute from registration form

* Move password set and reset forms to LV

* Add tests for SetPasswordForm LV component

* Add tests for password checks in `Auth.User`

* Document code a bit

* Implement simpler approach to hCaptcha integration

* Update CHANGELOG.md

* Improve consistency of color scheme

* Introduce debounce across all text inputs in registration and password forms

* Fix email input background in register form

* Ensure only single error is rendered for empty password confirmation case

* Remove `/password` form entirely in favor of preferred password reset

* Remove unnecessary `router` option from `live_render` calls

* Make expensive assigns in LV with `assign_new` (h/t @aerosol)

* Accept passwords longer than 32 bytes uniformly as very strong

* Avoid displaying blank error side by side with weak password error

* Make register actions handle errors gracefully

* Render only a single piece of feedback to reduce noise

* Make register and password reset forms pw manager friendly (h/t @cnkk)

* Move registration forms to live routes

* Delete no longer used deadviews

* Adjust registration form in accordance to changes in #3290

* Reintroduce dogfood page path for invitation form from #3290

* Use alternative approach to submitting plausible metrics from LV form

* Rename metrics events and extend tests to account for them
2023-09-25 10:27:29 +02:00
ruslandoga
c3910ae201
log mailer errors (#3336)
* log mailer errors

* update sentry to fix logger metadata bug

* update changelog
2023-09-21 13:06:28 +03:00
hq1
b3ff695797
Improve goal settings UX (#3293)
* Add Heroicons dependency

* Add name_of/1 html helper

Currently with Floki there's no way to query for
`[name=foo[some]]` selector

* Update changelog

* Make goal deletion possible with only goal id

* Remove stale goal controllers

* Improve ComboBox component

- make sure the list options are always of the parent input width
- allow passing a suggestion function instead of a module

* Stale fixup

* Update routes

* Use the new goals route in funnel settings

* Use a function in the funnel combo

* Use function in the props combo

* Remove old goals form

* Implement new goal settings

* Update moduledoc

* Fix revenue switch in dark mode

* Connect live socket on goal settings page

* Fixup

* Use Heroicons.trash icon

* Tweak goals search input

* Remove unused alias

* Fix search/button alignment

* Fix backspace icon alignment

* Delegate :superadmin check to get_for_user/3

I'll do props settings separately, it's work in progress
in a branch on top of this one already. cc @ukutaht

* Rename socket assigns

* Fixup to 5c9f58e

* Fixup

* Render ComboBox suggestions asynchronously

This commit:
  - prevents redundant work by checking the socket connection
  - allows passing no options to the ComboBox component,
    so that when combined with the `async` option, the options
    are asynchronously initialized post-render
  - allows updating the suggestions asynchronously with the
    `async` option set to `true` - helpful in case of DB
    queries used for suggestions

* Update tests

* Throttle comboboxes

* Update tests

* Dim the search input

* Use debounce=200 in ComboBox component

* Move creatable option to the top

* Ensure there's always a leading slash for goals

* Test pageview goals with leading / missing

* Make the modal scrollable on small viewports
2023-09-04 13:44:22 +02:00
hq1
20d31a8d0d
Bump ua_inspector/matomo db (#3263)
* Bump ua_inspector/matomo db

* Update changelog

* Bump matomo to master

* Pin matomo sha
2023-08-14 16:03:12 +02:00
dependabot[bot]
66bbef36bb
Bump timex from 3.7.8 to 3.7.11 (#3198)
Bumps [timex](https://github.com/bitwalker/timex) from 3.7.8 to 3.7.11.
- [Release notes](https://github.com/bitwalker/timex/releases)
- [Changelog](https://github.com/bitwalker/timex/blob/main/CHANGELOG.md)
- [Commits](https://github.com/bitwalker/timex/commits)

---
updated-dependencies:
- dependency-name: timex
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-27 14:27:41 +02:00
dependabot[bot]
eb3222f840
Bump phoenix_ecto from 4.4.0 to 4.4.2 (#3194)
Bumps [phoenix_ecto](https://github.com/phoenixframework/phoenix_ecto) from 4.4.0 to 4.4.2.
- [Changelog](https://github.com/phoenixframework/phoenix_ecto/blob/main/CHANGELOG.md)
- [Commits](https://github.com/phoenixframework/phoenix_ecto/compare/v4.4.0...v4.4.2)

---
updated-dependencies:
- dependency-name: phoenix_ecto
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-26 10:43:47 +02:00
dependabot[bot]
15daa0222f
Bump ex_doc from 0.29.4 to 0.30.3 (#3187)
Bumps [ex_doc](https://github.com/elixir-lang/ex_doc) from 0.29.4 to 0.30.3.
- [Changelog](https://github.com/elixir-lang/ex_doc/blob/main/CHANGELOG.md)
- [Commits](https://github.com/elixir-lang/ex_doc/compare/v0.29.4...v0.30.3)

---
updated-dependencies:
- dependency-name: ex_doc
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-25 09:57:12 +02:00
dependabot[bot]
79274480aa
Bump ex_money from 5.12.4 to 5.15.0 (#3177)
Bumps [ex_money](https://github.com/kipcole9/money) from 5.12.4 to 5.15.0.
- [Release notes](https://github.com/kipcole9/money/releases)
- [Changelog](https://github.com/kipcole9/money/blob/master/CHANGELOG.md)
- [Commits](https://github.com/kipcole9/money/compare/v5.12.4...v5.15.0)

---
updated-dependencies:
- dependency-name: ex_money
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Vini Brasil <vini@hey.com>
2023-07-24 10:31:09 +01:00
hq1
e6996ccfda
Fix weekly/monthly report e-mails (MJML) (#3150)
* Render weekly/monthly reports with MJML

* Update changelog

* Fix APP-2NA (#3119)
2023-07-17 16:59:01 +02:00
dependabot[bot]
19914112c4
Bump floki from 0.32.1 to 0.34.3 (#3135)
Bumps [floki](https://github.com/philss/floki) from 0.32.1 to 0.34.3.
- [Release notes](https://github.com/philss/floki/releases)
- [Changelog](https://github.com/philss/floki/blob/main/CHANGELOG.md)
- [Commits](https://github.com/philss/floki/compare/v0.32.1...v0.34.3)

---
updated-dependencies:
- dependency-name: floki
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-13 13:19:40 +01:00
dependabot[bot]
8058a06d2d
Bump phoenix_live_view from 0.19.0 to 0.19.4 (#3131)
Bumps [phoenix_live_view](https://github.com/phoenixframework/phoenix_live_view) from 0.19.0 to 0.19.4.
- [Changelog](https://github.com/phoenixframework/phoenix_live_view/blob/main/CHANGELOG.md)
- [Commits](https://github.com/phoenixframework/phoenix_live_view/compare/v0.19.0...v0.19.4)

---
updated-dependencies:
- dependency-name: phoenix_live_view
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-12 10:24:27 +02:00
dependabot[bot]
4615866c13
Bump phoenix_live_reload from 1.3.3 to 1.4.1 (#2849)
Bumps [phoenix_live_reload](https://github.com/phoenixframework/phoenix_live_reload) from 1.3.3 to 1.4.1.
- [Release notes](https://github.com/phoenixframework/phoenix_live_reload/releases)
- [Changelog](https://github.com/phoenixframework/phoenix_live_reload/blob/main/CHANGELOG.md)
- [Commits](https://github.com/phoenixframework/phoenix_live_reload/compare/v1.3.3...v1.4.1)

---
updated-dependencies:
- dependency-name: phoenix_live_reload
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-11 13:59:31 +02:00
ruslandoga
7e2627ae4c
array join (#3021) 2023-07-11 12:43:53 +02:00
Vini Brasil
e4d4f7d954
Revenue tracking: Ingestion and breakdown queries (#2957)
* Add revenue fields to ClickHouse events

This commit adds 4 fields to the ClickHouse events_v2 table:

* `revenue_source_amount` and `revenue_source_currency` store revenue in
  the original currency sent during ingestion

* `revenue_reporting_amount` and `revenue_reporting_currency` store
  revenue in a common currency to perform calculations, and this
  currency is defined by the user when setting up the goal

The type of amount fields is `Nullable(Decimal64(3))`. That covers all
fiat currencies and allows us to store huge amounts. Even though
ClickHouse does not suggest using `Nullable`, this is a good use case,
because otherwise additional work would have to be done to
differentiate missing values from real zeroes.

I ran a benchmark with the data pattern we expect in production, where
we have more missing values than real decimals. I created 100 million
records where 90% of decimals are missing. The difference between the
tables in storage is just 0.4Mb.

* Add revenue parameter to Events API

This commit adds support for sending revenue data in ingestion using the
`revenue` parameter - aliased to `$`.

* Add revenue parameter to mix send_pageview

* Add average and total revenue to breakdown queries
2023-06-12 18:29:17 +01:00
hq1
49ca6b859a
Prepare base for the funnel PRs merge (#2993)
* Update dependencies as per the funnels branch

Co-authored-by: Uku Taht <uku.taht@gmail.com>

* Mute sigil_E deprecation errors

* Use Phoenix.Component in PlausbileWeb

* Add tooltip to react/no-unknown-property whitelist

---------

Co-authored-by: Uku Taht <uku.taht@gmail.com>
Co-authored-by: Vinicius Brasil <vini@hey.com>
2023-06-01 15:26:23 +02:00
ruslandoga
447ed12155
use ecto_ch (#2890)
* use ecto_ch

* use Ch type in BoolUInt8

* update ecto_ch and ch to latest versions
2023-05-25 09:26:50 +02:00
Vini Brasil
10d9e3b083
Revenue tracking: Add currency field to goal creation (#2948)
* Add revenue goal option to goal creation

This commit adds a currency field to the goals form. Goals that have a
currency set are now revenue goals, and are cached with sites to later
be used during ingestion.

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>

* Enable feature flag in tests

---------

Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
2023-05-23 12:08:09 +02:00
RobertJoonas
d161c1be0b
Upgrade phoenix (#2902)
* upgrade phoenix

Co-authored-by: Vini Brasil <vini@hey.com>

* fix a test (flash message)

The flash message in focus.html.eex was not covered by any test. This
commit fixes also fixes that.

* change function name

* remove unnecessary formatter and format

* update CI cache

* fix dialyzer error

---------

Co-authored-by: Vini Brasil <vini@hey.com>
2023-05-09 11:51:35 +03:00
dependabot[bot]
ef73816bb5
Bump fun_with_flags_ui from 0.8.0 to 0.8.1 (#2837)
Bumps [fun_with_flags_ui](https://github.com/tompave/fun_with_flags_ui) from 0.8.0 to 0.8.1.
- [Release notes](https://github.com/tompave/fun_with_flags_ui/releases)
- [Changelog](https://github.com/tompave/fun_with_flags_ui/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tompave/fun_with_flags_ui/compare/v0.8.0...v0.8.1)

---
updated-dependencies:
- dependency-name: fun_with_flags_ui
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-13 12:42:26 +02:00
dependabot[bot]
2df7d25df3
Bump ex_doc from 0.28.5 to 0.29.4 (#2832)
Bumps [ex_doc](https://github.com/elixir-lang/ex_doc) from 0.28.5 to 0.29.4.
- [Release notes](https://github.com/elixir-lang/ex_doc/releases)
- [Changelog](https://github.com/elixir-lang/ex_doc/blob/main/CHANGELOG.md)
- [Commits](https://github.com/elixir-lang/ex_doc/compare/v0.28.5...v0.29.4)

---
updated-dependencies:
- dependency-name: ex_doc
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-07 11:12:45 -03:00
hq1
509c549c01
Update ua_inspector database (#2827)
* Update `ua_inspector` database

* Add changelog entry
2023-04-06 12:45:19 +02:00
dependabot[bot]
d069d0a164
Bump hammer from 6.0.0 to 6.1.0 (#2824)
Bumps [hammer](https://github.com/ExHammer/hammer) from 6.0.0 to 6.1.0.
- [Release notes](https://github.com/ExHammer/hammer/releases)
- [Changelog](https://github.com/ExHammer/hammer/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ExHammer/hammer/compare/v6.0.0...v6.1.0)

---
updated-dependencies:
- dependency-name: hammer
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-06 10:41:11 +02:00
dependabot[bot]
4edf126196
Bump bcrypt_elixir from 2.3.1 to 3.0.1 (#2391)
Bumps [bcrypt_elixir](https://github.com/riverrun/bcrypt_elixir) from 2.3.1 to 3.0.1.
- [Release notes](https://github.com/riverrun/bcrypt_elixir/releases)
- [Changelog](https://github.com/riverrun/bcrypt_elixir/blob/master/CHANGELOG.md)
- [Commits](https://github.com/riverrun/bcrypt_elixir/commits)

---
updated-dependencies:
- dependency-name: bcrypt_elixir
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-04-05 12:00:13 +02:00
ruslandoga
66be99b38a
update chto for missing source in telemetry fix (#2787) 2023-03-28 07:06:03 +02:00
hq1
e8509203a2
Update location dependency (faster startup) (#2793)
* Update location dependency (faster startup)

* Update with https://github.com/plausible/location/pull/6
2023-03-27 16:56:34 +02:00
hq1
d2f2c69387
Conditionally support switching between v1 and v2 clickhouse schemas (#2780)
* Remove ClickhouseSetup module

This has been an implicit point of contact to many
tests. From now on the goal is for each test to maintain
its own, isolated setup so that no accidental clashes
and implicit assumptions are relied upon.

* Implement v2 schema check

An environment variable V2_MIGRATION_DONE acts like
a feature flag, switching plausible from using old events/sessions
schemas to v2 schemas introduced by NumericIDs migration.

* Run both test suites sequentially

While the code for v1 and v2 schemas must be kept still,
we will from now on run tests against both code paths.
Secondary test run will set V2_MIGRATION_DONE=1 variable,
thus making all `Plausible.v2?()` checks return `true'.

* Remove unused function

This is a remnant from the short period when
we would check for existing events before allowing
creating a new site.

* Update test setups/factories with v2 migration check

* Make GateKeeper return site id along with :allow

* Make Billing module check for v2 schema

* Make ingestion aware of v2 schema

* Disable site transfers for when v2 is live

In a separate changeset we will implement simplified
site transfer for when v2 migration is complete.
The new transfer will only rename the site domain in postgres
and keep track of the original site prior to the transfer
so we keep an ingestion grace period until the customers
redeploy their scripting.

* Make Stats base queries aware of v2 schema switch

* Update breakdown with v2 conditionals

* Update pageview local start with v2 check

* Update current visitoris with v2 check

* Update stats controller with v2 checks

* Update external controller with v2 checks

* Update remaining tests with proper fixtures

* Rewrite redundant assignment

* Remove unused alias

* Mute credo, this is not the right time

* Add test_helper prompt

* Fetch priv dir so it works with a release

* Fetch distinct partitions only

* Don't limit inspect output for partitions

* Ensure SQL is printed to IO

* Remove redundant domain fixture
2023-03-27 13:52:42 +02:00
ruslandoga
a129bf82e0
use github: plausible/chto for chto (#2782)
* use github: plausible/chto for chto

* use chto with correct ch github reference (plausible/ch)
2023-03-22 14:56:57 +01:00
Adam
6d79ca5093
Switch to new clickhouse adapter (ch/chto) (#2733)
* another clickhouse adapter

* don't restore stats_removal.ex

* fix events main-graph error (#2746)

* update ch, chto

* update chto again (#2759)

* Stop treating page filter as an entry_page filter (#2752)

* remove dead code

* stop treating page filter as entry page filter in breakdown queries

* stop treating page filter as entry page filter in aggregate queries

* stop treating page filter as entry page filter in timeseries queries

* mix format

* update changelog

* break code down to smaller functions to keep credo happy

* remove unused functions

* make CSV export return only conversions with goal filter (#2760)

* make CSV export return only conversions with goal filter

* update changelog

* update elixir version in mix.exs (#2742)

* revert admin.ex changes (#2776)

---------

Co-authored-by: ruslandoga <67764432+ruslandoga@users.noreply.github.com>
Co-authored-by: ruslandoga <rusl@n-do.ga>
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
2023-03-21 09:55:59 +01:00
Adam
de0f68a647
Pin merged deps after #2693 (#2710) 2023-02-27 15:23:05 +01:00
Adam Rutkowski
867dad6da7
Implement ingest counters (#2693)
* Clickhouse migration: add ingest_counters table

* Configure ingest counters per MIX_ENV

* Emit telemetry for ingest events with rich metadata

* Allow building Request.t() with fake now() - for testing purposes

* Use clickhousex branch where session_id is assigned to each connection

* Add helper function for getting site id via cache

* Add Ecto schema for `ingest_counters` table

* Implement metrics buffer

* Implement buffering handler for `Plausible.Ingestion.Event` telemetry

* Implement periodic metrics aggregation

* Update counters docs

* Add toStartOfMinute() to ordering key

* Reset the sync connection state in `after` clause

* Flush counters on app termination

* Use separate Repo with async settings enabled at config level

* Switch to clickhouse_settings repo root config key

* Add AsyncInsertRepo module
2023-02-23 14:34:24 +01:00
Vini Brasil
1cb07efe6d
Save city name when importing from GA (#2608)
This commit adds city data to imported records from Google Analytics. The
current implementation sets city to 0 because GA does not use the GeoNames
database.

Google Analytics Reporting API uses [Geographical IDs](https://developers.google.com/analytics/devguides/collection/protocol/v1/geoid)
to identify cities and countries. Plausible uses
[GeoNames](https://geonames.org/) and I couldn't find databases corelating the
two.

Fortunately, GA also returns the city name and this commit uses the city name
and the country ISO code to find the Geoname ID. To avoid making expensive ETS
searches, I created another ETS table in the Location library that uses
{country, city} as a key.

Related PR: https://github.com/plausible/location/pull/3
2023-02-14 09:32:18 -03:00
Adam Rutkowski
912c9374d4
Bump clickhousex (per-query timeout override) (#2645)
https://github.com/plausible/clickhousex/pull/21
2023-02-06 12:30:39 +01:00
ruslandoga
166748dcf2
Replace Geolix with Locus (#2362)
This PR replaces geolix with locus to simplify self-hosted setup. locus can auto-update maxmind dbs which are recommended for self-hosters if they want city-level geolocation. locus is also a bit faster.

This PR also uses a test mmdb file from https://github.com/maxmind/MaxMind-DB for e2e geolocation tests without stubs.
2023-01-17 12:05:09 -03:00
Adam Rutkowski
97a3ce72e4
Update clickhousex & fingerprint sentry errors (#2587)
* Revert "Revert "Fingerprint Clickhousex.Error exceptions in Sentry (#2578)" (#2586)"

This reverts commit a78e07236a.

* Bump clickhousex to include https://github.com/plausible/clickhousex/pull/20
2023-01-12 17:22:45 +01:00
Adam Rutkowski
a78e07236a
Revert "Fingerprint Clickhousex.Error exceptions in Sentry (#2578)" (#2586)
This reverts commit 2428a2bf36.
2023-01-12 06:49:41 +01:00
Adam Rutkowski
2428a2bf36
Fingerprint Clickhousex.Error exceptions in Sentry (#2578)
* Fingerprint Clickhousex.Error exceptions in Sentry

* Update clickhousex
2023-01-11 15:19:07 +01:00
ruslandoga
a6e75ada7b
update ecto to include db url scheme (#2570)
* update ecto to include db url scheme

* update clickhousex

* update changelog
2023-01-10 13:02:28 +01:00
Adam Rutkowski
89da676215
Update clickhousex (start using Mint and proper pool management) (#2502)
* Pin clickhousex to @ruslandoga's rebase branch

* Bump lockfile

* Use HTTP pings

* Update lockfile

* Pin to master
2023-01-05 09:38:52 +01:00
Vini Brasil
96182da57d
Update bamboo_postmark to fork repository (#2477)
This commit updates mix.exs to resolve bamboo_postmark to our fork. The
fork encodes names with quotes when building e-mails, adding support for
special names with commas and quotes. Related to
plausible/bamboo_postmark#1.

Closes #1885
2022-11-29 17:48:44 +02:00
Adam Rutkowski
457a558471
Kick off sites by domain cache implementation (#2434)
* Implement sites by domain caching interface + warmer

* Add test

* Implement hit rate interface

* Add moduledocs

* Fix up typespec

* s/warmer/warmer_fn

* Extract measure_duration/2

* Fix up typespec

* Log errors and return nil on cache internal errors

* Fix up non-existing cache test

* Retrieve specific db columns when pre-filling the cache

* Reduce the subset of fields retrieved from the DB

See 63f3c6233d (r89871536)
2022-11-16 10:06:23 +01:00
dependabot[bot]
499b46f4c4
Bump bamboo_smtp from 4.1.0 to 4.2.2 (#2337)
Bumps [bamboo_smtp](https://github.com/fewlinesco/bamboo_smtp) from 4.1.0 to 4.2.2.
- [Release notes](https://github.com/fewlinesco/bamboo_smtp/releases)
- [Changelog](https://github.com/fewlinesco/bamboo_smtp/blob/main/CHANGELOG.md)
- [Commits](https://github.com/fewlinesco/bamboo_smtp/compare/v4.1.0...v4.2.2)

---
updated-dependencies:
- dependency-name: bamboo_smtp
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-26 17:59:36 -03:00
Vinicius Brasil
9220d0034d
OpenTelemetry (OTEL) Implementation (#2317)
This pull request improves the current OpenTelemetry implementation. Currently only 1% of the spans are sent, due to the high volume of ingestion requests to /api/event. I enabled the 1% sampling to /api/event only, recording 100% of the other traces.
2022-10-18 12:11:30 -03:00
dependabot[bot]
735ef94397
Bump csv from 2.4.1 to 2.5.0 (#2328)
Bumps csv from 2.4.1 to 2.5.0.

---
updated-dependencies:
- dependency-name: csv
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-13 14:11:28 +02:00
dependabot[bot]
97f6332a79
Bump httpoison from 1.8.1 to 1.8.2 (#2307)
Bumps [httpoison](https://github.com/edgurgel/httpoison) from 1.8.1 to 1.8.2.
- [Release notes](https://github.com/edgurgel/httpoison/releases)
- [Commits](https://github.com/edgurgel/httpoison/compare/v1.8.1...v1.8.2)

---
updated-dependencies:
- dependency-name: httpoison
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-13 08:02:14 +02:00
Uku Taht
aadf892a71
Increase DB query timeout for site transfers (#2309) 2022-10-06 14:17:35 +03:00
dependabot[bot]
24ed29fdee
Bump plug from 1.13.4 to 1.13.6 (#2301)
Bumps [plug](https://github.com/elixir-plug/plug) from 1.13.4 to 1.13.6.
- [Release notes](https://github.com/elixir-plug/plug/releases)
- [Changelog](https://github.com/elixir-plug/plug/blob/main/CHANGELOG.md)
- [Commits](https://github.com/elixir-plug/plug/compare/v1.13.4...v1.13.6)

---
updated-dependencies:
- dependency-name: plug
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-04 14:33:48 +02:00
dependabot[bot]
c079ff05c1
Bump credo from 1.6.4 to 1.6.7 (#2296)
Bumps [credo](https://github.com/rrrene/credo) from 1.6.4 to 1.6.7.
- [Release notes](https://github.com/rrrene/credo/releases)
- [Changelog](https://github.com/rrrene/credo/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rrrene/credo/compare/v1.6.4...v1.6.7)

---
updated-dependencies:
- dependency-name: credo
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-04 10:46:43 +03:00
Uku Taht
12a513f3cd
Remove Mimic dependency (#2280) 2022-10-03 11:41:51 -03:00
Uku Taht
e849e03058
Fix favicons (#2257) 2022-09-23 07:22:43 -03:00
dependabot[bot]
81a4e92b0a
Bump dialyxir from 1.1.0 to 1.2.0 (#2252)
Bumps [dialyxir](https://github.com/jeremyjh/dialyxir) from 1.1.0 to 1.2.0.
- [Release notes](https://github.com/jeremyjh/dialyxir/releases)
- [Changelog](https://github.com/jeremyjh/dialyxir/blob/master/CHANGELOG.md)
- [Commits](https://github.com/jeremyjh/dialyxir/compare/1.1.0...1.2.0)

---
updated-dependencies:
- dependency-name: dialyxir
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-22 17:25:47 -03:00
dependabot[bot]
5c0a74dc16
Bump fun_with_flags from 1.8.1 to 1.9.0 (#2241)
Bumps [fun_with_flags](https://github.com/tompave/fun_with_flags) from 1.8.1 to 1.9.0.
- [Release notes](https://github.com/tompave/fun_with_flags/releases)
- [Changelog](https://github.com/tompave/fun_with_flags/blob/master/CHANGELOG.md)
- [Commits](https://github.com/tompave/fun_with_flags/compare/v1.8.1...v1.9.0)

---
updated-dependencies:
- dependency-name: fun_with_flags
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-09-20 19:10:45 -03:00
Vinicius Brasil
54260c294f
Publish Elixir documentation to GitHub Pages (#2231)
* Install and setup ex_doc

* Create GitHub Action to publish docs
2022-09-16 10:56:35 +03:00
Vinicius Brasil
178e34b9e7
Fix timezone input (#2222)
* Update Timex version from 3.7.7 to 3.7.8

* Generate timezone list from Tzdata

This commit fixes a bug where timezone changes weren't updating the
timezone list displayed when editing or creating a site.

Timezones were being pulled from a static list. This commit changes it
to generate the list from Tzdata, that uses a timezone database with
updated information on time changes. Additionally it adds more timezones
with aliases and links to the list.

Closes #1340

* Use timezone name from browser to recommend timezone

This commit matches the timezone name instead of offset to recommend a
timezone when creating a new site. The JavaScript Intl.DateTimeFormat
API is widely supported according to the link. In any case, if the
timezone fails to match by name, it fallbacks to the offset strategy.

https://caniuse.com/mdn-javascript_builtins_intl_datetimeformat_resolvedoptions_computed_timezone

Closes #904
2022-09-14 09:58:42 -03:00
Adam Rutkowski
e2aa519c31
Test HTTPClient (#2185)
* Sort dependencies

* Add :bypass dependency

* Test HTTPClient

* Rename test module
2022-09-07 14:17:57 +02:00
RobertJoonas
e8d197f434
updated to most recent referrer blocklist commit (#2146) 2022-08-29 11:25:42 +03:00
Vinicius Brasil
4b9032d822
Google Analytics Import Refactor (#2046)
* Create separate module for GA HTTP requests

* Fetch GA data entirely instead of monthly

* Add buffering to GA imports

* Change positional args to maps when serializing from GA

* Create Google Analytics VCR tests
2022-08-03 12:25:50 +03:00
Adam Rutkowski
3b82ba0e25
Upgrade to Geolix 2.0 (#1997)
* Upgrade geolix

* Remove geolix pool config

* Save unnecessary Task.async_stream roundtrip

Normally the Geolix API accepts `:where` keyword option that designates
the database to look up. In case no parameter is supplied, it'll spawn
a parallel map over all databases available. In this case we have only
one DB anyway, so there is no need for the extra instrumentation.

* Follow up on direct :geolocation lookups
2022-07-12 11:39:04 +03:00
Manu S Ajith
81f18ff0a5
Setup promex (#1999)
* Setup promex

Signed-off-by: Manu S Ajith <neo@codingarena.in>

* Cleanup promex config file

Signed-off-by: Manu S Ajith <neo@codingarena.in>
2022-07-11 15:00:04 +03:00
Uku Taht
2b8e3ea62a
Use finch in sentry client (#1996)
* Introduce Finch for Sentry integration

* Make sure the DummyAgent can be started

* No need to sanitize the dsn, finch takes care of that

* Simplify the dummy child spec

* Annotate redirects clause

* Make use of new `get_int_from_path_or_env`

* Actually use finch in Sentry config

* Configure `excluded_domains` correctly for Sentry

The way sentry is configured currently, when we get an HTTP error it
will be logged twice - once from Sentry.PlugCapture and once from
Sentry.LoggerBackend. The logger backend module does the right thing
by default but for some reason we've been overriding the config
parameter that by default stops double-counting errors. This commit
returns to the default configuration which is better.

* Default to 15s timeout

* Attempt to send twice at most

* Warn in sentry client

* Use warn level in sentry client

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2022-07-08 11:14:52 +03:00
Uku Taht
0553fa041b Parse geolix pool config as integers 2022-07-07 11:38:18 +03:00
Uku Taht
44e4b429ae Update clickhouse_ecto 2022-07-07 11:13:56 +03:00
Uku Taht
292a419473
Add feature flag to block event ingest (#1991)
Event ingest can be blocked using the flag `block_event_ingest`
2022-07-06 12:25:40 +03:00
Uku Taht
6cc9afff50 Add observer_cli 2022-07-05 15:42:59 +03:00
Uku Taht
bbcd40b352 Upgrade opentelemetry_ecto 2022-05-01 23:18:48 +03:00
Uku Taht
fae81c1632 Update clickhouse_ecto 2022-04-28 17:00:43 +03:00
Uku Taht
e23cbfcb46 Only nodes that run cron should be elected as leader 2022-04-28 16:57:56 +03:00
Uku Taht
f18a211dcc
Ingest throughput improvement test setup (#1867)
* Add OTEL and test Cachex for sessions

* Move load test

* Start apps in the appropriate order
2022-04-28 12:24:29 +03:00
Uku Taht
ce55dc4ae9
Retries Google Analytics API request if it returns empty rows (#1858)
* Retries Google Analytics API request if it returns empty rows

* Remove HTTPoison.Retry completely
2022-04-26 13:57:32 +03:00
Uku Taht
7c1d64458e Add fun with flags library 2022-04-21 10:54:08 +03:00
Uku Taht
83c407c016
Upgrade Oban & configure Stager plugin (#1822) 2022-04-08 11:05:21 +03:00
Uku Taht
97f0dbbcbd Retry 502 response 2022-04-08 09:57:36 +03:00
Uku Taht
ef552c9a32 Upgrade dependencies 2022-04-08 08:56:09 +03:00
Uku Taht
06b165eb6d Run GA import in monthly batches 2022-04-08 08:43:07 +03:00
Uku Taht
0ab8d653be Add retry mechanism to HTTPoison 2022-03-30 13:47:41 +03:00
Marc Neudert
1c3085050c
Upgrade ua_inspector to 3.0 (#1762)
* Upgrade ua_inspector to 3.0

* Update ua_inspector database
2022-03-25 11:41:04 +02:00
RobertJoonas
492f47ba1e
Crm transfer data (#1749)
* pull from master

* added query generation by struct fields

* ready, improved tests

* fixed a naming mistake
2022-03-24 16:11:04 +02:00
Uku Taht
a9879de1f4 Remove more OTEL stuff 2022-03-21 13:05:34 +02:00
Uku Taht
83df555f55 Remove OTEL 2022-03-21 12:59:14 +02:00
Uku Taht
e27734ed79
[Continued] Google Analytics import (#1753)
* Add has_imported_stats boolean to Site

* Add Google Analytics import panel to general settings

* Get GA profiles to display in import settings panel

* Add import_from_google method as entrypoint to import data

* Add imported_visitors table

* Remove conflicting code from migration

* Import visitors data into clickhouse database

* Pass another dataset to main graph for rendering in red

This adds another entry to the JSON data returned via the main graph API
called `imported_plot`, which is similar to `plot` in form but will be
completed with previously imported data.  Currently it simply returns
the values from `plot` / 2. The data is rendered in the main graph in
red without fill, and without an indicator for the present. Rationale:
imported data will not continue to grow so there is no projection
forward, only backwards.

* Hook imported GA data to dashboard timeseries plot

* Add settings option to forget imported data

* Import sources from google analytics

* Merge imported sources when queried

* Merge imported source data native data when querying sources

* Start converting metrics to atoms so they can be subqueried

This changes "visitors" and in some places "sources" to atoms. This does
not change the behaviour of the functions - the tests all pass unchanged
following this commit. This is necessary as joining subqueries requires
that the keys in `select` statements be atoms and not strings.

* Convery GA (direct) source to empty string

* Import utm campaign and utm medium from GA

* format

* Import all data types from GA into new tables

* Handle large amounts of more data more safely

* Fix some mistakes in tables

* Make GA requests in chunks of 5 queries

* Only display imported timeseries when there is no filter

* Correctly show last 30 minutes timeseries when 'realtime'

* Add with_imported key to Query struct

* Account for injected :is_not filter on sources from dashboard

* Also add tentative imported_utm_sources table

This needs a bit more work on the google import side, as GA do not
report sources and utm sources as distinct things.

* Return imported data to dashboard for rest of Sources panel

This extends the merge_imported function definition for sources to
utm_sources, utm_mediums and utm_campaigns too. This appears to be
working on the DB side but something is incomplete on the client side.

* Clear imported stats from all tables when requested

* Merge entry pages and exit pages from imported data into unfiltered dashboard view

This requires converting the `"visits"` and `"visit_duration"` metrics
to atoms so that they can be used in ecto subqueries.

* Display imported devices, browsers and OSs on dashboard

* Display imported country data on dashboard

* Add more metrics to entries/exits for modals

* make sure data is returned via API with correct keys

* Import regions and cities from GA

* Capitalize device upon import to match native data

* Leave query limits/offsets until after possibly joining with imported data

* Also import timeOnPage and pageviews for pages from GA

* imported_countries -> imported_locations

* Get timeOnPage and pageviews for pages from GA

These are needed for the pages modal, and for calculating exit rates for
exit pages.

* Add indicator to dashboard when imported data is being used

* Don't show imported data as separately line on main graph

* "bounce_rate" -> :bounce_rate, so it works in subqueries

* Drop imported browser and OS versions

These are not needed.

* Toggle displaying imported data by clicking indicator

* Parse referrers with RefInspector

- Use 'ga:fullReferrer' instead of 'ga:source'. This provides the actual
  referrer host + path, whereas 'ga:source' includes utm_mediums and
  other values when relevant.
- 'ga:fullReferror' does however include search engine names directly,
  so they are manually checked for as RefInspector won't pick up on
  these.

* Keep imported data indicator on dashboard and strikethrough when hidden

* Add unlink google button to import panel

* Rename some GA browsers and OSes to plausible versions

* Get main top pages and exit pages panels working correctly with imported data

* mix format

* Fetch time_on_pages for imported data when needed

* entry pages need to fetch bounces from GA

* "sample_percent" -> :sample_percent as only atoms can be used in subqueries

* Calculate bounce_rate for joined native and imported data for top pages modal

* Flip some query bindings around to be less misleading

* Fixup entry page modal visit durations

* mix format

* Fetch bounces and visit_duration for sources from GA

* add more source metrics used for data in modals

* Make sources modals display correct values

* imported_visitors: bounce_rate -> bounces, avg_visit_duration -> visit_duration

* Merge imported data into aggregate stats

* Reformat top graph side icons

* Ensure sample_percent is yielded from aggregate data

* filter event_props should be strings

* Hide imported data from frontend when using filter

* Fix existing tests

* fix tests

* Fix imported indicator appearing when filtering

* comma needed, lost when rebasing

* Import utm_terms and utm_content from GA

* Merge imported utm_term and utm_content

* Rename imported Countries data as Locations

* Set imported city schema field to int

* Remove utm_terms and utm_content when clearing imported

* Clean locations import from Google Analytics

- Country and region should be set to "" when GA provides "(not set)"
- City should be set to 0 for "unknown", as we cannot reliably import
  city data from GA.

* Display imported region and city in dashboard

* os -> operating_system in some parts of code

The inconsistency of using os in some places and operating_system in
others causes trouble with subqueries and joins for the native and
imported data, which would require additional logic to account for. The
simplest solution is the just use a consistent word for all uses. This
doesn't make any user-facing or database changes.

* to_atom -> to_existing_atom

* format

* "events" metric -> :events

* ignore imported data when "events" in metrics

* update "bounce_rate"

* atomise some more metrics from new city and region api

* atomise some more metrics for email handlers

* "conversion_rate" -> :conversion_rate during csv export

* Move imported data stats code to own module

* Move imported timeseries function to Stats.Imported

* Use Timex.parse to import dates from GA

* has_imported_stats -> imported_source

* "time_on_page" -> :time_on_page

* Convert imported GA data to UTC

* Clean up GA request code a bit

There was some weird logic here with two separate lists that really
ought to be together, so this merges those.

* Fail sooner if GA timezone can't be identified

* Link imported tables to site by id

* imported_utm_content -> imported_utm_contents

* Imported GA from all of time

* Reorganise GA data fetch logic

- Fetch data from the start of time (2005)
- Check whether no data was fetched, and if so, inform user and don't
  consider data to be imported.

* Clarify removal of "visits" data when it isn't in metrics

* Apply location filters from API

This makes it consistent with the sources etc which filter out 'Direct /
None' on the API side. These filters are used by both the native and
imported data handling code, which would otherwise both duplicate the
filters in their `where` clauses.

* Do not use changeset for setting site.imported_source

* Add all metrics to all dimensions

* Run GA import in the background

* Send email when GA import completes

* Add handler to insert imported data into tests and imported_browsers_factory

* Add remaining import data test factories

* Add imported location data to test

* Test main graph with imported data

* Add imported data to operating systems tests

* Add imported data to pages tests

* Add imported data to entry pages tests

* Add imported data to exit pages tests

* Add imported data to devices tests

* Add imported data to sources tests

* Add imported data to UTM tests

* Add new test module for the data import step

* Test import of sources GA data

* Test import of utm_mediums GA data

* Test import of utm_campaigns GA data

* Add tests for UTM terms

* Add tests for UTM contents

* Add test for importing pages and entry pages data from GA

* Add test for importing exit page data

* Fix module file name typo

* Add test for importing location data from GA

* Add test for importing devices data from GA

* Add test for importing browsers data from GA

* Add test for importing OS data from GA

* Paginate GA requests to download all data

* Bump clickhouse_ecto version

* Move RefInspector wrapper function into module

* Drop timezone transform on import

* Order imported by side_id then date

* More strings -> atoms

Also changes a conditional to be a bit nicer

* Remove parallelisation of data import

* Split sources and UTM sources from fetched GA data

GA has only a "source" dimension and no "UTM source" dimension. Instead
it returns these combined. The logic herein to tease these apart is:

1. "(direct)" -> it's a direct source
2. if the source is a domain -> it's a source
3. "google" -> it's from adwords; let's make this a UTM source "adwords"
4. else -> just a UTM source

* Keep prop names in queries as strings

* fix typo

* Fix import

* Insert data to clickhouse in batches

* Fix link when removing imported data

* Merge source tables

* Import hostname as well as pathname

* Record start and end time of imported data

* Track import progress

* Fix month interval with imported data

* Do not JOIN when imported date range has no overlap

* Fix time on page using exits

Co-authored-by: mcol <mcol@posteo.net>
2022-03-10 15:04:59 -06:00
RobertJoonas
b4992cedc1
Referrer spam blocklist (#1750)
* integrating blocklist library

* loads blocklist dependency from Github
2022-03-10 13:58:30 -06:00
Uku Taht
e92cfe30bc Update location 2022-01-20 15:28:22 -06:00
Uku Taht
6f080bf2b2 Update location translations 2022-01-20 14:43:03 -06:00
Uku Taht
711cbdfca2 Remove unused custom domain code 2021-12-20 11:49:50 +02:00
Uku Taht
b1f64454d0 Update location dependency 2021-12-13 15:39:51 +02:00
Uku Taht
dcf31502c2 Show regions when filtering by country etc 2021-12-13 14:45:19 +02:00
Uku Taht
6ccd48c046 Update location dependency 2021-12-13 13:58:29 +02:00
Uku Taht
1dba113e2f
[Draft] Improve location translations (#1526)
* WIP

* Use location library for search suggestions

* Remove unused code

* Remove Countries completely

* Fix tests
2021-12-13 12:03:27 +02:00
Uku Taht
3243cbcd8e Update location dependency 2021-12-09 16:11:35 +02:00
Uku Taht
442e401ede Use new location library 2021-12-09 16:02:11 +02:00
Uku Taht
01706b7590 Remove dead code 2021-12-03 11:19:39 +02:00
Uku Taht
effe56b3e4 Use new iso_codes package 2021-12-01 15:31:50 +02:00
Uku Taht
5c02811d26 Add opentelemetry oban 2021-11-09 09:42:47 +02:00
Uku Taht
76f6a00bec Add honeycomb 2021-11-05 14:58:57 +02:00
Uku Taht
9edbbe805d Update dependencies 2021-10-25 11:16:56 +02:00
Uku Taht
f77c37800f Use safe version of Cachex.fetch
Fixes #905
2021-10-18 11:27:52 +02:00
Uku Taht
7cb148a4f1 Use only root domain in user id generation 2021-09-10 12:35:00 +03:00
Uku Taht
a84429ba0f Update ref and ua databases 2021-08-23 11:41:43 +03:00
Uku Taht
669866a16b
Finish stats module (#1248)
* Fix small inconsitencies in stats module

* Format
2021-08-17 15:21:12 +03:00
Uku Taht
ad6faece87
Stats API: Support IN filter expression (#1090)
* Support IN filter expression

* Update clickhouse_ecto
2021-06-02 15:17:12 +03:00
Uku Taht
6228995d4f Rate limit login attempts to 5 per minute 2021-05-25 11:32:54 +03:00