Commit Graph

49 Commits

Author SHA1 Message Date
ruslandoga
13a95c879a
bump cache version (#4339) 2024-07-11 11:37:57 +03:00
Cenk Kücük
39cf8c4179
Upgrade Clickhouse version in CI (#4062)
* Update elixir.yml

* Fix error

See https://github.com/ClickHouse/ClickHouse/issues/62431#issuecomment-2090283419

* fix order to pass test

* Revert "fix order to pass test"

This reverts commit d79fbeb297.

* Update Makefile

Update CH version
2024-05-13 21:53:29 +02:00
hq1
d6824de1ad
Rename internal build symbols (#3942)
* Rename internal build symbols

* Rename remaining + add `on_ce` macro

cc @ruslandoga
2024-04-29 08:05:33 +02:00
Adrian Gruntkowski
c1c03b729c
Reapply "Local CSV exports/imports and S3/UI updates (#3989)" (#3995) (#3996)
* Reapply "Local CSV exports/imports and S3/UI updates (#3989)" (#3995)

This reverts commit aee69e44c8.

* remove unused functions

* eh, that one was actually used

* ugh, they were both used

---------

Co-authored-by: ruslandoga <67764432+ruslandoga@users.noreply.github.com>
2024-04-11 09:15:01 +02:00
Adrian Gruntkowski
aee69e44c8
Revert "Local CSV exports/imports and S3/UI updates (#3989)" (#3995)
This reverts commit 1a0cb52f95.
2024-04-09 21:26:23 +02:00
ruslandoga
1a0cb52f95
Local CSV exports/imports and S3/UI updates (#3989)
* local CSV exports/imports and S3 updates

* credo

* dialyzer

* refactor input columns

* fix ci minio/clickhouse tests

* Update lib/plausible_web/live/csv_export.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* fix date range filter in export_pages_q and process only pageviews

* remove toTimeZone(zero_timestamp) note

* use SiteImport.pending(), SiteImport.importing()

* escape [SiteImport.pending(), SiteImport.importing()]

* use random s3 keys for imports to avoid collisions (sometimes makes the upload get stuck)

* clamp import date ranges

* site is already in assigns

* recompute cutoff date each time

* use toDate(timestamp[, timezone]) shortcut

* show alreats on export cancel/delete and extract hint into a component

* switch to Imported.clamp_dates/4

* reprocess tables when imports are added

* recompute cutoff_date on each call

* actually use clamped_date_range on submit

* add warning message

* add expiry rules to buckets in make minio

* add site_id to imports notifications and use it in csv_importer

* try/catch safer

* return :ok

* date range is not available when no uploads

* improve ui and warning messages

* use Generic.notice

* fix flaky exports test

* begin tests

* Improve `Importer` notification payload shape

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-04-09 20:59:48 +02:00
Karl-Aksel Puulmann
3115c6e7a8
Reducing JOINs in queries (#3966)
* Move experimental_session_count? logic to within query object

* WIP new querying system for deciding what tables to query

* both -> either

* Include sample_percent in both tables

* Remove a hanging TODO

* Allow filtering by visit props on event queries if flag is on

* Make default sessions join more conditional

* Simplify events_join_sessions?

* Add some TODOs

* Fix assignment

* Handle entry/exit page visit props separately from props stored in events table

* Update test which created sessions/events differently from everyone else

* Make query_events private

* Dont filter by session properties on events table if querying sessions and joining in events

* Handle visits, pageviews, events and visitors metrics from other table

* both -> either

* events, pageviews are strictly event metrics

* Add support for (plain) breakdowns deciding which table to use

* Run tests with experimental_reduced_joins as a separate job

Also refactor which tests are run with postgres:15 to reduce number of jobs

* moduledocs for TableDecider

* Fix matrix

* Custom build name

* Move TEST_EXPERIMENTAL_REDUCED_JOINS check

* Handle percentage separately from other metrics

* Remove debug code

* TableDecider tests

* both => sample_percent

* Improve naming

* Simplify code

* Breakdowns retain old behavior if getting metric visitors

* Unify behavior of entry/exit page hostnames with rest

* Fix test naming
2024-04-04 13:54:23 +03:00
ruslandoga
4ae00540dc
Split Elixir CI jobs (#3898)
* split elixir tests

* more understandable matrix

* add unused deps in mix.lock check

* get all deps for static checks
2024-03-18 09:17:04 +01:00
ruslandoga
f3423aefec
Add csv importer (#3795)
* add csv importer

* make table validation explicit

* update some docs

* improve docs

* add minio container to ci

* more tests

* eh

* continue

* add passing test

* add failing test

* add config test

* add minio to Makefile

* testcontainers

* remove extra whitespace

* explain the implementation a bit

* account for async deletes in tests

* bounces is UInt32

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-02-27 14:19:09 +01:00
Cenk Kücük
cc867396b2
Update elixir.yml (#3831)
Only test against the last 2 major versions
2024-02-26 14:10:57 +00:00
ruslandoga
bea8ceeffb
use newer postgres versions in the ci (#3821) 2024-02-26 11:36:11 +00:00
dependabot[bot]
c8b25347ed
Bump buildjet/cache from 3 to 4 (#3753)
Bumps [buildjet/cache](https://github.com/buildjet/cache) from 3 to 4.
- [Changelog](https://github.com/BuildJet/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/buildjet/cache/compare/v3...v4)

---
updated-dependencies:
- dependency-name: buildjet/cache
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Cenk Kücük <cenk@plausible.io>
2024-02-07 16:23:54 +00:00
Karl-Aksel Puulmann
e2eb861799
Github Actions: Run tests on all PRs (#3709)
Chained PRs (like https://github.com/plausible/analytics/pull/3708)
don't currently run tests. This PR fixes that.
2024-01-23 09:32:20 +02:00
hq1
32456d4348
Upgrade stack (deps, elixir 1.16, OTP 26.2.1) (#3678)
* Bump asdf erlang & elixir

* Bump erlang/elixir in the dockerfile

* Remove Oban.Stager config as per https://github.com/sorentwo/oban/blob/main/guides/upgrading/v2.14.md

* Configure Oban for tests as per https://github.com/sorentwo/oban/blob/main/guides/upgrading/v2.14.md

* Mark sampling hint clause with unsafe fragment

Any other/recommended way to do that? cc @ruslandoga

* Address String.slice/2 deprecation

* Update deps

* Address 0.0 matching warning

* Make funnel settings work

although this is probably not the best way to do it.
Needs revisiting, not sure what broke it - could not find
any breaking changes in related dependencies.

* Migrate oban as per https://github.com/sorentwo/oban/blob/main/guides/upgrading/v2.17.md

* Update credo

* Bump CI cache

* Use `Bypass.pass` to avoid exit shutdown message
2024-01-11 08:48:04 +01:00
hq1
b9ec38038c
Add small build option (#3536)
* Update applications

* Clone community config

* Move modules to experimental dir

* Update runtime config

* Apply first set of compile-time conditionals

* Move funnel schemas to experimental

* Make funnel schema-less build compile

* Use experimental/lib for elixir code

* Move JS funnels to experimental

* Clean up conditional rendering

* Tidy up the pipeline

* Make two builds pass tests without warnings

* Reuse existing dotenvs

* Do a bunch of renames

* Clean up naming

* Run secondary CI

* Update router

* Remove RewriteFunnelDupes migration

Tests were disabled already and it was a one-off shot

* Fixup quota mixins

* Add moduledoc

* Change MIX_ENV for seconary test run

* Skip crm on small

* !fixup

* Exclude flags pipeline

* Update lib/plausible_web/controllers/stats_controller.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2023-11-20 12:52:20 +01:00
hq1
117eef000d
Upgrade Erlang/Elixir stack (#3454)
* Bump deps

* Bump stack

* Fix deprecation warnings

* Fix VCR cassettes mismatch due to OTP-18414

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Format & fix flaky tests

* Handle raw IPv4 hostnames; test public suffix TLD

* Configure locus db cache_dir

So that maxmind unavailability doesn't affect
application startup. PERSISTENT_CACHE_DIR env var is used
to point locus at the GeoIP DB file.

* WIP: Remove ExVCR

* Fix test env config

* Fixup exvcr

* Remove exvcr from deps

* Add convert script

* Remove exvcr cassettes

* Remove convert script

* Rename test

* Update moduledoc

* Update dockerfile

* Bump CI cache

* Tag more slow tests, why not?

* Use charlist for locus cache option

* Pin nodejs

* Merge google tests, make them async

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2023-10-24 10:33:48 +02:00
Cenk Kücük
5339db7152
Move back to GitHub managed actions runner (#3383)
This fixes CI for PRs coming from external repos/contributors
2023-09-29 11:10:46 +01:00
Cenk Kücük
1a9dd0de90
Update Workflow Concurrency (#3359)
Ensures that only one workflow is running per branch at a time.

If you are working on a branch and push several commits in succession, any workflows in progress will be cancelled and only the last one will run.
2023-09-20 14:06:04 +02:00
dependabot[bot]
ce4c9fced5
Bump actions/checkout from 3 to 4 (#3338)
Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-14 14:13:34 +02:00
Cenk Kücük
ec747ad401
add dockerhub creds optional 2023-08-09 12:45:27 +02:00
Cenk Kücük
e26efdb7c0
remove dockerhub credentials from pull (#3188)
I previously introduced using the dockerhub credentials when pulling images from dockerhub. But this leads to problems when creating PRs from a fork.
And it is not so clear if we really need to use the dockerhub creds, as it seems that there were no problems before.
2023-07-25 12:19:00 +02:00
Cenk Kücük
36bfdb35f5
upgrade github runner (#3114) 2023-07-12 10:39:46 +02:00
Cenk Kücük
13a38aee14
upgrade clickhouse (#3113) 2023-07-11 13:54:50 +02:00
Cenk Kücük
f374650330
Postgres upgrade (#3062)
* Update list to the last 3 versions of postgres

* Update postgres prod version
2023-06-30 14:01:12 +02:00
hq1
d8543c81cc
Funnel site settings (#3039)
* Update formatter config

* Install LiveView JS integration & hooks

* Temporarily update endpoint/session config

* Optionally allow preloading funnels for goals

* Site controller

* Implement funnel settings

lib/plausible_web/live/funnel_settings/combo_box.ex - restored from:

054de6e2 Fix the tab/blur bug again
20da4c89 Rename InputPicker to ComboBox

lib/plausible_web/live/funnel_settings/form.ex - restored from:

9bedda3b Remove potential FIXME
20da4c89 Rename InputPicker to ComboBox
028036ad Review comments
aea4ebc4 Access Funnel min/max steps via the __using__/1 macro
0dde27fd Remove inspect call
eed588a7 Start testing the funnel editor
0e95228b Extract funnel settings test module
7b16ace5 Leverage aplinejs to deal with the tyranny
8dc6a3e7 wip
cf228630 wip
30a43fd1 wip
89f10ecb wip
950a18d9 Dirty funnel save
298a6a53 wip
7690d50f wip
639c6238 fixup
aa59adeb wip
ff75c00b wip

lib/plausible_web/live/funnel_settings/list.ex - restored from:

4eae122c Fix data-confirm attr interpolation
51f0397d Implement deleting funnels
1f6fe25d Add number of steps to funnels list
298a6a53 wip
ff75c00b wip

test/plausible_web/live/funnel_settings/funnel_settings/combo_box_test.exs - restored from:

20da4c89 Rename InputPicker to ComboBox

test/plausible_web/live/funnel_settings/funnel_settings_test.exs - restored from:

34822ff4 Bootstrap InputPicker tests

lib/plausible_web/live/funnel_settings.ex - restored from:

028036ad Review comments
acd9c4f2 Prepare ephemeral funnel definitions so that users can test funnels
51f0397d Implement deleting funnels
0e95228b Extract funnel settings test module
8dc6a3e7 wip
89f10ecb wip
950a18d9 Dirty funnel save
298a6a53 wip
aa59adeb wip
ff75c00b wip

test/plausible_web/controllers/error_report_controller_test.exs - restored from:

34822ff4 Bootstrap InputPicker tests

test/support/html.ex - restored from:

0a53979d Improve InputPicker tests - include AlpineJS assertions
34822ff4 Bootstrap InputPicker tests

lib/plausible_web/views/layout_view.ex - restored from:

b490403b !ifxup

lib/plausible_web/templates/site/settings_funnels.html.eex - restored from:

51f0397d Implement deleting funnels
ea1315f3 Test funnels list in settings
7b16ace5 Leverage aplinejs to deal with the tyranny
ff75c00b wip
4da25c35 Fixup

lib/plausible_web/templates/layout/app.html.eex - restored from:

ff75c00b wip

* Add funnel settings route

* Warn about funnels deletion when deleting goals

lib/plausible_web/templates/site/settings_goals.html.eex - restored from:

fdd9bcd0 Fixup
f1e6364d Merge remote-tracking branch 'origin/master' into funnels-rebase
9d0b7c6d Fix markup error
4a4ddbdc Optionally preload funnels for goals and stub funnel-goal deletion
ebdc4333 Extend the prompt in case of funnel-goal deletion
639c6238 fixup
aa59adeb wip

* Split new JS LiveView additions

* Put funnels behind a feature flag

* Integrate dashboard feature toggle

* Update signing salt for live view

* Update moduledocs

* Update live reloader config

* Use Phoenix.HTML.Safe for goal names

* Workaround to get flashes working in embedded liveview

* Keep feature toggles idempotent, rename property to setting

We'll still retain the ability to flip bools on a lower level.

* Update moduledocs

* Make live flash disappear after 5s

* Tailwind: purge .heex files too

* Update docs link

* Add live components to tailwind purge config

* Update another flaky test

Ref f0bdf872
cc @vinibrsl

* Fix combobox input length w/ WebKit

* Intoduce generic notice component

* Revert "Fix combobox input length w/ WebKit"

This reverts commit 3c653a6d85d5000167631e10ef45a93c13b41ed1.

* Fix combobox input length on webkit

* Make whole combobox item clickable, not only text

* Fix glitch moving Save button on activation

* Tweak dark mode

* Show funnel form without waiting for funnel name input

* Tweak dark mode

* Include static Phoenix components in tailwind purge

* Tune funnels form into a liveview of its own

This is so that ComboBoxes can publish their selections
and unavailable choices can be propagated to other siblings.

* Push less data over websocket

* Undo Lsp/formatter race condition

* Fixup typespecs

* Bust CI cache
2023-06-22 09:00:07 +02:00
dependabot[bot]
42022fcb29
Bump actions/cache from 2 to 3 (#2915)
Bumps [actions/cache](https://github.com/actions/cache) from 2 to 3.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-05-11 14:08:39 +02:00
RobertJoonas
d161c1be0b
Upgrade phoenix (#2902)
* upgrade phoenix

Co-authored-by: Vini Brasil <vini@hey.com>

* fix a test (flash message)

The flash message in focus.html.eex was not covered by any test. This
commit fixes also fixes that.

* change function name

* remove unnecessary formatter and format

* update CI cache

* fix dialyzer error

---------

Co-authored-by: Vini Brasil <vini@hey.com>
2023-05-09 11:51:35 +03:00
ruslandoga
adcce15632
Make self-hosted data migration easier (#2865)
* default to v2

* allow N defaults in data migration prompt and custom messages

* join domains lookup

* remove duplicate test runs from ci (both are v2)
2023-04-21 09:33:57 +02:00
hq1
d2f2c69387
Conditionally support switching between v1 and v2 clickhouse schemas (#2780)
* Remove ClickhouseSetup module

This has been an implicit point of contact to many
tests. From now on the goal is for each test to maintain
its own, isolated setup so that no accidental clashes
and implicit assumptions are relied upon.

* Implement v2 schema check

An environment variable V2_MIGRATION_DONE acts like
a feature flag, switching plausible from using old events/sessions
schemas to v2 schemas introduced by NumericIDs migration.

* Run both test suites sequentially

While the code for v1 and v2 schemas must be kept still,
we will from now on run tests against both code paths.
Secondary test run will set V2_MIGRATION_DONE=1 variable,
thus making all `Plausible.v2?()` checks return `true'.

* Remove unused function

This is a remnant from the short period when
we would check for existing events before allowing
creating a new site.

* Update test setups/factories with v2 migration check

* Make GateKeeper return site id along with :allow

* Make Billing module check for v2 schema

* Make ingestion aware of v2 schema

* Disable site transfers for when v2 is live

In a separate changeset we will implement simplified
site transfer for when v2 migration is complete.
The new transfer will only rename the site domain in postgres
and keep track of the original site prior to the transfer
so we keep an ingestion grace period until the customers
redeploy their scripting.

* Make Stats base queries aware of v2 schema switch

* Update breakdown with v2 conditionals

* Update pageview local start with v2 check

* Update current visitoris with v2 check

* Update stats controller with v2 checks

* Update external controller with v2 checks

* Update remaining tests with proper fixtures

* Rewrite redundant assignment

* Remove unused alias

* Mute credo, this is not the right time

* Add test_helper prompt

* Fetch priv dir so it works with a release

* Fetch distinct partitions only

* Don't limit inspect output for partitions

* Ensure SQL is printed to IO

* Remove redundant domain fixture
2023-03-27 13:52:42 +02:00
Adam
4654a50365
Revert "Fail CI when tests are logging (#2748)" (#2758)
This reverts commit 05d14774be.
2023-03-15 13:13:46 +01:00
Adam
05d14774be
Fail CI when tests are logging (#2748)
A re-iteration of a miserable attempt from #2694

Hi @dgvncsz0f, how you've been ;D
2023-03-14 07:45:41 -03:00
Adam
59a76dedf1
Revert "Fail the CI if tests emit log errors/warnings (#2694)" (#2745)
* Revert "Fail the CI if tests emit log errors/warnings (#2694)"

This reverts commit ef894b0546.

* Mute a test
2023-03-13 16:13:17 +01:00
Adam Rutkowski
ecb552479f
Upgrade erlang/elixir/alpine stack (#2695)
* Dockerfile: pin elixir-1.14.3, erlang-25.2.3, alpine-3.17.0

* asdf: pin erlang 25.2.3, elixir 1.14.3-otp-25

* Remove nonsense

* Remove erlang dupe

* Remove erlang, it's in the base image already

* Remove empty file

* Truncate seconds in NaiveDateTime instances

* Bump cache key

* Mitigate Error:error:0308010C:digital envelope routines::unsupported

See: https://www.newline.co/@kchan/how-to-fix-the-error-errorerror0308010cdigital-envelope-routinesunsupported--0f8d3f17

* Upgrade to latest alpine image and pin to an exact digest

* Revert "Upgrade to latest alpine image and pin to an exact digest"

This reverts commit cdf6624efa.

* Pin alpine image to an exact digest

---------

Co-authored-by: Cenk Kücük <c@cenk.me>
2023-02-22 09:49:43 +01:00
Adam Rutkowski
ef894b0546
Fail the CI if tests emit log errors/warnings (#2694)
* Make CI fail if tests emit warning/error logs

* Swallow log output in GA test case
2023-02-21 10:38:22 +01:00
Adam Rutkowski
ac1f44f89a
Bump clickhouse images to 22.8.13.20-alpine (prod/CI) (#2660)
* Bump clickhouse images to 22.8.13.20-alpine (prod/CI)

* !fixup

* Remove clickhouse-ci target
2023-02-09 08:11:03 +01:00
Adam Rutkowski
5de43b758d
Run tests in async mode where applicable (#2542)
* Set pg pool size for MIX_ENV=test

* Include slow tests in CI run

* Exclude slow tests by default

* Mark tests slow/async where applicable

* Restructure captcha mocks

* Revert async where env is relied upon

* Add --max-failures=1 to CI run

* Set warnings as errors

* Disable async where various mocks are used

* Revert "Disable async where various mocks are used"

This reverts commit 2446b72a29.

* Disable async for test using vcr
2022-12-26 10:20:29 -03:00
Uku Taht
8bdbabbdad
Enable Dialyzer and Credo in CI (#2315) 2022-11-04 23:42:03 -03:00
RobertJoonas
c0da024b23
Remove static tracker files (#2116)
* remove tracker files from git index

* generate tracker files on npm test

* generate tracker files for elixir tests/dev/CI

* update tracker/package-lock.json

* exclude npm run deploy from mix test + some docs
2022-10-11 12:19:28 +02:00
Uku Taht
eca71b9b60 Run CI against all supported PostgreSQL versions 2022-10-05 11:01:41 +03:00
Uku Taht
df9f96970c Clear github actions cache for Elixir compilation 2022-09-21 13:41:03 +03:00
Kian-Meng Ang
ae61aecfd0
Use erlef/setup-beam for GitHub Actions (#1880) 2022-06-06 10:42:24 +03:00
Adam from Buildjet
1146ec0960
BuildJet 4vCPU (#1523) 2021-12-08 15:57:15 +02:00
Uku Taht
b01b569cd5 Update clickhouse version in tests 2021-11-16 10:45:21 +02:00
Uku Taht
1e0c711506 Disable dialyzer in CI 2021-08-19 13:23:44 +03:00
Uku Taht
435c1289ae Clear github actions cache 2021-04-21 16:06:20 +03:00
Uku Taht
844af698ce
Embeddable dashboard (#812)
* Add embedded mode for shared links

* Add test for X-Frame-Options

* Improve embed code generator

* Add changelog entry

Co-authored-by: aymanterra <aymanterra@users.noreply.github.com>
Co-authored-by: Brad Kane <bradkane@users.noreply.github.com>
2021-03-10 14:41:17 +02:00
Oliver Kriska
ae42b86792
Dialyzer and Credo checks (#558)
* Checks

 - added Dialyzer
 - fixed Dialyzer errors
 - added Dialyzer check to GitHub Actions with cache
 - added Credo
 - fixed Credo Warnings
 - added Credo Warnings check to GitHub Actions with cache
 - added compile warnings check to GitHub Actions
 - reformated GitHub Actions YAML

* Dialyzer

 - allow it in test env

* Dialyzer

 - fixed test env
 - renamed GitHub actions steps

* AppSignal

 - upgraded deprecated version
 - Upgraded:
  appsignal 2.0.5 => 2.0.7
  certifi 2.5.2 => 2.5.3
  hackney 1.16.0 => 1.17.0
  idna 6.0.1 => 6.1.1
  parse_trans 3.3.0 => 3.3.1
  unicode_util_compat 0.5.0 => 0.7.0 (minor)

* Credo

 - fixed CRM plug
2021-01-07 15:16:04 +02:00
Uku Taht
9fe74af4c3 Add workflows on stable 2020-12-29 16:32:31 +02:00
Uku Taht
81c12884cd
Add elixir action (#526)
* Add elixir action

* Format the codebase

* Add postgresql

* Postgres config

* Run postgres on localhost

* Add clickhouse to CI
2020-12-29 15:17:27 +02:00