* Channels: Migration to add column, backfill code
This change adds `acqusition_channel` columns to events_v2 and
sessions_v2 tables. These columns are materialized - we don't ingest
into them directly. Instead they're calculated based on other columns.
The data migration changes now allow to also backfill the column.
Tested the ability to change definitions by changing the function
definitions and re-running the migration with backfill. Confirmed that
the underlying data changed as expected.
* quiet option
* Exclude data migrations from validation
* Migration consistency
* Remove query.v2 flag
This was originally used for making sure queries use the right table as
migrating to APIv2. This is no longer needed
* Remove experimental_reduced_joins flag
* Run all tests with `read_team_schemas` flag up and down on CI
Replacing the existing setup for `experimental_reduced_joins` which
wasn't actively used and will be removed some time in the future.
This way we don't make CI run longer than necessary.
* Add one more test case for listing invitations on /sites LV
* Intentionally reintroduce regression to test CI
* Revert "Intentionally reintroduce regression to test CI"
This reverts commit cdb83c1494.
* Run tests for `experimental_reduced_joins` together with `read_team_schemas`
* order migrations
* migrate in 'streaks'
* cleanup
* docs
* tests
* add comment
* changelog
* continue
* run migrations tests in ci
* add pending_streams cmd
* fix ci
* add docs
* fix test
* simplify test
* only test v2+
* fix test
* return async: true
* cleanup
* add 'no pending migration'
* drop migrate/0 and pending_migrations/0
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Reapply "Local CSV exports/imports and S3/UI updates (#3989)" (#3995)
This reverts commit aee69e44c8.
* remove unused functions
* eh, that one was actually used
* ugh, they were both used
---------
Co-authored-by: ruslandoga <67764432+ruslandoga@users.noreply.github.com>
* local CSV exports/imports and S3 updates
* credo
* dialyzer
* refactor input columns
* fix ci minio/clickhouse tests
* Update lib/plausible_web/live/csv_export.ex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* fix date range filter in export_pages_q and process only pageviews
* remove toTimeZone(zero_timestamp) note
* use SiteImport.pending(), SiteImport.importing()
* escape [SiteImport.pending(), SiteImport.importing()]
* use random s3 keys for imports to avoid collisions (sometimes makes the upload get stuck)
* clamp import date ranges
* site is already in assigns
* recompute cutoff date each time
* use toDate(timestamp[, timezone]) shortcut
* show alreats on export cancel/delete and extract hint into a component
* switch to Imported.clamp_dates/4
* reprocess tables when imports are added
* recompute cutoff_date on each call
* actually use clamped_date_range on submit
* add warning message
* add expiry rules to buckets in make minio
* add site_id to imports notifications and use it in csv_importer
* try/catch safer
* return :ok
* date range is not available when no uploads
* improve ui and warning messages
* use Generic.notice
* fix flaky exports test
* begin tests
* Improve `Importer` notification payload shape
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Move experimental_session_count? logic to within query object
* WIP new querying system for deciding what tables to query
* both -> either
* Include sample_percent in both tables
* Remove a hanging TODO
* Allow filtering by visit props on event queries if flag is on
* Make default sessions join more conditional
* Simplify events_join_sessions?
* Add some TODOs
* Fix assignment
* Handle entry/exit page visit props separately from props stored in events table
* Update test which created sessions/events differently from everyone else
* Make query_events private
* Dont filter by session properties on events table if querying sessions and joining in events
* Handle visits, pageviews, events and visitors metrics from other table
* both -> either
* events, pageviews are strictly event metrics
* Add support for (plain) breakdowns deciding which table to use
* Run tests with experimental_reduced_joins as a separate job
Also refactor which tests are run with postgres:15 to reduce number of jobs
* moduledocs for TableDecider
* Fix matrix
* Custom build name
* Move TEST_EXPERIMENTAL_REDUCED_JOINS check
* Handle percentage separately from other metrics
* Remove debug code
* TableDecider tests
* both => sample_percent
* Improve naming
* Simplify code
* Breakdowns retain old behavior if getting metric visitors
* Unify behavior of entry/exit page hostnames with rest
* Fix test naming
* wip
* more env, setup user before packages to avoid them picking our uid
* make entrypoint.sh executable by all again
* apparently it's a best practice to have executables be owned by root
* make entrypoint executable in COPY
* stop writing to /app, write to /tmp if needed
* fewer changes
* system user
* keep same style for multiline commands
* fewer changes
* add changelog entry
* fix group assignment for plausible user
* use gid=999
* no home
* no home
* add gecos
* add plausible user to nogroup instead of creating a custom one
* eh
* fewer changes
* fewer changes
* fewer changes
* use PERSISTENT_CACHE_DIR instead of STORAGE_DIR
* ignore more
* cleanup
* remove hex timeout env var
* use ERL_FLAGS=+JMsingle true in public builds
* fallback to /tmp and nest under /tzdata_data/ for tzdata
---------
Co-authored-by: Cenk Kücük <cenk@plausible.io>
* add csv importer
* make table validation explicit
* update some docs
* improve docs
* add minio container to ci
* more tests
* eh
* continue
* add passing test
* add failing test
* add config test
* add minio to Makefile
* testcontainers
* remove extra whitespace
* explain the implementation a bit
* account for async deletes in tests
* bounces is UInt32
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
