* Add data migration for creating and syncing location_data table and dictionary
* Migration to populate location data
* Daily cron to refresh location dataset if changed
* Add support for visit:country_name, visit:region_name and visit:city_name dimensions
Under the hood this relies on a `location_data` table in clickhouse being regularly synced with
plausible/location repo and dictionary lookups used in ALIAS columns
* Update queue name
* Update documentation
* Explicit structs
* Improve docs further
* Migration comment
* Add queues
* Add error when already loaded
* Test for filtering by new dimensions
* Update deps
* dimension -> select_dimension
* Update a test
* WIP mutation to populate event session columns
* Remove duplication
* report errors, allow_nondeterministic_updates
* use right columns
* Update existing columns instead of session_* ones
* Make dialyzer happy
* Fix issue with passing pre-existing params in
* Logger -> IO.puts
* Use IngestRepo.config for connection settings
* Make dictionary options configurable
* Move allow_nondeterministic_mutations to within the migration
* Solve credo warning about too deep nesting
* Missed logger call
* Pattern matching in function head
* Add data migration for moving to VersionedCollapsingMergeTree
This has been tested locally and partially on staging. Still requires a bit of work to verify.
Verification query:
```
SELECT main._partition_id, tmp.count, main.count
FROM (
SELECT _partition_id, count() AS count
FROM sessions_v2_tmp_versioned
GROUP BY _partition_id
) AS tmp
FULL OUTER JOIN (
SELECT _partition_id, count() AS count
FROM sessions_v2
GROUP BY _partition_id
) AS main
ON (tmp._partition_id == main._partition_id)
ORDER BY main._partition_id
```
* Add an early exit to migration
* cluster? extract common code
* Add `RewriteFunnelDupes` data fix
To rewrite funnels referencing goals whose names
are the same. This enables us to enforce all goals
within the site unique later on.
* Credo
* Update priv/data_migrations/FunnelDupeGoals/sql/list-funnels-with-dupe-goal-ids.sql.eex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Revert unrelated changes
* Remove dead code
* Update lib/plausible/data_migration/rewrite_funnel_dupes.ex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Update lib/plausible/data_migration/rewrite_funnel_dupes.ex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* default to v2
* allow N defaults in data migration prompt and custom messages
* join domains lookup
* remove duplicate test runs from ci (both are v2)
* Get rid of PASS_V2_SCHEMA_MIGRATION
* Use in-memory domain lookup + regular table settings
* Remove faulty date arithmetic + prev part calculation
* Set V2_MIGRATION_DONE in Mix.env == :dev
* Mute credo
* Remove ClickhouseSetup module
This has been an implicit point of contact to many
tests. From now on the goal is for each test to maintain
its own, isolated setup so that no accidental clashes
and implicit assumptions are relied upon.
* Implement v2 schema check
An environment variable V2_MIGRATION_DONE acts like
a feature flag, switching plausible from using old events/sessions
schemas to v2 schemas introduced by NumericIDs migration.
* Run both test suites sequentially
While the code for v1 and v2 schemas must be kept still,
we will from now on run tests against both code paths.
Secondary test run will set V2_MIGRATION_DONE=1 variable,
thus making all `Plausible.v2?()` checks return `true'.
* Remove unused function
This is a remnant from the short period when
we would check for existing events before allowing
creating a new site.
* Update test setups/factories with v2 migration check
* Make GateKeeper return site id along with :allow
* Make Billing module check for v2 schema
* Make ingestion aware of v2 schema
* Disable site transfers for when v2 is live
In a separate changeset we will implement simplified
site transfer for when v2 migration is complete.
The new transfer will only rename the site domain in postgres
and keep track of the original site prior to the transfer
so we keep an ingestion grace period until the customers
redeploy their scripting.
* Make Stats base queries aware of v2 schema switch
* Update breakdown with v2 conditionals
* Update pageview local start with v2 check
* Update current visitoris with v2 check
* Update stats controller with v2 checks
* Update external controller with v2 checks
* Update remaining tests with proper fixtures
* Rewrite redundant assignment
* Remove unused alias
* Mute credo, this is not the right time
* Add test_helper prompt
* Fetch priv dir so it works with a release
* Fetch distinct partitions only
* Don't limit inspect output for partitions
* Ensure SQL is printed to IO
* Remove redundant domain fixture
