Karl-Aksel Puulmann
850a843d82
Add migration to add ALIAS columns to common session/visit properties ( #4058 )
...
This allows for simplifications in the API code
2024-05-05 11:30:39 +03:00
Adrian Gruntkowski
069170eb1d
Add active_visitors column to imported_pages CH table ( #4028 )
2024-04-22 09:34:01 +02:00
RobertJoonas
bd73fc8266
CH Migration - add more imported metrics and properties ( #3949 )
...
* add migration
* add utm_source to imported_sources
* quickfix
* satisfy credo
* Revert "satisfy credo"
This reverts commit bb0b228164ab6f70c79e
2024-04-04 09:12:53 +01:00
hq1
1f778e0c11
CH Migration: exit page hostname on sessions_v2 ( #3953 )
...
* CH Migration: exit/entry hostnames in sessions_v2
* Leave only exit_page_hostname, we already record hostnames
* Use ClickHouse DDL in favour of ecto so that cluster is included
* Compress with ZSTD(3)
2024-04-03 09:42:47 +02:00
Karl-Aksel Puulmann
70d21f1b69
Improve compression of event table session columns ( #3864 )
2024-03-05 12:45:19 +02:00
Karl-Aksel Puulmann
ea38b45685
Ecto migration to move sessions_v2 to VersionedCollapsingMergeTree ( #3809 )
...
This migration will noop in staging/production as it already has been run. It also leaves
behind a backup table that initially takes no extra space but will need to be cleaned up
manually
2024-02-28 11:22:27 +02:00
Karl-Aksel Puulmann
acd4f33c8e
Migration to improve compression options of sessions_v2 and events_v2 tables ( #3803 )
...
Part of https://3.basecamp.com/5308029/buckets/35611491/messages/7061875211
Based on work in https://3.basecamp.com/5308029/buckets/35611491/messages/6949640880#__recording_7015576045
2024-02-23 09:19:57 +02:00
Adrian Gruntkowski
6915dd5e78
Add table migrations for supporting multiple imports per site ( #3785 )
...
* Add migration deleting jobs from legacy analytics import queue
* Add `site_imports` table
* Add `import_id` column to imported_* CH tables
* Extend one of `site_imports` indexes
2024-02-13 17:26:19 +01:00
Karl-Aksel Puulmann
f5e15941de
Add migration for minmax indexes on session timestamps ( #3772 )
2024-02-13 10:35:24 +02:00
hq1
52d5fac362
Fixup 4cdf843ad: ( #3434 )
...
- migration is irreversible for now,
likely due to https://github.com/plausible/ecto_ch/issues/81#issuecomment-1624754909
or maybe https://github.com/plausible/ecto_ch/pull/58/files
- IO.inspect call removed
2023-10-17 16:07:06 +02:00
Adrian Gruntkowski
4cdf843ade
Disable deduplication for improted_* tables in CH ( #3429 )
...
* Disable deduplication for improted_* tables in CH
* Make migration cluster-aware
* Bump cache to avoid problems
* Fix cache dump
* Remove cache bump
2023-10-17 12:51:13 +02:00
Vini Brasil
d98242895b
Add revenue fields to events_v2 ( #3018 )
2023-06-12 18:12:41 +01:00
ruslandoga
f489d96251
skip v1 table drop in self-host ( #2945 )
2023-05-25 10:32:33 +03:00
hq1
57af1f19ec
Drop events/sessions tables after V2 migration ( #2908 )
2023-05-10 09:19:50 +02:00
hq1
b9c2110472
V2 migration tweaks for self hosted release ( #2825 )
...
* Get rid of PASS_V2_SCHEMA_MIGRATION
* Use in-memory domain lookup + regular table settings
* Remove faulty date arithmetic + prev part calculation
* Set V2_MIGRATION_DONE in Mix.env == :dev
* Mute credo
2023-04-13 12:09:39 +02:00
Adam
6637751a5e
Implement Numeric IDs migration ( #2762 )
...
* Implement Numeric IDs migration
* Fix typo
* Mute credo for now
* Improve configurability and add stop_t
* Adjust to Ch/Chto only
* Fix opts key for dictionary password
* Add regular ecto migration with numeric ids v2 schemas (#2768 )
* Add regular ecto migration
* Fix typo
* Update priv/ingest_repo/migrations/20230320094327_create_v2_schemas.exs
Co-authored-by: Vini Brasil <vini@hey.com>
* Implement v2 events/sessions schema modules (#2777 )
* Implement v2 events/sessions schema modules
* Clean up session schemas
---------
Co-authored-by: Vini Brasil <vini@hey.com>
* Update moduledocs
---------
Co-authored-by: Vini Brasil <vini@hey.com>
2023-03-23 09:47:41 +01:00
Adam
6d79ca5093
Switch to new clickhouse adapter (ch/chto) ( #2733 )
...
* another clickhouse adapter
* don't restore stats_removal.ex
* fix events main-graph error (#2746 )
* update ch, chto
* update chto again (#2759 )
* Stop treating page filter as an entry_page filter (#2752 )
* remove dead code
* stop treating page filter as entry page filter in breakdown queries
* stop treating page filter as entry page filter in aggregate queries
* stop treating page filter as entry page filter in timeseries queries
* mix format
* update changelog
* break code down to smaller functions to keep credo happy
* remove unused functions
* make CSV export return only conversions with goal filter (#2760 )
* make CSV export return only conversions with goal filter
* update changelog
* update elixir version in mix.exs (#2742 )
* revert admin.ex changes (#2776 )
---------
Co-authored-by: ruslandoga <67764432+ruslandoga@users.noreply.github.com>
Co-authored-by: ruslandoga <rusl@n-do.ga>
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
2023-03-21 09:55:59 +01:00
Adam Rutkowski
043e3ed572
Clickhouse migration: add ingest_counters table ( #2692 )
...
* Clickhouse migration: add ingest_counters table
* Add toStartOfMinute() to ordering key
* Explicitly include column to be summarized
2023-02-23 09:34:44 +01:00
Vini Brasil
ce03b5ebd7
Move CH migration from clickhouse_repo/ to ingest_repo/ ( #2683 )
...
This commit moves a migration that was created in `clickhouse_repo/` to
its correct folder, `ingest_repo/`. The migration was created in
1cb07efe6d
2023-02-16 09:00:26 +01:00
Adam Rutkowski
8f85b110aa
Split Clickhouse pools into Read-Only and Read/Write (dedicated to writes) ( #2661 )
...
* Configure ingest repo access/pool size
If I'm not mistaken 3 is a sane default, the only
inserts we're doing are:
- session buffer dump
- events buffer dump
- GA import dump
And all are serializable within their scopes?
* Add IngestRepo
* Start IngestRepo
* Use IngestRepo for inserts
* Annotate ClickhouseRepo as read_only
So no insert* functions are expanded
* Update moduledoc
* rename alias
* Fix default env var value so it can be casted
* Use IngestRepo for migrations
* Set default ingest pool size from 3 to 5
in case conns are restarting or else...
* Ensure all Repo prometheus metrics are collected
2023-02-12 17:50:57 +01:00
