analytics/extra
Karl-Aksel Puulmann 969f094e61
fractional_sample_rate: Vary sample rate according to traffic estimation (#4839)
* fractional_sample_rate: Vary sample rate according to traffic in the past 30 days

Our old sampling mechanism used SAMPLE 20000000 syntax. This was
wonderful since it allowed essentially dynamic sampling based on the
data being queried. However this ran into many issues relating to JOINs
and sample rate being different for different tables.

Instead, we now start to dynamically vary sample rates fractionally.

At query time we check the time window being queried and estimate how
many rows this query might reach. For large queries, we then dynamically
decide the sample rate.

For getting the traffic estimate for a site, we have a new SampingCache class which queries `ingest_counters`.

The query being cached is slightly expensive and can be sped up with a
ClickHouse projection.

* ce behavior

* Fix test

* Update test/plausible/stats/sampling_cache_test.exs

Co-authored-by: hq1 <hq@mtod.org>

* Update tests

* Update tests

---------

Co-authored-by: hq1 <hq@mtod.org>
2024-11-25 07:02:19 +00:00
..
lib fractional_sample_rate: Vary sample rate according to traffic estimation (#4839) 2024-11-25 07:02:19 +00:00
COPYING.txt Branding stuff (#3816) 2024-02-23 13:35:22 +02:00