ref
https://linear.app/tryghost/issue/ANAL-60/click-through-filtering-for-sources
- In our stats page we use the referrer without a protocol or www, that
is the pure domain as our source that we output
- Meanwhile all the data pipelines had the full url as the referrer
passed through
- When we come to add clickthroughs/filtering, we'll need to use this
value to filter the data. If we have a different value locally in the UI
to what is in the DB, we won't be able to make the filters match
- Also, we pay for everything we store, and this removes all the
https:// and www. data
- Whilst we are in development, we can safely make changes to all
aspects of our pipeline without worrying
- This is because currently, it's safe to delete all data and start over
- This script removes everything excepts the analytics_events
datasource, and then recreates everything fresh, repopulating from the
datasource where possible
- This shouldn't be used after tinybird is in production, we need a
better change process
closes https://linear.app/tryghost/issue/ANAL-77/na-data-should-be-zero
ref https://www.tinybird.co/blog-posts/tips-9-filling-gaps-in-time-series-on-clickhouse
- Sometimes we have no matching data for a particular date/date range, which makes our charts look super janky
- Clickhouse has a feature to fill these in called WITH FILL, which makes it really easy to fix this!
- WITH FILL works except for on bounce rate. That seems to be due to the column being marked as nullable and so WITH FILL fills missing data with NULL instead of 0
- To fix that, I've updated the code that generates the bounce rate so that it doesn't generate nulls, and that seems to result in a not-nullable column, which then works with WITH FILL
- When I went off, I quickly recreated all our endpoints with some new functionality
- However, I forgot that I was manually managing tokens, this meant the UI for stats broke with a token error
- Adding the tokens to the endpoint definitions should prevent this happening again, by automating the management of the token scopes
closes https://linear.app/tryghost/issue/ANAL-23/filtering-by-logged-out-logged-in-traffic
- Updated all of our tinybird datasources and pipes to handle member status
- Added member_status as an array query param to the API endpoints
- Added a really dodgy power select multiple to the stats page to demonstrate it works (needs styling)
- Added all of the wiring so each chart updates
- This was done pretty fast, and may not be 100% right yet
ref https://linear.app/tryghost/issue/ANAL-27/setup-tinybird-project-and-cicd
- Tinybird has a system for managing it's configuration as code, with full ci/cd support
- The tinybird CLI tool uses python, so we'll run that using docker, via `yarn tb`
- Some of the files tinybird adds should not be in source control, so we've added those to git ignore
- Everything in /ghost/tinybird is tinybird's init config
