analytics/test/workers
Adrian Gruntkowski f8b4d5066a
Add multiple imports per site (#3724)
* Clean up references to no longer active `google_analytics_imports` Oban queue

* Stub CSV importer

* Add SiteImport schema

* Rename `Plausible.Imported` module file to match module name

* Add `import_id` column to `Imported.*` CH schemas

* Implement Importer behavior and manage imports state using new entities

* Implement importer callbacks and maintain site.imported_data for UA

* Keep imports in sync when forgetting all imports

* Scope imported data queries to completed import IDs

* Mark newly imported data with respective import ID

* Clean up Importer implementation a bit

* Test querying legacy and new imported data

* Send Oban notifications on import worker failure too

* Fix checking for forgettable imports and remove redundant function

* Fix UA integration test

* Change site import source to atom enum and add source label

* Add typespecs and reduce repetition in `Plausible.Imported`

* Improve documentation and typespecs

* Add test for purging particular import

* Switch email notification templates depending on import source

* Document running import synchronously

* Fix UA importer args parsing and ensure it's covered by tests

* Clear `site.stats_start_date` on complete import to force recalculation

* Test Oban notifications (h/t @ruslandoga)

* Purge stats on import failure right away to reduce a chance of leaving debris behind

* Fix typos

Co-authored-by: hq1 <hq@mtod.org>

* Fix another typo

* Refactor fetching earliest import and earliest stats start date

* Use `Date.after?` instead of `Timex.after?`

* Cache import data in site virtual fields and limit queried imports to 5

* Ensure always current `stats_start_date` is used

* Work around broken typespec in Timex

* Make `SiteController.forget_imported` action idempotent

* Discard irrecoverably failed import tasks

* Use macros for site import statuses

There's also a fix ensuring only complete imports are considered
where relevant - couldn't isolate it as it was in a common hunk

* Use `import_id` as worker job uniqueness criterion

* Do not load imported stats data in plugins API context

---------

Co-authored-by: hq1 <hq@mtod.org>
2024-02-14 09:32:36 +01:00
..
accept_traffic_until_test.exs Update accept_traffic_until notifications (#3665) 2024-01-03 11:19:25 +01:00
check_usage_test.exs Prevent upgrades that would leave the customer locked (#3683) 2024-01-15 14:59:56 +00:00
clean_invitations_test.exs Invitations (#1122) 2021-06-16 15:00:07 +03:00
expire_domain_change_transitions_test.exs Fix flaky log assertion (#3199) 2023-07-27 12:51:54 +02:00
import_analytics_test.exs Add multiple imports per site (#3724) 2024-02-14 09:32:36 +01:00
lock_sites_test.exs Prevent upgrades that would leave the customer locked (#3683) 2024-01-15 14:59:56 +00:00
notify_annual_renewal_test.exs Refactor enterprise plan upgrade and change-plan actions (#3397) 2023-10-10 20:35:17 +03:00
schedule_email_reports_test.exs Do not send email reports for locked sites 2021-06-17 10:20:54 +03:00
send_check_stats_emails_test.exs Conditionally support switching between v1 and v2 clickhouse schemas (#2780) 2023-03-27 13:52:42 +02:00
send_email_report_test.exs Small bugfix + refactor email reports (#3642) 2023-12-21 12:56:06 +00:00
send_site_setup_emails_test.exs Conditionally support switching between v1 and v2 clickhouse schemas (#2780) 2023-03-27 13:52:42 +02:00
send_trial_notifications_test.exs Reapply "Define a better monthly pageview usage (#3564)" (#3574) 2023-11-30 13:30:04 +01:00
spike_notifier_test.exs Refactor spike detection top sources query (#3770) 2024-02-13 08:28:32 +02:00