* Add runtime config option for enabled/disabling csv imports and exports
* Use the new option to toggle rendering exports UI
* Disable import buttons when at maximum imports or when option disabled for CSV
* Improve forms for GA import flow
* Add test for maximum imports reached
* Remove "Changed your mind?" prefixing back button
* Hide UA imports in Integrations when `imports_exports` flag is enabled
* Implement `csv_imports_exports` feature flag
* Revert "Add runtime config option for enabled/disabling csv imports and exports"
This reverts commit e30f202dd3.
* Send import notification email only to the user who ran the import
* Improve rendering of disabled button state
* Put import status heroicon in front of import label
* Reapply "Local CSV exports/imports and S3/UI updates (#3989)" (#3995)
This reverts commit aee69e44c8.
* remove unused functions
* eh, that one was actually used
* ugh, they were both used
---------
Co-authored-by: ruslandoga <67764432+ruslandoga@users.noreply.github.com>
* local CSV exports/imports and S3 updates
* credo
* dialyzer
* refactor input columns
* fix ci minio/clickhouse tests
* Update lib/plausible_web/live/csv_export.ex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* fix date range filter in export_pages_q and process only pageviews
* remove toTimeZone(zero_timestamp) note
* use SiteImport.pending(), SiteImport.importing()
* escape [SiteImport.pending(), SiteImport.importing()]
* use random s3 keys for imports to avoid collisions (sometimes makes the upload get stuck)
* clamp import date ranges
* site is already in assigns
* recompute cutoff date each time
* use toDate(timestamp[, timezone]) shortcut
* show alreats on export cancel/delete and extract hint into a component
* switch to Imported.clamp_dates/4
* reprocess tables when imports are added
* recompute cutoff_date on each call
* actually use clamped_date_range on submit
* add warning message
* add expiry rules to buckets in make minio
* add site_id to imports notifications and use it in csv_importer
* try/catch safer
* return :ok
* date range is not available when no uploads
* improve ui and warning messages
* use Generic.notice
* fix flaky exports test
* begin tests
* Improve `Importer` notification payload shape
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Create a stub of site settings section for imports and exports
* Use legacy site import indication to determine UA import handling
* Add provisional logos for upcoming import sources
* Stub basics of import page
* Add very rudimentary support for multiple UA imports
* Implement imports list as live view
* Add support for opening LV modal from backend and closing from frontend
* Introduce notion of themes to `button` and `button_link` components
* Add confirmation modal on deleting import
* Swap GA4 logo
* Implement disabled state support for `button_link` component
* Disable export and non-implemented import sources
* Use native starts start date for upper boundary of import time range
* Ensure integrations view uses legacy UA import flow
* Remove unnecessary preload in SiteController
* Remove unnecessary exception for legacy imports
* Move API controller stats tests under PlausibleWeb
* Test listing imports
* Add test for explicit listener setup
* Add tests for legacy flag state in UA importer
* Add test for purging legacy import data
* Add tests for `Sites.native_stats_start_date`
* Test forgetting imports
* Add `Stats.Clickhouse.imported_pageview_counts/1` and fix test flakiness
* Show page view counts on imports list
* Add tests for static imports and exports view
* Adjust button look slightly
* Use `case` instead of `cond`
* Make feature flag customisable per site
* Fix buttons and empty state styling
* Add another import to seeds
* Use JS confirm dialog instead of modal for deletion confirmations
* Revert "Add support for opening LV modal from backend and closing from frontend"
This reverts commit 260e6c753032b451542e24be9edc2118790b5a00.
* Default `legacy` to false when inserting new import jobs
* Drop `method` attribute from `button_link` and `unstyled_link` components
* Clean up references to no longer active `google_analytics_imports` Oban queue
* Stub CSV importer
* Add SiteImport schema
* Rename `Plausible.Imported` module file to match module name
* Add `import_id` column to `Imported.*` CH schemas
* Implement Importer behavior and manage imports state using new entities
* Implement importer callbacks and maintain site.imported_data for UA
* Keep imports in sync when forgetting all imports
* Scope imported data queries to completed import IDs
* Mark newly imported data with respective import ID
* Clean up Importer implementation a bit
* Test querying legacy and new imported data
* Send Oban notifications on import worker failure too
* Fix checking for forgettable imports and remove redundant function
* Fix UA integration test
* Change site import source to atom enum and add source label
* Add typespecs and reduce repetition in `Plausible.Imported`
* Improve documentation and typespecs
* Add test for purging particular import
* Switch email notification templates depending on import source
* Document running import synchronously
* Fix UA importer args parsing and ensure it's covered by tests
* Clear `site.stats_start_date` on complete import to force recalculation
* Test Oban notifications (h/t @ruslandoga)
* Purge stats on import failure right away to reduce a chance of leaving debris behind
* Fix typos
Co-authored-by: hq1 <hq@mtod.org>
* Fix another typo
* Refactor fetching earliest import and earliest stats start date
* Use `Date.after?` instead of `Timex.after?`
* Cache import data in site virtual fields and limit queried imports to 5
* Ensure always current `stats_start_date` is used
* Work around broken typespec in Timex
* Make `SiteController.forget_imported` action idempotent
* Discard irrecoverably failed import tasks
* Use macros for site import statuses
There's also a fix ensuring only complete imports are considered
where relevant - couldn't isolate it as it was in a common hunk
* Use `import_id` as worker job uniqueness criterion
* Do not load imported stats data in plugins API context
---------
Co-authored-by: hq1 <hq@mtod.org>
* Move imported tables schemas to separate modules outside Google ns
* Move buffer for imports to Imported ns
* fix schema newlines
* Extract UA import processing and persistence
* Decouple analytics worker implementation from UA
* Rename env variable for import buffer size
* Preserve old import queue until release
