* Reapply "Local CSV exports/imports and S3/UI updates (#3989)" (#3995)
This reverts commit aee69e44c8.
* remove unused functions
* eh, that one was actually used
* ugh, they were both used
---------
Co-authored-by: ruslandoga <67764432+ruslandoga@users.noreply.github.com>
* local CSV exports/imports and S3 updates
* credo
* dialyzer
* refactor input columns
* fix ci minio/clickhouse tests
* Update lib/plausible_web/live/csv_export.ex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* fix date range filter in export_pages_q and process only pageviews
* remove toTimeZone(zero_timestamp) note
* use SiteImport.pending(), SiteImport.importing()
* escape [SiteImport.pending(), SiteImport.importing()]
* use random s3 keys for imports to avoid collisions (sometimes makes the upload get stuck)
* clamp import date ranges
* site is already in assigns
* recompute cutoff date each time
* use toDate(timestamp[, timezone]) shortcut
* show alreats on export cancel/delete and extract hint into a component
* switch to Imported.clamp_dates/4
* reprocess tables when imports are added
* recompute cutoff_date on each call
* actually use clamped_date_range on submit
* add warning message
* add expiry rules to buckets in make minio
* add site_id to imports notifications and use it in csv_importer
* try/catch safer
* return :ok
* date range is not available when no uploads
* improve ui and warning messages
* use Generic.notice
* fix flaky exports test
* begin tests
* Improve `Importer` notification payload shape
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Move experimental_session_count? logic to within query object
* WIP new querying system for deciding what tables to query
* both -> either
* Include sample_percent in both tables
* Remove a hanging TODO
* Allow filtering by visit props on event queries if flag is on
* Make default sessions join more conditional
* Simplify events_join_sessions?
* Add some TODOs
* Fix assignment
* Handle entry/exit page visit props separately from props stored in events table
* Update test which created sessions/events differently from everyone else
* Make query_events private
* Dont filter by session properties on events table if querying sessions and joining in events
* Handle visits, pageviews, events and visitors metrics from other table
* both -> either
* events, pageviews are strictly event metrics
* Add support for (plain) breakdowns deciding which table to use
* Run tests with experimental_reduced_joins as a separate job
Also refactor which tests are run with postgres:15 to reduce number of jobs
* moduledocs for TableDecider
* Fix matrix
* Custom build name
* Move TEST_EXPERIMENTAL_REDUCED_JOINS check
* Handle percentage separately from other metrics
* Remove debug code
* TableDecider tests
* both => sample_percent
* Improve naming
* Simplify code
* Breakdowns retain old behavior if getting metric visitors
* Unify behavior of entry/exit page hostnames with rest
* Fix test naming
* wip
* more env, setup user before packages to avoid them picking our uid
* make entrypoint.sh executable by all again
* apparently it's a best practice to have executables be owned by root
* make entrypoint executable in COPY
* stop writing to /app, write to /tmp if needed
* fewer changes
* system user
* keep same style for multiline commands
* fewer changes
* add changelog entry
* fix group assignment for plausible user
* use gid=999
* no home
* no home
* add gecos
* add plausible user to nogroup instead of creating a custom one
* eh
* fewer changes
* fewer changes
* fewer changes
* use PERSISTENT_CACHE_DIR instead of STORAGE_DIR
* ignore more
* cleanup
* remove hex timeout env var
* use ERL_FLAGS=+JMsingle true in public builds
* fallback to /tmp and nest under /tzdata_data/ for tzdata
---------
Co-authored-by: Cenk Kücük <cenk@plausible.io>
* add csv importer
* make table validation explicit
* update some docs
* improve docs
* add minio container to ci
* more tests
* eh
* continue
* add passing test
* add failing test
* add config test
* add minio to Makefile
* testcontainers
* remove extra whitespace
* explain the implementation a bit
* account for async deletes in tests
* bounces is UInt32
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Update applications
* Clone community config
* Move modules to experimental dir
* Update runtime config
* Apply first set of compile-time conditionals
* Move funnel schemas to experimental
* Make funnel schema-less build compile
* Use experimental/lib for elixir code
* Move JS funnels to experimental
* Clean up conditional rendering
* Tidy up the pipeline
* Make two builds pass tests without warnings
* Reuse existing dotenvs
* Do a bunch of renames
* Clean up naming
* Run secondary CI
* Update router
* Remove RewriteFunnelDupes migration
Tests were disabled already and it was a one-off shot
* Fixup quota mixins
* Add moduledoc
* Change MIX_ENV for seconary test run
* Skip crm on small
* !fixup
* Exclude flags pipeline
* Update lib/plausible_web/controllers/stats_controller.ex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Bump deps
* Bump stack
* Fix deprecation warnings
* Fix VCR cassettes mismatch due to OTP-18414
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Format & fix flaky tests
* Handle raw IPv4 hostnames; test public suffix TLD
* Configure locus db cache_dir
So that maxmind unavailability doesn't affect
application startup. PERSISTENT_CACHE_DIR env var is used
to point locus at the GeoIP DB file.
* WIP: Remove ExVCR
* Fix test env config
* Fixup exvcr
* Remove exvcr from deps
* Add convert script
* Remove exvcr cassettes
* Remove convert script
* Rename test
* Update moduledoc
* Update dockerfile
* Bump CI cache
* Tag more slow tests, why not?
* Use charlist for locus cache option
* Pin nodejs
* Merge google tests, make them async
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
