View Source Plausible.Imported.CSVImporter (Plausible v0.0.1)
CSV importer from either S3 for which it uses ClickHouse s3 table function or from local storage for which it uses input function.
Summary
Functions
Extracts min/max date range from a list of uploads.
Extracts the table name from the provided filename.
Returns local directory for CSV imports storage.
Extracts table name and min/max dates from the filename.
Checks if the provided filename conforms to the expected format.
Functions
@spec date_range([String.t() | %{required(String.t()) => String.t()}, ...]) :: Date.Range.t() | nil
Extracts min/max date range from a list of uploads.
Examples:
iex> date_range([
...> %{"filename" => "imported_devices_20190101_20210101.csv"},
...> "pages_20200101_20220101.csv"
...> ])
Date.range(~D[2019-01-01], ~D[2022-01-01])
iex> date_range([])
nilExtracts the table name from the provided filename.
Raises if the filename doesn't conform to the expected format.
Examples:
iex> extract_table("my_data.csv")
** (ArgumentError) invalid filename
iex> extract_table("imported_devices_00010101_20250101.csv")
"imported_devices"
iex> extract_table("devices_00010101_20250101.csv")
"imported_devices"Returns local directory for CSV imports storage.
Builds upon $DATA_DIR, $PERSISTENT_CACHE_DIR or $DEFAULT_DATA_DIR (if set) and falls back to /tmp.
$DEFAULT_DATA_DIR is set to /var/lib/plausible in container images.
Examples:
iex> local_dir = local_dir(_site_id = 37)
iex> String.ends_with?(local_dir, "/plausible-imports/37")
true@spec new_import(Plausible.Site.t(), Plausible.Auth.User.t(), Keyword.t()) :: {:ok, Oban.Job.t()} | {:error, Ecto.Changeset.t() | :import_in_progress | any()}
@spec parse_filename!(String.t()) :: {table :: String.t(), start_date :: Date.t(), end_date :: Date.t()}
Extracts table name and min/max dates from the filename.
Examples:
iex> parse_filename!("my_data.csv")
** (ArgumentError) invalid filename
iex> parse_filename!("imported_devices_00010101_20250101.csv")
{"imported_devices", ~D[0001-01-01], ~D[2025-01-01]}
iex> parse_filename!("devices_00010101_20250101.csv")
{"imported_devices", ~D[0001-01-01], ~D[2025-01-01]}Checks if the provided filename conforms to the expected format.
Examples:
iex> valid_filename?("my_data.csv")
false
iex> valid_filename?("imported_devices_00010101_20250101.csv")
true
iex> valid_filename?("devices_00010101_20250101.csv")
true