analytics/Plausible.Imported.CSVImporter.html

467 lines
21 KiB
HTML
Raw Normal View History

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="x-ua-compatible" content="ie=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="generator" content="ExDoc v0.31.1">
<meta name="project" content="Plausible v0.0.1">
<title>Plausible.Imported.CSVImporter — Plausible v0.0.1</title>
<link rel="stylesheet" href="dist/html-elixir-FM2CSD74.css" />
<script src="dist/handlebars.runtime-NWIB6V2M.js"></script>
<script src="dist/handlebars.templates-43PMFBC7.js"></script>
<script src="dist/sidebar_items-76B64739.js"></script>
<script src="docs_config.js"></script>
<script async src="dist/html-L4O5OK2K.js"></script>
</head>
<body data-type="modules" class="page-module">
<script>
try {
var settings = JSON.parse(localStorage.getItem('ex_doc:settings') || '{}');
if (settings.theme === 'dark' ||
((settings.theme === 'system' || settings.theme == null) &&
window.matchMedia('(prefers-color-scheme: dark)').matches)
) {
document.body.classList.add('dark')
}
} catch (error) { }
</script>
<div class="main">
<button id="sidebar-menu" class="sidebar-button sidebar-toggle" aria-label="toggle sidebar" aria-controls="sidebar">
<i class="ri-menu-line ri-lg" title="Collapse/expand sidebar"></i>
</button>
<div class="background-layer"></div>
<nav id="sidebar" class="sidebar">
<div class="sidebar-header">
<div class="sidebar-projectInfo">
<a href="readme.html" class="sidebar-projectImage">
<img src="assets/logo.png" alt="Plausible" />
</a>
<div>
<a href="readme.html" class="sidebar-projectName" translate="no">
Plausible
</a>
<div class="sidebar-projectVersion" translate="no">
v0.0.1
</div>
</div>
</div>
<ul id="sidebar-listNav" class="sidebar-listNav" role="tablist">
<li>
<button id="extras-list-tab-button" role="tab" data-type="extras" aria-controls="extras-tab-panel" aria-selected="true" tabindex="0">
Pages
</button>
</li>
<li>
<button id="modules-list-tab-button" role="tab" data-type="modules" aria-controls="modules-tab-panel" aria-selected="false" tabindex="-1">
Modules
</button>
</li>
<li>
<button id="tasks-list-tab-button" role="tab" data-type="tasks" aria-controls="tasks-tab-panel" aria-selected="false" tabindex="-1">
<span translate="no">Mix</span> Tasks
</button>
</li>
</ul>
</div>
<div id="extras-tab-panel" class="sidebar-tabpanel" role="tabpanel" aria-labelledby="extras-list-tab-button">
<ul id="extras-full-list" class="full-list"></ul>
</div>
<div id="modules-tab-panel" class="sidebar-tabpanel" role="tabpanel" aria-labelledby="modules-list-tab-button" hidden>
<ul id="modules-full-list" class="full-list"></ul>
</div>
<div id="tasks-tab-panel" class="sidebar-tabpanel" role="tabpanel" aria-labelledby="tasks-list-tab-button" hidden>
<ul id="tasks-full-list" class="full-list"></ul>
</div>
</nav>
<main class="content">
<output role="status" id="toast"></output>
<div class="content-outer">
<div id="content" class="content-inner">
<div class="top-search">
<div class="search-settings">
<form class="search-bar" action="search.html">
<label class="search-label">
<span class="sr-only">Search documentation of Plausible</span>
<input name="q" type="text" class="search-input" placeholder="Search Documentation (press /)" autocomplete="off" autocorrect="off" autocapitalize="off" spellcheck="false" />
</label>
<button type="submit" class="search-button" aria-label="Submit Search">
<i class="ri-search-2-line ri-lg" aria-hidden="true" title="Submit search"></i>
</button>
<button type="button" tabindex="-1" class="search-close-button" aria-hidden="true">
<i class="ri-close-line ri-lg" title="Cancel search"></i>
</button>
</form>
<div class="autocomplete">
</div>
<button class="icon-settings display-settings">
<i class="ri-settings-3-line"></i>
<span class="sr-only">Settings</span>
</button>
</div>
</div>
<h1>
<a href="https://github.com/plausible/analytics/blob/main/lib/plausible/imported/csv_importer.ex#L1" title="View Source" class="icon-action" rel="help">
<i class="ri-code-s-slash-line" aria-hidden="true"></i>
<span class="sr-only">View Source</span>
</a>
<span translate="no">Plausible.Imported.CSVImporter</span>
<small class="app-vsn" translate="no">(Plausible v0.0.1)</small>
</h1>
<section id="moduledoc">
<p>CSV importer from either S3 for which it uses ClickHouse <a href="https://clickhouse.com/docs/en/sql-reference/table-functions/s3">s3 table function</a>
or from local storage for which it uses <a href="https://clickhouse.com/docs/en/sql-reference/table-functions/input">input function.</a></p>
</section>
<section id="summary" class="details-list">
<h1 class="section-heading">
<a class="hover-link" href="#summary">
<i class="ri-link-m" aria-hidden="true"></i>
</a>
<span class="text">Summary</span>
</h1>
<div class="summary-functions summary">
<h2>
<a href="#functions">Functions</a>
</h2>
<div class="summary-row">
<div class="summary-signature">
<a href="#date_range/1" translate="no">date_range(uploads)</a>
</div>
<div class="summary-synopsis"><p>Extracts min/max date range from a list of uploads.</p></div>
</div>
<div class="summary-row">
<div class="summary-signature">
<a href="#extract_table/1" translate="no">extract_table(filename)</a>
</div>
<div class="summary-synopsis"><p>Extracts the table name from the provided filename.</p></div>
</div>
<div class="summary-row">
<div class="summary-signature">
<a href="#local_dir/1" translate="no">local_dir(site_id)</a>
</div>
<div class="summary-synopsis"><p>Returns local directory for CSV imports storage.</p></div>
</div>
<div class="summary-row">
<div class="summary-signature">
<a href="#new_import/3" translate="no">new_import(site, user, opts)</a>
</div>
</div>
<div class="summary-row">
<div class="summary-signature">
<a href="#parse_filename!/1" translate="no">parse_filename!(filename)</a>
</div>
<div class="summary-synopsis"><p>Extracts table name and min/max dates from the filename.</p></div>
</div>
<div class="summary-row">
<div class="summary-signature">
<a href="#valid_filename?/1" translate="no">valid_filename?(filename)</a>
</div>
<div class="summary-synopsis"><p>Checks if the provided filename conforms to the expected format.</p></div>
</div>
</div>
</section>
<section id="functions" class="details-list">
<h1 class="section-heading">
<a class="hover-link" href="#functions">
<i class="ri-link-m" aria-hidden="true"></i>
</a>
<span class="text">Functions</span>
</h1>
<div class="functions-list">
<section class="detail" id="date_range/1">
<div class="detail-header">
<a href="#date_range/1" class="detail-link" title="Link to this function">
<i class="ri-link-m" aria-hidden="true"></i>
<span class="sr-only">Link to this function</span>
</a>
<h1 class="signature" translate="no">date_range(uploads)</h1>
<a href="https://github.com/plausible/analytics/blob/main/lib/plausible/imported/csv_importer.ex#L187" class="icon-action" rel="help" title="View Source">
<i class="ri-code-s-slash-line" aria-hidden="true"></i>
<span class="sr-only">View Source</span>
</a>
</div>
<section class="docstring">
<div class="specs">
<pre translate="no"><span class="attribute">@spec</span> date_range([<a href="https://hexdocs.pm/elixir/String.html#t:t/0">String.t</a>() | %{required(<a href="https://hexdocs.pm/elixir/String.html#t:t/0">String.t</a>()) =&gt; <a href="https://hexdocs.pm/elixir/String.html#t:t/0">String.t</a>()}, ...]) ::
<a href="https://hexdocs.pm/elixir/Date.Range.html#t:t/0">Date.Range.t</a>() | nil</pre>
</div>
<p>Extracts min/max date range from a list of uploads.</p><p>Examples:</p><pre><code class="makeup elixir" translate="no"><span class="gp unselectable">iex&gt; </span><span class="n">date_range</span><span class="p" data-group-id="4471457274-1">(</span><span class="p" data-group-id="4471457274-2">[</span><span class="w">
</span><span class="gp unselectable">...&gt; </span><span class="w"> </span><span class="p" data-group-id="4471457274-3">%{</span><span class="s">&quot;filename&quot;</span><span class="w"> </span><span class="p">=&gt;</span><span class="w"> </span><span class="s">&quot;imported_devices_20190101_20210101.csv&quot;</span><span class="p" data-group-id="4471457274-3">}</span><span class="p">,</span><span class="w">
</span><span class="gp unselectable">...&gt; </span><span class="w"> </span><span class="s">&quot;pages_20200101_20220101.csv&quot;</span><span class="w">
</span><span class="gp unselectable">...&gt; </span><span class="p" data-group-id="4471457274-2">]</span><span class="p" data-group-id="4471457274-1">)</span><span class="w">
</span><span class="nc">Date</span><span class="o">.</span><span class="n">range</span><span class="p" data-group-id="4471457274-4">(</span><span class="ld">~D[2019-01-01]</span><span class="p">,</span><span class="w"> </span><span class="ld">~D[2022-01-01]</span><span class="p" data-group-id="4471457274-4">)</span><span class="w">
</span><span class="gp unselectable">iex&gt; </span><span class="n">date_range</span><span class="p" data-group-id="4471457274-5">(</span><span class="p" data-group-id="4471457274-6">[</span><span class="p" data-group-id="4471457274-6">]</span><span class="p" data-group-id="4471457274-5">)</span><span class="w">
</span><span class="no">nil</span></code></pre>
</section>
</section>
<section class="detail" id="extract_table/1">
<div class="detail-header">
<a href="#extract_table/1" class="detail-link" title="Link to this function">
<i class="ri-link-m" aria-hidden="true"></i>
<span class="sr-only">Link to this function</span>
</a>
<h1 class="signature" translate="no">extract_table(filename)</h1>
<a href="https://github.com/plausible/analytics/blob/main/lib/plausible/imported/csv_importer.ex#L318" class="icon-action" rel="help" title="View Source">
<i class="ri-code-s-slash-line" aria-hidden="true"></i>
<span class="sr-only">View Source</span>
</a>
</div>
<section class="docstring">
<div class="specs">
<pre translate="no"><span class="attribute">@spec</span> extract_table(<a href="https://hexdocs.pm/elixir/String.html#t:t/0">String.t</a>()) :: <a href="https://hexdocs.pm/elixir/String.html#t:t/0">String.t</a>()</pre>
</div>
<p>Extracts the table name from the provided filename.</p><p>Raises if the filename doesn't conform to the expected format.</p><p>Examples:</p><pre><code class="makeup elixir" translate="no"><span class="gp unselectable">iex&gt; </span><span class="n">extract_table</span><span class="p" data-group-id="2538899030-1">(</span><span class="s">&quot;my_data.csv&quot;</span><span class="p" data-group-id="2538899030-1">)</span><span class="w">
</span><span class="gt">** (ArgumentError) invalid filename</span><span class="w">
</span><span class="gp unselectable">iex&gt; </span><span class="n">extract_table</span><span class="p" data-group-id="2538899030-2">(</span><span class="s">&quot;imported_devices_00010101_20250101.csv&quot;</span><span class="p" data-group-id="2538899030-2">)</span><span class="w">
</span><span class="s">&quot;imported_devices&quot;</span><span class="w">
</span><span class="gp unselectable">iex&gt; </span><span class="n">extract_table</span><span class="p" data-group-id="2538899030-3">(</span><span class="s">&quot;devices_00010101_20250101.csv&quot;</span><span class="p" data-group-id="2538899030-3">)</span><span class="w">
</span><span class="s">&quot;imported_devices&quot;</span></code></pre>
</section>
</section>
<section class="detail" id="local_dir/1">
<div class="detail-header">
<a href="#local_dir/1" class="detail-link" title="Link to this function">
<i class="ri-link-m" aria-hidden="true"></i>
<span class="sr-only">Link to this function</span>
</a>
<h1 class="signature" translate="no">local_dir(site_id)</h1>
<a href="https://github.com/plausible/analytics/blob/main/lib/plausible/imported/csv_importer.ex#L335" class="icon-action" rel="help" title="View Source">
<i class="ri-code-s-slash-line" aria-hidden="true"></i>
<span class="sr-only">View Source</span>
</a>
</div>
<section class="docstring">
<p>Returns local directory for CSV imports storage.</p><p>Builds upon <code class="inline">$DATA_DIR</code> or <code class="inline">$PERSISTENT_CACHE_DIR</code> (if set) and falls back to /tmp</p><p>Examples:</p><pre><code class="makeup elixir" translate="no"><span class="gp unselectable">iex&gt; </span><span class="n">local_dir</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">local_dir</span><span class="p" data-group-id="7830264737-1">(</span><span class="c">_site_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">37</span><span class="p" data-group-id="7830264737-1">)</span><span class="w">
</span><span class="gp unselectable">iex&gt; </span><span class="nc">String</span><span class="o">.</span><span class="n">ends_with?</span><span class="p" data-group-id="7830264737-2">(</span><span class="n">local_dir</span><span class="p">,</span><span class="w"> </span><span class="s">&quot;/plausible-imports/37&quot;</span><span class="p" data-group-id="7830264737-2">)</span><span class="w">
</span><span class="no">true</span></code></pre>
</section>
</section>
<section class="detail" id="new_import/3">
<div class="detail-header">
<a href="#new_import/3" class="detail-link" title="Link to this function">
<i class="ri-link-m" aria-hidden="true"></i>
<span class="sr-only">Link to this function</span>
</a>
<h1 class="signature" translate="no">new_import(site, user, opts)</h1>
<a href="https://github.com/plausible/analytics/blob/main/lib/plausible/imported/csv_importer.ex#L7" class="icon-action" rel="help" title="View Source">
<i class="ri-code-s-slash-line" aria-hidden="true"></i>
<span class="sr-only">View Source</span>
</a>
</div>
<section class="docstring">
<div class="specs">
<pre translate="no"><span class="attribute">@spec</span> new_import(<a href="Plausible.Site.html#t:t/0">Plausible.Site.t</a>(), <a href="Plausible.Auth.User.html#t:t/0">Plausible.Auth.User.t</a>(), <a href="https://hexdocs.pm/elixir/Keyword.html#t:t/0">Keyword.t</a>()) ::
{:ok, <a href="https://hexdocs.pm/oban/2.17.2/Oban.Job.html#t:t/0">Oban.Job.t</a>()}
| {:error, <a href="https://hexdocs.pm/ecto/3.11.2/Ecto.Changeset.html#t:t/0">Ecto.Changeset.t</a>() | :import_in_progress | <a href="https://hexdocs.pm/elixir/typespecs.html#basic-types">any</a>()}</pre>
</div>
</section>
</section>
<section class="detail" id="parse_filename!/1">
<div class="detail-header">
<a href="#parse_filename!/1" class="detail-link" title="Link to this function">
<i class="ri-link-m" aria-hidden="true"></i>
<span class="sr-only">Link to this function</span>
</a>
<h1 class="signature" translate="no">parse_filename!(filename)</h1>
<a href="https://github.com/plausible/analytics/blob/main/lib/plausible/imported/csv_importer.ex#L240" class="icon-action" rel="help" title="View Source">
<i class="ri-code-s-slash-line" aria-hidden="true"></i>
<span class="sr-only">View Source</span>
</a>
</div>
<section class="docstring">
<div class="specs">
<pre translate="no"><span class="attribute">@spec</span> parse_filename!(<a href="https://hexdocs.pm/elixir/String.html#t:t/0">String.t</a>()) ::
{table :: <a href="https://hexdocs.pm/elixir/String.html#t:t/0">String.t</a>(), start_date :: <a href="https://hexdocs.pm/elixir/Date.html#t:t/0">Date.t</a>(), end_date :: <a href="https://hexdocs.pm/elixir/Date.html#t:t/0">Date.t</a>()}</pre>
</div>
<p>Extracts table name and min/max dates from the filename.</p><p>Examples:</p><pre><code class="makeup elixir" translate="no"><span class="gp unselectable">iex&gt; </span><span class="n">parse_filename!</span><span class="p" data-group-id="9451571659-1">(</span><span class="s">&quot;my_data.csv&quot;</span><span class="p" data-group-id="9451571659-1">)</span><span class="w">
</span><span class="gt">** (ArgumentError) invalid filename</span><span class="w">
</span><span class="gp unselectable">iex&gt; </span><span class="n">parse_filename!</span><span class="p" data-group-id="9451571659-2">(</span><span class="s">&quot;imported_devices_00010101_20250101.csv&quot;</span><span class="p" data-group-id="9451571659-2">)</span><span class="w">
</span><span class="p" data-group-id="9451571659-3">{</span><span class="s">&quot;imported_devices&quot;</span><span class="p">,</span><span class="w"> </span><span class="ld">~D[0001-01-01]</span><span class="p">,</span><span class="w"> </span><span class="ld">~D[2025-01-01]</span><span class="p" data-group-id="9451571659-3">}</span><span class="w">
</span><span class="gp unselectable">iex&gt; </span><span class="n">parse_filename!</span><span class="p" data-group-id="9451571659-4">(</span><span class="s">&quot;devices_00010101_20250101.csv&quot;</span><span class="p" data-group-id="9451571659-4">)</span><span class="w">
</span><span class="p" data-group-id="9451571659-5">{</span><span class="s">&quot;imported_devices&quot;</span><span class="p">,</span><span class="w"> </span><span class="ld">~D[0001-01-01]</span><span class="p">,</span><span class="w"> </span><span class="ld">~D[2025-01-01]</span><span class="p" data-group-id="9451571659-5">}</span></code></pre>
</section>
</section>
<section class="detail" id="valid_filename?/1">
<div class="detail-header">
<a href="#valid_filename?/1" class="detail-link" title="Link to this function">
<i class="ri-link-m" aria-hidden="true"></i>
<span class="sr-only">Link to this function</span>
</a>
<h1 class="signature" translate="no">valid_filename?(filename)</h1>
<a href="https://github.com/plausible/analytics/blob/main/lib/plausible/imported/csv_importer.ex#L290" class="icon-action" rel="help" title="View Source">
<i class="ri-code-s-slash-line" aria-hidden="true"></i>
<span class="sr-only">View Source</span>
</a>
</div>
<section class="docstring">
<div class="specs">
<pre translate="no"><span class="attribute">@spec</span> valid_filename?(<a href="https://hexdocs.pm/elixir/String.html#t:t/0">String.t</a>()) :: <a href="https://hexdocs.pm/elixir/typespecs.html#built-in-types">boolean</a>()</pre>
</div>
<p>Checks if the provided filename conforms to the expected format.</p><p>Examples:</p><pre><code class="makeup elixir" translate="no"><span class="gp unselectable">iex&gt; </span><span class="n">valid_filename?</span><span class="p" data-group-id="1983947570-1">(</span><span class="s">&quot;my_data.csv&quot;</span><span class="p" data-group-id="1983947570-1">)</span><span class="w">
</span><span class="no">false</span><span class="w">
</span><span class="gp unselectable">iex&gt; </span><span class="n">valid_filename?</span><span class="p" data-group-id="1983947570-2">(</span><span class="s">&quot;imported_devices_00010101_20250101.csv&quot;</span><span class="p" data-group-id="1983947570-2">)</span><span class="w">
</span><span class="no">true</span><span class="w">
</span><span class="gp unselectable">iex&gt; </span><span class="n">valid_filename?</span><span class="p" data-group-id="1983947570-3">(</span><span class="s">&quot;devices_00010101_20250101.csv&quot;</span><span class="p" data-group-id="1983947570-3">)</span><span class="w">
</span><span class="no">true</span></code></pre>
</section>
</section>
</div>
</section>
<footer class="footer">
<p>
<span class="line">
<button class="a-main footer-button display-quick-switch" title="Search HexDocs packages">
Search HexDocs
</button>
<a href="Plausible.epub" title="ePub version">
Download ePub version
</a>
</span>
</p>
<p class="built-using">
Built using
<a href="https://github.com/elixir-lang/ex_doc" title="ExDoc" target="_blank" rel="help noopener" translate="no">ExDoc</a> (v0.31.1) for the
<a href="https://elixir-lang.org" title="Elixir" target="_blank" translate="no">Elixir programming language</a>
</p>
</footer>
</div>
</div>
</main>
</div>
<script src="https://cdn.jsdelivr.net/npm/mermaid/dist/mermaid.min.js"></script>
<script>mermaid.initialize({startOnLoad: true})</script>
</body>
</html>