Bring Stats API up to speed: Add conversion_rate to Aggregate and Breakdown (#3739)

* disable event metric with include_imported in every case

* add missing test for metric validation

* refactor metric validation functions

* implement conversion_rate metric validation

* move calculate_cr function into Stats.Util

* Refactor: Move aggregate CR logic into Stats.aggregate

* define atoms to exist

* Ensure that CR does not depend on visitors being queried

If 'visitors' are already queried, we'll use that value. Otherwise we'll
need to make another query to fetch it.

* confirm Stats API aggregate supports CR (tests only)

* small refactor

This is the only 'event_property' left after pattern matching on all
others in the function clauses defined above.

* Make it possible to optionally query conversion_rate

...in breakdown queries (excluding goal and custom prop breakdown)

* A little refactor asking for revenue metrics

1. The `@revenue_metrics` module attribute is an empty list on full build
   anyway
2. We don't need to query for revenue metrics if there are no revenue goals
   returned in the given query (even if revenue goals exist in site.goals)
3. Revenue metrics are already dropped in prop breakdown without a goal
   filter via (get_revenue_tracking_currency/3)

* Make it possible to optionally query conversion_rate (continuation)

... also from a custom prop and goal breakdown

* Frontend adjustments to the Locations report

* Display conversion rate in Regions and Cities (ListReport view)
* Display total conversions, conversions (visitors), and CR in the
  "Details" modals of Countries, Regions, and Cities
* Move the percentage into a separate column in the Countries details table

* confirm Stats API breakdown supports conversion_rate (tests only)

* small refactor: extract maybe_add_time_on_page function

* Make it possible to query cr alone

... (without the visitors metric). Already supported in aggregate, this
commit only implements it for the breakdown API.

* Reuse Stats.Util helper functions from b02db88 for aggregate API

We can follow the same logic as with breakdown for manually adding
`visitors` into the metrics list and taking it out of the response
later on.

That way we don't have to make another query, e.g. in a case where
only pageviews and conversion rate is queried. Also keeps things
consistent.

* changelog update

* fix test after resolving merge conflict

* Use explicit string->atom mapping instead of casting

* alias Util module instead of importing it

* use Enum.empty instead of Enum.any

* improve readability

* rename special_metrics to computed_metrics and explain with a comment

* rename visitors_without_event_filters to total_visitors

* keep a single function for removing unwanted metrics

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

CHANGELOG.md

@@ -2,6 +2,9 @@
 All notable changes to this project will be documented in this file.
 
 ### Added
+- Display `Total visitors`, `Conversions`, and `CR` in the "Details" views of Countries, Regions and Cities (when filtering by a goal)
+- Add `conversion_rate` to Regions and Cities reports (when filtering by a goal)
+- Add the `conversion_rate` metric to Stats API Breakdown and Aggregate endpoints
 - IP Block List in Site Settings
 - Allow filtering with `contains`/`matches` operator for Sources, Browsers and Operating Systems.
 - Allow filtering by multiple custom properties

assets/js/dashboard/router.js

@@ -52,7 +52,7 @@ export default function Router({ site, loggedIn, currentUserRole }) {
             <ExitPagesModal site={site} />
           </Route>
           <Route path="/:domain/countries">
-            <ModalTable title="Top countries" site={site} endpoint={url.apiPath(site, '/countries')} filter={{ country: 'code', country_labels: 'name' }} keyLabel="Country" renderIcon={renderCountryIcon} />
+            <ModalTable title="Top countries" site={site} endpoint={url.apiPath(site, '/countries')} filter={{ country: 'code', country_labels: 'name' }} keyLabel="Country" renderIcon={renderCountryIcon} showPercentage={true}/>
           </Route>
           <Route path="/:domain/regions">
             <ModalTable title="Top regions" site={site} endpoint={url.apiPath(site, '/regions')} filter={{ region: 'code', region_labels: 'name' }} keyLabel="Region" renderIcon={renderRegionIcon} />

assets/js/dashboard/stats/locations/index.js

@@ -57,7 +57,7 @@ function Regions({query, site, onClick}) {
       getFilterFor={getFilterFor}
       onClick={onClick}
       keyLabel="Region"
-      metrics={[VISITORS_METRIC]}
+      metrics={maybeWithCR([VISITORS_METRIC], query)}
       detailsLink={sitePath(site, '/regions')}
       query={query}
       renderIcon={renderIcon}
@@ -84,7 +84,7 @@ function Cities({query, site}) {
       fetchData={fetchData}
       getFilterFor={getFilterFor}
       keyLabel="City"
-      metrics={[VISITORS_METRIC]}
+      metrics={maybeWithCR([VISITORS_METRIC], query)}
       detailsLink={sitePath(site, '/cities')}
       query={query}
       renderIcon={renderIcon}

assets/js/dashboard/stats/modals/table.js

@@ -20,8 +20,24 @@ class ModalTable extends React.Component {
       .then((res) => this.setState({loading: false, list: res}))
   }
 
+  showConversionRate() {
+    return !!this.state.query.filters.goal
+  }
+
+  showPercentage() {
+    return this.props.showPercentage && !this.showConversionRate()
+  }
+
   label() {
-    return this.state.query.period === 'realtime' ? 'Current visitors' : 'Visitors'
+    if (this.state.query.period === 'realtime') {
+      return 'Current visitors'
+    }
+
+    if (this.showConversionRate()) {
+      return 'Conversions'
+    }
+
+    return 'Visitors'
   }
 
   renderTableItem(tableItem) {
@@ -40,11 +56,10 @@ class ModalTable extends React.Component {
             {tableItem.name}
           </Link>
         </td>
-        <td className="p-2 w-32 font-medium" align="right">
+        {this.showConversionRate() && <td className="p-2 w-32 font-medium" align="right">{numberFormatter(tableItem.total_visitors)}</td>}
-          {numberFormatter(tableItem.visitors)}
+        <td className="p-2 w-32 font-medium" align="right">{numberFormatter(tableItem.visitors)}</td>
-          {tableItem.percentage >= 0 &&
+        {this.showPercentage() && <td className="p-2 w-32 font-medium" align="right">{tableItem.percentage}</td>}
-            <span className="inline-block text-xs w-8 pl-1 text-right">({tableItem.percentage}%)</span> }
+        {this.showConversionRate() && <td className="p-2 w-32 font-medium" align="right">{numberFormatter(tableItem.conversion_rate)}%</td>}
-        </td>
       </tr>
     )
   }
@@ -66,19 +81,11 @@ class ModalTable extends React.Component {
             <table className="w-max overflow-x-auto md:w-full table-striped table-fixed">
               <thead>
                 <tr>
-                  <th
+                  <th className="p-2 w-48 md:w-56 lg:w-1/3 text-xs tracking-wide font-bold text-gray-500 dark:text-gray-400" align="left">{this.props.keyLabel}</th>
-                    className="p-2 w-48 lg:w-1/2 text-xs tracking-wide font-bold text-gray-500 dark:text-gray-400"
+                  {this.showConversionRate() && <th className="p-2 w-32 text-xs tracking-wide font-bold text-gray-500 dark:text-gray-400" align="right" >Total Visitors</th>}
-                    align="left"
+                  <th className="p-2 w-32 text-xs tracking-wide font-bold text-gray-500 dark:text-gray-400" align="right">{this.label()}</th>
-                  >
+                  {this.showPercentage() && <th className="p-2 w-32 text-xs tracking-wide font-bold text-gray-500 dark:text-gray-400" align="right">%</th>}
-                    {this.props.keyLabel}
+                  {this.showConversionRate() && <th className="p-2 w-32 text-xs tracking-wide font-bold text-gray-500 dark:text-gray-400" align="right">CR</th>}
-                  </th>
-                  <th
-                    // eslint-disable-next-line max-len
-                    className="p-2 w-32 lg:w-1/2 text-xs tracking-wide font-bold text-gray-500 dark:text-gray-400"
-                    align="right"
-                  >
-                    {this.label()}
-                  </th>
                 </tr>
               </thead>
               <tbody>

lib/plausible/stats/aggregate.ex

@@ -1,9 +1,9 @@
 defmodule Plausible.Stats.Aggregate do
-  alias Plausible.Stats.Query
   use Plausible.ClickhouseRepo
   use Plausible
-  import Plausible.Stats.{Base, Imported, Util}
+  import Plausible.Stats.{Base, Imported}
   import Ecto.Query
+  alias Plausible.Stats.{Query, Util}
 
   @revenue_metrics on_full_build(do: Plausible.Stats.Goal.Revenue.revenue_metrics(), else: [])
 
@@ -26,8 +26,13 @@ defmodule Plausible.Stats.Aggregate do
 
     Query.trace(query, metrics)
 
-    event_metrics = Enum.filter(metrics, &(&1 in @event_metrics))
+    event_metrics =
+      metrics
+      |> Util.maybe_add_visitors_metric()
+      |> Enum.filter(&(&1 in @event_metrics))
+
     event_task = fn -> aggregate_events(site, query, event_metrics) end
+
     session_metrics = Enum.filter(metrics, &(&1 in @session_metrics))
     session_task = fn -> aggregate_sessions(site, query, session_metrics) end
 
@@ -40,12 +45,38 @@ defmodule Plausible.Stats.Aggregate do
 
     Plausible.ClickhouseRepo.parallel_tasks([session_task, event_task, time_on_page_task])
     |> Enum.reduce(%{}, fn aggregate, task_result -> Map.merge(aggregate, task_result) end)
+    |> maybe_put_cr(site, query, metrics)
+    |> Util.keep_requested_metrics(metrics)
     |> cast_revenue_metrics_to_money(currency)
     |> Enum.map(&maybe_round_value/1)
     |> Enum.map(fn {metric, value} -> {metric, %{value: value}} end)
     |> Enum.into(%{})
   end
 
+  defp maybe_put_cr(aggregate_result, site, query, metrics) do
+    if :conversion_rate in metrics do
+      all =
+        query
+        |> Query.remove_event_filters([:goal, :props])
+        |> then(fn query -> aggregate_events(site, query, [:visitors]) end)
+        |> Map.fetch!(:visitors)
+
+      converted = aggregate_result.visitors
+
+      cr = Util.calculate_cr(all, converted)
+
+      aggregate_result = Map.put(aggregate_result, :conversion_rate, cr)
+
+      if :total_visitors in metrics do
+        Map.put(aggregate_result, :total_visitors, all)
+      else
+        aggregate_result
+      end
+    else
+      aggregate_result
+    end
+  end
+
   defp aggregate_events(_, _, []), do: %{}
 
   defp aggregate_events(site, query, metrics) do
@@ -63,7 +94,7 @@ defmodule Plausible.Stats.Aggregate do
     |> select_session_metrics(metrics, query)
     |> merge_imported(site, query, :aggregate, metrics)
     |> ClickhouseRepo.one()
-    |> remove_internal_visits_metric()
+    |> Util.keep_requested_metrics(metrics)
   end
 
   defp aggregate_time_on_page(site, query) do

lib/plausible/stats/breakdown.ex

@@ -3,9 +3,9 @@ defmodule Plausible.Stats.Breakdown do
   use Plausible
   use Plausible.Stats.Fragments
 
-  import Plausible.Stats.{Base, Imported, Util}
+  import Plausible.Stats.{Base, Imported}
   require OpenTelemetry.Tracer, as: Tracer
-  alias Plausible.Stats.Query
+  alias Plausible.Stats.{Query, Util}
 
   @no_ref "Direct / None"
   @not_set "(not set)"
@@ -16,7 +16,12 @@ defmodule Plausible.Stats.Breakdown do
 
   @event_metrics [:visitors, :pageviews, :events] ++ @revenue_metrics
 
-  @event_props Plausible.Stats.Props.event_props()
+  # These metrics can be asked from the `breakdown/5` function,
+  # but they are different from regular metrics such as `visitors`,
+  # or `bounce_rate` - we cannot currently "select them" directly in
+  # the db queries. Instead, we need to artificially append them to
+  # the breakdown results later on.
+  @computed_metrics [:conversion_rate, :total_visitors]
 
   def breakdown(site, query, property, metrics, pagination, opts \\ [])
 
@@ -29,15 +34,22 @@ defmodule Plausible.Stats.Breakdown do
 
     if !Keyword.get(opts, :skip_tracing), do: trace(query, property, metrics)
 
+    {revenue_goals, metrics} =
+      if full_build?() && Plausible.Billing.Feature.RevenueGoals.enabled?(site) do
+        revenue_goals = Enum.filter(event_goals, &Plausible.Goal.Revenue.revenue?/1)
+        metrics = if Enum.empty?(revenue_goals), do: metrics -- @revenue_metrics, else: metrics
+
+        {revenue_goals, metrics}
+      else
+        {nil, metrics -- @revenue_metrics}
+      end
+
+    metrics_to_select = Util.maybe_add_visitors_metric(metrics) -- @computed_metrics
+
     event_results =
       if Enum.any?(event_goals) do
-        revenue_goals =
-          on_full_build do
-            Enum.filter(event_goals, &Plausible.Goal.Revenue.revenue?/1)
-          end
-
         site
-        |> breakdown(event_query, "event:name", metrics, pagination, skip_tracing: true)
+        |> breakdown(event_query, "event:name", metrics_to_select, pagination, skip_tracing: true)
         |> transform_keys(%{name: :goal})
         |> cast_revenue_metrics_to_money(revenue_goals)
       else
@@ -68,14 +80,16 @@ defmodule Plausible.Stats.Breakdown do
             goal: fragment("concat('Visit ', ?[index])", ^page_exprs)
           }
         )
-        |> select_event_metrics(metrics -- @revenue_metrics)
+        |> select_event_metrics(metrics_to_select -- @revenue_metrics)
         |> ClickhouseRepo.all()
         |> Enum.map(fn row -> Map.delete(row, :index) end)
       else
         []
       end
 
-    zip_results(event_results, page_results, :goal, metrics)
+    zip_results(event_results, page_results, :goal, metrics_to_select)
+    |> maybe_add_cr(site, query, nil, metrics)
+    |> Util.keep_requested_metrics(metrics)
   end
 
   def breakdown(site, query, "event:props:" <> custom_prop = property, metrics, pagination, opts) do
@@ -86,6 +100,8 @@ defmodule Plausible.Stats.Breakdown do
         {nil, metrics}
       end
 
+    metrics_to_select = Util.maybe_add_visitors_metric(metrics) -- @computed_metrics
+
     {_limit, page} = pagination
 
     none_result =
@@ -97,7 +113,7 @@ defmodule Plausible.Stats.Breakdown do
           select_merge: %{^custom_prop => "(none)"},
           having: fragment("uniq(?)", e.user_id) > 0
         )
-        |> select_event_metrics(metrics)
+        |> select_event_metrics(metrics_to_select)
         |> ClickhouseRepo.all()
       else
         []
@@ -105,29 +121,28 @@ defmodule Plausible.Stats.Breakdown do
 
     if !Keyword.get(opts, :skip_tracing), do: trace(query, property, metrics)
 
-    breakdown_events(site, query, "event:props:" <> custom_prop, metrics, pagination)
+    breakdown_events(site, query, "event:props:" <> custom_prop, metrics_to_select, pagination)
     |> Kernel.++(none_result)
     |> Enum.map(&cast_revenue_metrics_to_money(&1, currency))
-    |> Enum.sort_by(& &1[sorting_key(metrics)], :desc)
+    |> Enum.sort_by(& &1[sorting_key(metrics_to_select)], :desc)
+    |> maybe_add_cr(site, query, nil, metrics)
+    |> Util.keep_requested_metrics(metrics)
   end
 
   def breakdown(site, query, "event:page" = property, metrics, pagination, opts) do
-    event_metrics = Enum.filter(metrics, &(&1 in @event_metrics))
+    event_metrics =
-    session_metrics = Enum.filter(metrics, &(&1 in @session_metrics))
+      metrics
-
+      |> Util.maybe_add_visitors_metric()
-    event_result = breakdown_events(site, query, "event:page", event_metrics, pagination)
+      |> Enum.filter(&(&1 in @event_metrics))
 
     event_result =
-      if :time_on_page in metrics do
+      site
-        pages = Enum.map(event_result, & &1[:page])
+      |> breakdown_events(query, "event:page", event_metrics, pagination)
-        time_on_page_result = breakdown_time_on_page(site, query, pages)
+      |> maybe_add_time_on_page(site, query, metrics)
+      |> maybe_add_cr(site, query, property, metrics)
+      |> Util.keep_requested_metrics(metrics)
 
-        Enum.map(event_result, fn row ->
+    session_metrics = Enum.filter(metrics, &(&1 in @session_metrics))
-          Map.put(row, :time_on_page, time_on_page_result[row[:page]])
-        end)
-      else
-        event_result
-      end
 
     new_query =
       case event_result do
@@ -161,14 +176,19 @@ defmodule Plausible.Stats.Breakdown do
     end
   end
 
-  def breakdown(site, query, property, metrics, pagination, opts) when property in @event_props do
+  def breakdown(site, query, "event:name" = property, metrics, pagination, opts) do
     if !Keyword.get(opts, :skip_tracing), do: trace(query, property, metrics)
     breakdown_events(site, query, property, metrics, pagination)
   end
 
   def breakdown(site, query, property, metrics, pagination, opts) do
     if !Keyword.get(opts, :skip_tracing), do: trace(query, property, metrics)
-    breakdown_sessions(site, query, property, metrics, pagination)
+
+    metrics_to_select = Util.maybe_add_visitors_metric(metrics) -- @computed_metrics
+
+    breakdown_sessions(site, query, property, metrics_to_select, pagination)
+    |> maybe_add_cr(site, query, property, metrics)
+    |> Util.keep_requested_metrics(metrics)
   end
 
   defp zip_results(event_result, session_result, property, metrics) do
@@ -211,7 +231,7 @@ defmodule Plausible.Stats.Breakdown do
     |> apply_pagination(pagination)
     |> ClickhouseRepo.all()
     |> transform_keys(%{operating_system: :os})
-    |> remove_internal_visits_metric(metrics)
+    |> Util.keep_requested_metrics(metrics)
   end
 
   defp breakdown_events(_, _, _, [], _), do: []
@@ -229,6 +249,19 @@ defmodule Plausible.Stats.Breakdown do
     |> transform_keys(%{operating_system: :os})
   end
 
+  defp maybe_add_time_on_page(event_results, site, query, metrics) do
+    if :time_on_page in metrics do
+      pages = Enum.map(event_results, & &1[:page])
+      time_on_page_result = breakdown_time_on_page(site, query, pages)
+
+      Enum.map(event_results, fn row ->
+        Map.put(row, :time_on_page, time_on_page_result[row[:page]])
+      end)
+    else
+      event_results
+    end
+  end
+
   defp breakdown_time_on_page(_site, _query, []) do
     %{}
   end
@@ -626,6 +659,82 @@ defmodule Plausible.Stats.Breakdown do
     )
   end
 
+  defp maybe_add_cr(breakdown_results, site, query, property, metrics) do
+    cond do
+      :conversion_rate not in metrics -> breakdown_results
+      Enum.empty?(breakdown_results) -> breakdown_results
+      is_nil(property) -> add_absolute_cr(breakdown_results, site, query)
+      true -> add_cr(breakdown_results, site, query, property, metrics)
+    end
+  end
+
+  # This function injects a conversion_rate metric into every
+  # breakdown result map. It is calculated as X / Y, where:
+  #
+  #   * X is the number of conversions for a breakdown
+  #     result (conversion = number of visitors who
+  #     completed the filtered goal with the filtered
+  #     custom properties).
+  #
+  #  * Y is the number of all visitors for this breakdown
+  #    result without the `event:goal` and `event:props:*`
+  #    filters.
+  defp add_cr(breakdown_results, site, query, property, metrics) do
+    property_atom = Plausible.Stats.Filters.without_prefix(property)
+
+    items =
+      Enum.map(breakdown_results, fn item -> Map.fetch!(item, property_atom) end)
+
+    query_without_goal =
+      query
+      |> Query.put_filter(property, {:member, items})
+      |> Query.remove_event_filters([:goal, :props])
+
+    # Here, we're always only interested in the first page of results
+    # - the :member filter makes sure that the results always match with
+    # the items in the given breakdown_results list
+    pagination = {length(items), 1}
+
+    res_without_goal = breakdown(site, query_without_goal, property, [:visitors], pagination)
+
+    Enum.map(breakdown_results, fn item ->
+      without_goal =
+        Enum.find(res_without_goal, fn s ->
+          Map.fetch!(s, property_atom) == Map.fetch!(item, property_atom)
+        end)
+
+      item =
+        item
+        |> Map.put(:conversion_rate, Util.calculate_cr(without_goal.visitors, item.visitors))
+
+      if :total_visitors in metrics do
+        Map.put(item, :total_visitors, without_goal.visitors)
+      else
+        item
+      end
+    end)
+  end
+
+  # Similar to `add_cr/5`, injects a conversion_rate metric into
+  # every breakdown result. However, a single divisor is used in
+  # the CR calculation across all breakdown results. That is the
+  # number of visitors without `event:goal` and `event:props:*`
+  # filters.
+  #
+  # This is useful when we're only interested in the conversions
+  # themselves - not how well a certain property such as browser
+  # or page converted.
+  defp add_absolute_cr(breakdown_results, site, query) do
+    total_q = Query.remove_event_filters(query, [:goal, :props])
+
+    %{visitors: %{value: total_visitors}} = Plausible.Stats.aggregate(site, total_q, [:visitors])
+
+    breakdown_results
+    |> Enum.map(fn goal ->
+      Map.put(goal, :conversion_rate, Util.calculate_cr(total_visitors, goal[:visitors]))
+    end)
+  end
+
   defp sorting_key(metrics) do
     if Enum.member?(metrics, :visitors), do: :visitors, else: List.first(metrics)
   end

lib/plausible/stats/filters/filters.ex

@@ -6,34 +6,30 @@ defmodule Plausible.Stats.Filters do
   alias Plausible.Stats.Filters.{DashboardFilterParser, StatsAPIFilterParser}
 
   @visit_props [
-    "source",
+    :source,
-    "referrer",
+    :referrer,
-    "utm_medium",
+    :utm_medium,
-    "utm_source",
+    :utm_source,
-    "utm_campaign",
+    :utm_campaign,
-    "utm_content",
+    :utm_content,
-    "utm_term",
+    :utm_term,
-    "screen",
+    :screen,
-    "device",
+    :device,
-    "browser",
+    :browser,
-    "browser_version",
+    :browser_version,
-    "os",
+    :os,
-    "os_version",
+    :os_version,
-    "country",
+    :country,
-    "region",
+    :region,
-    "city",
+    :city,
-    "entry_page",
+    :entry_page,
-    "exit_page"
+    :exit_page
   ]
-  def visit_props(), do: @visit_props
+  def visit_props(), do: @visit_props |> Enum.map(&to_string/1)
 
-  @event_props [
+  @event_props [:name, :page, :goal]
-    "name",
-    "page",
-    "goal"
-  ]
 
-  def event_props(), do: @event_props
+  def event_props(), do: @event_props |> Enum.map(&to_string/1)
 
   @doc """
   Parses different filter formats.
@@ -67,4 +63,11 @@ defmodule Plausible.Stats.Filters do
 
   def parse(filters) when is_map(filters), do: DashboardFilterParser.parse_and_prefix(filters)
   def parse(_), do: %{}
+
+  def without_prefix(property) do
+    property
+    |> String.split(":")
+    |> List.last()
+    |> String.to_existing_atom()
+  end
 end

lib/plausible/stats/timeseries.ex

@@ -1,8 +1,8 @@
 defmodule Plausible.Stats.Timeseries do
   use Plausible.ClickhouseRepo
   use Plausible
-  alias Plausible.Stats.Query
+  alias Plausible.Stats.{Query, Util}
-  import Plausible.Stats.{Base, Util}
+  import Plausible.Stats.{Base}
   use Plausible.Stats.Fragments
 
   @typep metric ::
@@ -69,7 +69,7 @@ defmodule Plausible.Stats.Timeseries do
     |> select_session_metrics(metrics, query)
     |> Plausible.Stats.Imported.merge_imported_timeseries(site, query, metrics)
     |> ClickhouseRepo.all()
-    |> remove_internal_visits_metric(metrics)
+    |> Util.keep_requested_metrics(metrics)
   end
 
   defp buckets(%Query{interval: "month"} = query) do

lib/plausible/stats/util.ex

@@ -3,21 +3,59 @@ defmodule Plausible.Stats.Util do
   Utilities for modifying stat results
   """
 
+  @manually_removable_metrics [:__internal_visits, :visitors]
+
   @doc """
-  `__internal_visits` is fetched when querying bounce rate and visit duration, as it
+  Sometimes we need to manually add metrics in order to calculate the value for
-  is needed to calculate these from imported data. This function removes that metric
+  other metrics. E.g:
-  from all entries in the results list.
+
+  * `__internal_visits` is fetched when querying bounce rate and visit duration,
+    as it is needed to calculate these from imported data.
+
+  * `visitors` metric might be added manually via `maybe_add_visitors_metric/1`,
+    in order to be able to calculate conversion rate.
+
+  This function can be used for stripping those metrics from a breakdown (list),
+  or an aggregate (map) result. We do not want to return metrics that we're not
+  requested.
   """
-  def remove_internal_visits_metric(results, metrics) when is_list(results) do
+  def keep_requested_metrics(results, requested_metrics) when is_list(results) do
-    if :bounce_rate in metrics or :visit_duration in metrics do
+    Enum.map(results, fn results_map ->
-      results
+      keep_requested_metrics(results_map, requested_metrics)
-      |> Enum.map(&remove_internal_visits_metric/1)
+    end)
+  end
+
+  def keep_requested_metrics(results, requested_metrics) do
+    Map.drop(results, @manually_removable_metrics -- requested_metrics)
+  end
+
+  @doc """
+  This function adds the `visitors` metric into the list of
+  given metrics if it's not already there and if there is a
+  `conversion_rate` metric in the list.
+
+  Currently, the conversion rate cannot be queried from the
+  database with a simple select clause - instead, we need to
+  fetch the database result first, and then manually add it
+  into the aggregate map or every entry of thebreakdown list.
+
+  In order for us to be able to calculate it based on the
+  results returned by the database query, the visitors metric
+  needs to be queried.
+  """
+  def maybe_add_visitors_metric(metrics) do
+    if :conversion_rate in metrics and :visitors not in metrics do
+      metrics ++ [:visitors]
     else
-      results
+      metrics
     end
   end
 
-  def remove_internal_visits_metric(result) when is_map(result) do
+  def calculate_cr(nil, _converted_visitors), do: nil
-    Map.delete(result, :__internal_visits)
+
+  def calculate_cr(unique_visitors, converted_visitors) do
+    if unique_visitors > 0,
+      do: Float.round(converted_visitors / unique_visitors * 100, 1),
+      else: 0.0
   end
 end

lib/plausible_web/controllers/api/external_stats_controller.ex

@@ -4,6 +4,19 @@ defmodule PlausibleWeb.Api.ExternalStatsController do
   use PlausibleWeb.Plugs.ErrorHandler
   alias Plausible.Stats.Query
 
+  @metrics [
+    :visitors,
+    :visits,
+    :pageviews,
+    :views_per_visit,
+    :bounce_rate,
+    :visit_duration,
+    :events,
+    :conversion_rate
+  ]
+
+  @metric_mappings Enum.into(@metrics, %{}, fn metric -> {to_string(metric), metric} end)
+
   def realtime_visitors(conn, _params) do
     site = conn.assigns.site
     query = Query.from(site, %{"period" => "realtime"})
@@ -96,13 +109,6 @@ defmodule PlausibleWeb.Api.ExternalStatsController do
   @default_breakdown_limit 100
   defp validate_or_default_limit(_), do: {:ok, @default_breakdown_limit}
 
-  defp event_only_property?("event:name"), do: true
-  defp event_only_property?("event:goal"), do: true
-  defp event_only_property?("event:props:" <> _), do: true
-  defp event_only_property?(_), do: false
-
-  @event_metrics ["visitors", "pageviews", "events"]
-  @session_metrics ["visits", "bounce_rate", "visit_duration", "views_per_visit"]
   defp parse_and_validate_metrics(params, property, query) do
     metrics =
       Map.get(params, "metrics", "visitors")
@@ -113,7 +119,7 @@ defmodule PlausibleWeb.Api.ExternalStatsController do
         {:error, reason}
 
       metrics ->
-        {:ok, Enum.map(metrics, &String.to_existing_atom/1)}
+        {:ok, Enum.map(metrics, &Map.fetch!(@metric_mappings, &1))}
     end
   end
 
@@ -155,26 +161,61 @@ defmodule PlausibleWeb.Api.ExternalStatsController do
     end)
   end
 
-  defp validate_metric("events", nil, %{include_imported: true}) do
+  defp validate_metric("conversion_rate" = metric, property, query) do
-    {:error, "Metric `events` cannot be queried with imported data"}
+    cond do
+      property == "event:goal" ->
+        {:ok, metric}
+
+      query.filters["event:goal"] ->
+        {:ok, metric}
+
+      true ->
+        {:error,
+         "Metric `#{metric}` can only be queried in a goal breakdown or with a goal filter"}
+    end
   end
 
-  defp validate_metric(metric, _, _) when metric in @event_metrics, do: {:ok, metric}
+  defp validate_metric("events" = metric, _, query) do
+    if query.include_imported do
+      {:error, "Metric `#{metric}` cannot be queried with imported data"}
+    else
+      {:ok, metric}
+    end
+  end
 
-  defp validate_metric(metric, property, query) when metric in @session_metrics do
+  defp validate_metric(metric, _, _) when metric in ["visitors", "pageviews"] do
-    event_only_filter = Map.keys(query.filters) |> Enum.find(&event_only_property?/1)
+    {:ok, metric}
+  end
 
+  defp validate_metric("views_per_visit" = metric, property, query) do
     cond do
-      metric == "views_per_visit" && query.filters["event:page"] ->
+      query.filters["event:page"] ->
         {:error, "Metric `#{metric}` cannot be queried with a filter on `event:page`."}
 
-      metric == "views_per_visit" && property != nil ->
+      property != nil ->
         {:error, "Metric `#{metric}` is not supported in breakdown queries."}
 
+      true ->
+        validate_session_metric(metric, property, query)
+    end
+  end
+
+  defp validate_metric(metric, property, query)
+       when metric in ["visits", "bounce_rate", "visit_duration"] do
+    validate_session_metric(metric, property, query)
+  end
+
+  defp validate_metric(metric, _, _) do
+    {:error,
+     "The metric `#{metric}` is not recognized. Find valid metrics from the documentation: https://plausible.io/docs/stats-api#metrics"}
+  end
+
+  defp validate_session_metric(metric, property, query) do
+    cond do
       event_only_property?(property) ->
         {:error, "Session metric `#{metric}` cannot be queried for breakdown by `#{property}`."}
 
-      event_only_filter ->
+      event_only_filter = find_event_only_filter(query) ->
         {:error,
          "Session metric `#{metric}` cannot be queried when using a filter on `#{event_only_filter}`."}
 
@@ -183,11 +224,15 @@ defmodule PlausibleWeb.Api.ExternalStatsController do
     end
   end
 
-  defp validate_metric(metric, _, _) do
+  defp find_event_only_filter(query) do
-    {:error,
+    Map.keys(query.filters) |> Enum.find(&event_only_property?/1)
-     "The metric `#{metric}` is not recognized. Find valid metrics from the documentation: https://plausible.io/docs/stats-api#metrics"}
   end
 
+  defp event_only_property?("event:name"), do: true
+  defp event_only_property?("event:goal"), do: true
+  defp event_only_property?("event:props:" <> _), do: true
+  defp event_only_property?(_), do: false
+
   def timeseries(conn, params) do
     site = Repo.preload(conn.assigns.site, :owner)
 

lib/plausible_web/controllers/api/stats_controller.ex

@@ -346,61 +346,23 @@ defmodule PlausibleWeb.Api.StatsController do
   end
 
   defp fetch_top_stats(site, %Query{filters: %{"event:goal" => _}} = query, comparison_query) do
-    query_without_filters = Query.remove_event_filters(query, [:goal, :props])
+    metrics =
-    metrics = [:visitors, :events] ++ @revenue_metrics
+      [:total_visitors, :visitors, :events, :conversion_rate] ++ @revenue_metrics
 
-    results_without_filters =
+    results = Stats.aggregate(site, query, metrics)
-      site
+    comparison = if comparison_query, do: Stats.aggregate(site, comparison_query, metrics)
-      |> Stats.aggregate(query_without_filters, [:visitors])
-      |> transform_keys(%{visitors: :unique_visitors})
-
-    results =
-      site
-      |> Stats.aggregate(query, metrics)
-      |> transform_keys(%{visitors: :converted_visitors, events: :completions})
-      |> Map.merge(results_without_filters)
-
-    comparison =
-      if comparison_query do
-        comparison_query_without_filters =
-          Query.remove_event_filters(comparison_query, [:goal, :props])
-
-        comparison_without_filters =
-          site
-          |> Stats.aggregate(comparison_query_without_filters, [:visitors])
-          |> transform_keys(%{visitors: :unique_visitors})
-
-        site
-        |> Stats.aggregate(comparison_query, metrics)
-        |> transform_keys(%{visitors: :converted_visitors, events: :completions})
-        |> Map.merge(comparison_without_filters)
-      end
-
-    conversion_rate = %{
-      cr: %{value: calculate_cr(results.unique_visitors.value, results.converted_visitors.value)}
-    }
-
-    comparison_conversion_rate =
-      if comparison do
-        value =
-          calculate_cr(comparison.unique_visitors.value, comparison.converted_visitors.value)
-
-        %{cr: %{value: value}}
-      else
-        nil
-      end
 
     [
-      top_stats_entry(results, comparison, "Unique visitors", :unique_visitors),
+      top_stats_entry(results, comparison, "Unique visitors", :total_visitors),
-      top_stats_entry(results, comparison, "Unique conversions", :converted_visitors),
+      top_stats_entry(results, comparison, "Unique conversions", :visitors),
-      top_stats_entry(results, comparison, "Total conversions", :completions),
+      top_stats_entry(results, comparison, "Total conversions", :events),
       on_full_build do
         top_stats_entry(results, comparison, "Average revenue", :average_revenue, &format_money/1)
       end,
       on_full_build do
         top_stats_entry(results, comparison, "Total revenue", :total_revenue, &format_money/1)
       end,
-      top_stats_entry(conversion_rate, comparison_conversion_rate, "Conversion rate", :cr)
+      top_stats_entry(results, comparison, "Conversion rate", :conversion_rate)
     ]
     |> Enum.reject(&is_nil/1)
     |> then(&{&1, 100})
@@ -469,17 +431,16 @@ defmodule PlausibleWeb.Api.StatsController do
 
   def sources(conn, params) do
     site = conn.assigns[:site]
-
     query = Query.from(site, params)
-
     pagination = parse_pagination(params)
 
-    metrics =
+    extra_metrics =
-      if params["detailed"], do: [:visitors, :bounce_rate, :visit_duration], else: [:visitors]
+      if params["detailed"], do: [:bounce_rate, :visit_duration], else: []
+
+    metrics = breakdown_metrics(query, extra_metrics)
 
     res =
       Stats.breakdown(site, query, "visit:source", metrics, pagination)
-      |> add_cr(site, query, pagination, :source, "visit:source")
       |> transform_keys(%{source: :name})
 
     if params["csv"] do
@@ -552,16 +513,12 @@ defmodule PlausibleWeb.Api.StatsController do
 
   def utm_mediums(conn, params) do
     site = conn.assigns[:site]
-
     query = Query.from(site, params)
-
     pagination = parse_pagination(params)
-
+    metrics = breakdown_metrics(query, [:bounce_rate, :visit_duration])
-    metrics = [:visitors, :bounce_rate, :visit_duration]
 
     res =
       Stats.breakdown(site, query, "visit:utm_medium", metrics, pagination)
-      |> add_cr(site, query, pagination, :utm_medium, "visit:utm_medium")
       |> transform_keys(%{utm_medium: :name})
 
     if params["csv"] do
@@ -579,16 +536,12 @@ defmodule PlausibleWeb.Api.StatsController do
 
   def utm_campaigns(conn, params) do
     site = conn.assigns[:site]
-
     query = Query.from(site, params)
-
     pagination = parse_pagination(params)
-
+    metrics = breakdown_metrics(query, [:bounce_rate, :visit_duration])
-    metrics = [:visitors, :bounce_rate, :visit_duration]
 
     res =
       Stats.breakdown(site, query, "visit:utm_campaign", metrics, pagination)
-      |> add_cr(site, query, pagination, :utm_campaign, "visit:utm_campaign")
       |> transform_keys(%{utm_campaign: :name})
 
     if params["csv"] do
@@ -606,15 +559,12 @@ defmodule PlausibleWeb.Api.StatsController do
 
   def utm_contents(conn, params) do
     site = conn.assigns[:site]
-
     query = Query.from(site, params)
-
     pagination = parse_pagination(params)
-    metrics = [:visitors, :bounce_rate, :visit_duration]
+    metrics = breakdown_metrics(query, [:bounce_rate, :visit_duration])
 
     res =
       Stats.breakdown(site, query, "visit:utm_content", metrics, pagination)
-      |> add_cr(site, query, pagination, :utm_content, "visit:utm_content")
       |> transform_keys(%{utm_content: :name})
 
     if params["csv"] do
@@ -632,15 +582,12 @@ defmodule PlausibleWeb.Api.StatsController do
 
   def utm_terms(conn, params) do
     site = conn.assigns[:site]
-
     query = Query.from(site, params)
-
     pagination = parse_pagination(params)
-    metrics = [:visitors, :bounce_rate, :visit_duration]
+    metrics = breakdown_metrics(query, [:bounce_rate, :visit_duration])
 
     res =
       Stats.breakdown(site, query, "visit:utm_term", metrics, pagination)
-      |> add_cr(site, query, pagination, :utm_term, "visit:utm_term")
       |> transform_keys(%{utm_term: :name})
 
     if params["csv"] do
@@ -658,16 +605,12 @@ defmodule PlausibleWeb.Api.StatsController do
 
   def utm_sources(conn, params) do
     site = conn.assigns[:site]
-
     query = Query.from(site, params)
-
     pagination = parse_pagination(params)
-
+    metrics = breakdown_metrics(query, [:bounce_rate, :visit_duration])
-    metrics = [:visitors, :bounce_rate, :visit_duration]
 
     res =
       Stats.breakdown(site, query, "visit:utm_source", metrics, pagination)
-      |> add_cr(site, query, pagination, :utm_source, "visit:utm_source")
       |> transform_keys(%{utm_source: :name})
 
     if params["csv"] do
@@ -685,16 +628,12 @@ defmodule PlausibleWeb.Api.StatsController do
 
   def referrers(conn, params) do
     site = conn.assigns[:site]
-
     query = Query.from(site, params)
-
     pagination = parse_pagination(params)
-
+    metrics = breakdown_metrics(query, [:bounce_rate, :visit_duration])
-    metrics = [:visitors, :bounce_rate, :visit_duration]
 
     res =
       Stats.breakdown(site, query, "visit:referrer", metrics, pagination)
-      |> add_cr(site, query, pagination, :referrer, "visit:referrer")
       |> transform_keys(%{referrer: :name})
 
     if params["csv"] do
@@ -754,12 +693,13 @@ defmodule PlausibleWeb.Api.StatsController do
 
     pagination = parse_pagination(params)
 
-    metrics =
+    extra_metrics =
-      if params["detailed"], do: [:visitors, :bounce_rate, :visit_duration], else: [:visitors]
+      if params["detailed"], do: [:bounce_rate, :visit_duration], else: []
+
+    metrics = breakdown_metrics(query, extra_metrics)
 
     referrers =
       Stats.breakdown(site, query, "visit:referrer", metrics, pagination)
-      |> add_cr(site, query, pagination, :referrer, "visit:referrer")
       |> transform_keys(%{referrer: :name})
 
     json(conn, referrers)
@@ -769,16 +709,16 @@ defmodule PlausibleWeb.Api.StatsController do
     site = conn.assigns[:site]
     query = Query.from(site, params)
 
-    metrics =
+    extra_metrics =
       if params["detailed"],
-        do: [:visitors, :pageviews, :bounce_rate, :time_on_page],
+        do: [:pageviews, :bounce_rate, :time_on_page],
-        else: [:visitors]
+        else: []
 
+    metrics = breakdown_metrics(query, extra_metrics)
     pagination = parse_pagination(params)
 
     pages =
       Stats.breakdown(site, query, "event:page", metrics, pagination)
-      |> add_cr(site, query, pagination, :page, "event:page")
       |> transform_keys(%{page: :name})
 
     if params["csv"] do
@@ -798,11 +738,10 @@ defmodule PlausibleWeb.Api.StatsController do
     site = conn.assigns[:site]
     query = Query.from(site, params)
     pagination = parse_pagination(params)
-    metrics = [:visitors, :visits, :visit_duration]
+    metrics = breakdown_metrics(query, [:visits, :visit_duration])
 
     entry_pages =
       Stats.breakdown(site, query, "visit:entry_page", metrics, pagination)
-      |> add_cr(site, query, pagination, :entry_page, "visit:entry_page")
       |> transform_keys(%{entry_page: :name})
 
     if params["csv"] do
@@ -829,11 +768,10 @@ defmodule PlausibleWeb.Api.StatsController do
     site = conn.assigns[:site]
     query = Query.from(site, params)
     {limit, page} = parse_pagination(params)
-    metrics = [:visitors, :visits]
+    metrics = breakdown_metrics(query, [:visits])
 
     exit_pages =
       Stats.breakdown(site, query, "visit:exit_page", metrics, {limit, page})
-      |> add_cr(site, query, {limit, page}, :exit_page, "visit:exit_page")
       |> add_exit_rate(site, query, limit)
       |> transform_keys(%{exit_page: :name})
 
@@ -889,10 +827,10 @@ defmodule PlausibleWeb.Api.StatsController do
     site = conn.assigns[:site]
     query = site |> Query.from(params)
     pagination = parse_pagination(params)
+    metrics = breakdown_metrics(query)
 
     countries =
-      Stats.breakdown(site, query, "visit:country", [:visitors], pagination)
+      Stats.breakdown(site, query, "visit:country", metrics, pagination)
-      |> add_cr(site, query, {300, 1}, :country, "visit:country")
       |> transform_keys(%{country: :code})
       |> add_percentages(site, query)
 
@@ -941,9 +879,10 @@ defmodule PlausibleWeb.Api.StatsController do
     site = conn.assigns[:site]
     query = site |> Query.from(params)
     pagination = parse_pagination(params)
+    metrics = breakdown_metrics(query)
 
     regions =
-      Stats.breakdown(site, query, "visit:region", [:visitors], pagination)
+      Stats.breakdown(site, query, "visit:region", metrics, pagination)
       |> transform_keys(%{region: :code})
       |> Enum.map(fn region ->
         region_entry = Location.get_subdivision(region[:code])
@@ -974,9 +913,10 @@ defmodule PlausibleWeb.Api.StatsController do
     site = conn.assigns[:site]
     query = site |> Query.from(params)
     pagination = parse_pagination(params)
+    metrics = breakdown_metrics(query)
 
     cities =
-      Stats.breakdown(site, query, "visit:city", [:visitors], pagination)
+      Stats.breakdown(site, query, "visit:city", metrics, pagination)
       |> transform_keys(%{city: :code})
       |> Enum.map(fn city ->
         city_info = Location.get_city(city[:code])
@@ -1012,10 +952,10 @@ defmodule PlausibleWeb.Api.StatsController do
     site = conn.assigns[:site]
     query = Query.from(site, params)
     pagination = parse_pagination(params)
+    metrics = breakdown_metrics(query)
 
     browsers =
-      Stats.breakdown(site, query, "visit:browser", [:visitors], pagination)
+      Stats.breakdown(site, query, "visit:browser", metrics, pagination)
-      |> add_cr(site, query, pagination, :browser, "visit:browser")
       |> transform_keys(%{browser: :name})
       |> add_percentages(site, query)
 
@@ -1036,10 +976,10 @@ defmodule PlausibleWeb.Api.StatsController do
     site = conn.assigns[:site]
     query = Query.from(site, params)
     pagination = parse_pagination(params)
+    metrics = breakdown_metrics(query)
 
     versions =
-      Stats.breakdown(site, query, "visit:browser_version", [:visitors], pagination)
+      Stats.breakdown(site, query, "visit:browser_version", metrics, pagination)
-      |> add_cr(site, query, pagination, :browser_version, "visit:browser_version")
       |> transform_keys(%{browser_version: :name})
       |> add_percentages(site, query)
 
@@ -1066,10 +1006,10 @@ defmodule PlausibleWeb.Api.StatsController do
     site = conn.assigns[:site]
     query = Query.from(site, params)
     pagination = parse_pagination(params)
+    metrics = breakdown_metrics(query)
 
     systems =
-      Stats.breakdown(site, query, "visit:os", [:visitors], pagination)
+      Stats.breakdown(site, query, "visit:os", metrics, pagination)
-      |> add_cr(site, query, pagination, :os, "visit:os")
       |> transform_keys(%{os: :name})
       |> add_percentages(site, query)
 
@@ -1090,10 +1030,10 @@ defmodule PlausibleWeb.Api.StatsController do
     site = conn.assigns[:site]
     query = Query.from(site, params)
     pagination = parse_pagination(params)
+    metrics = breakdown_metrics(query)
 
     versions =
-      Stats.breakdown(site, query, "visit:os_version", [:visitors], pagination)
+      Stats.breakdown(site, query, "visit:os_version", metrics, pagination)
-      |> add_cr(site, query, pagination, :os_version, "visit:os_version")
       |> transform_keys(%{os_version: :name})
       |> add_percentages(site, query)
 
@@ -1104,10 +1044,10 @@ defmodule PlausibleWeb.Api.StatsController do
     site = conn.assigns[:site]
     query = Query.from(site, params)
     pagination = parse_pagination(params)
+    metrics = breakdown_metrics(query)
 
     sizes =
-      Stats.breakdown(site, query, "visit:device", [:visitors], pagination)
+      Stats.breakdown(site, query, "visit:device", metrics, pagination)
-      |> add_cr(site, query, pagination, :device, "visit:device")
       |> transform_keys(%{device: :name})
       |> add_percentages(site, query)
 
@@ -1124,14 +1064,6 @@ defmodule PlausibleWeb.Api.StatsController do
     end
   end
 
-  defp calculate_cr(nil, _converted_visitors), do: nil
-
-  defp calculate_cr(unique_visitors, converted_visitors) do
-    if unique_visitors > 0,
-      do: Float.round(converted_visitors / unique_visitors * 100, 1),
-      else: 0.0
-  end
-
   def conversions(conn, params) do
     pagination = parse_pagination(params)
     site = Plausible.Repo.preload(conn.assigns.site, :goals)
@@ -1144,21 +1076,7 @@ defmodule PlausibleWeb.Api.StatsController do
         query
       end
 
-    total_q = Query.remove_event_filters(query, [:goal, :props])
+    metrics = [:visitors, :events, :conversion_rate] ++ @revenue_metrics
-
-    %{visitors: %{value: total_visitors}} = Stats.aggregate(site, total_q, [:visitors])
-
-    metrics =
-      on_full_build do
-        if Enum.any?(site.goals, &Plausible.Goal.Revenue.revenue?/1) and
-             Plausible.Billing.Feature.RevenueGoals.enabled?(site) do
-          [:visitors, :events] ++ @revenue_metrics
-        else
-          [:visitors, :events]
-        end
-      else
-        [:visitors, :events]
-      end
 
     conversions =
       site
@@ -1166,7 +1084,6 @@ defmodule PlausibleWeb.Api.StatsController do
       |> transform_keys(%{goal: :name})
       |> Enum.map(fn goal ->
         goal
-        |> Map.put(:conversion_rate, calculate_cr(total_visitors, goal[:visitors]))
         |> Enum.map(&format_revenue_metric/1)
         |> Map.new()
       end)
@@ -1240,32 +1157,19 @@ defmodule PlausibleWeb.Api.StatsController do
       |> Map.put(:include_imported, false)
 
     metrics =
-      if full_build?() and Map.has_key?(query.filters, "event:goal") do
+      if query.filters["event:goal"] do
-        [:visitors, :events] ++ @revenue_metrics
+        [:visitors, :events, :conversion_rate] ++ @revenue_metrics
       else
-        [:visitors, :events]
+        [:visitors, :events] ++ @revenue_metrics
       end
 
-    props =
+    Stats.breakdown(site, query, prefixed_prop, metrics, pagination)
-      Stats.breakdown(site, query, prefixed_prop, metrics, pagination)
+    |> transform_keys(%{prop_key => :name})
-      |> transform_keys(%{prop_key => :name})
+    |> Enum.map(fn entry ->
-      |> Enum.map(fn entry ->
+      Enum.map(entry, &format_revenue_metric/1)
-        Enum.map(entry, &format_revenue_metric/1)
+      |> Map.new()
-        |> Map.new()
+    end)
-      end)
+    |> add_percentages(site, query)
-      |> add_percentages(site, query)
-
-    if Map.has_key?(query.filters, "event:goal") do
-      total_q = Query.remove_event_filters(query, [:goal, :props])
-
-      %{visitors: %{value: total_unique_visitors}} = Stats.aggregate(site, total_q, [:visitors])
-
-      Enum.map(props, fn prop ->
-        Map.put(prop, :conversion_rate, calculate_cr(total_unique_visitors, prop.visitors))
-      end)
-    else
-      props
-    end
   end
 
   def current_visitors(conn, _) do
@@ -1324,37 +1228,6 @@ defmodule PlausibleWeb.Api.StatsController do
 
   defp add_percentages(breakdown_result, _, _), do: breakdown_result
 
-  defp add_cr([_ | _] = breakdown_results, site, query, pagination, key_name, filter_name)
-       when is_map_key(query.filters, "event:goal") do
-    items = Enum.map(breakdown_results, fn item -> Map.fetch!(item, key_name) end)
-
-    query_without_goal =
-      query
-      |> Query.put_filter(filter_name, {:member, items})
-      |> Query.remove_event_filters([:goal, :props])
-
-    # Here, we're always only interested in the first page of results
-    # - the :member filter makes sure that the results always match with
-    # the items in the given breakdown_results list
-    pagination = {elem(pagination, 0), 1}
-
-    res_without_goal =
-      Stats.breakdown(site, query_without_goal, filter_name, [:visitors], pagination)
-
-    Enum.map(breakdown_results, fn item ->
-      without_goal =
-        Enum.find(res_without_goal, fn s ->
-          Map.fetch!(s, key_name) == Map.fetch!(item, key_name)
-        end)
-
-      item
-      |> Map.put(:total_visitors, without_goal.visitors)
-      |> Map.put(:conversion_rate, calculate_cr(without_goal.visitors, item.visitors))
-    end)
-  end
-
-  defp add_cr(breakdown_results, _, _, _, _, _), do: breakdown_results
-
   defp to_csv(list, columns), do: to_csv(list, columns, columns)
 
   defp to_csv(list, columns, column_names) do
@@ -1485,4 +1358,12 @@ defmodule PlausibleWeb.Api.StatsController do
       {metric, value}
     end
   end
+
+  defp breakdown_metrics(query, extra_metrics \\ []) do
+    if query.filters["event:goal"] do
+      [:visitors, :conversion_rate, :total_visitors]
+    else
+      [:visitors] ++ extra_metrics
+    end
+  end
 end

test/plausible_web/controllers/api/external_stats_controller/aggregate_test.exs

@@ -140,6 +140,22 @@ defmodule PlausibleWeb.Api.ExternalStatsController.AggregateTest do
       end
     end
 
+    test "validates that conversion_rate cannot be queried without a goal filter", %{
+      conn: conn,
+      site: site
+    } do
+      conn =
+        get(conn, "/api/v1/stats/aggregate", %{
+          "site_id" => site.domain,
+          "metrics" => "conversion_rate"
+        })
+
+      assert %{"error" => msg} = json_response(conn, 400)
+
+      assert msg =~
+               "Metric `conversion_rate` can only be queried in a goal breakdown or with a goal filter"
+    end
+
     test "validates that views_per_visit cannot be used with event:page filter", %{
       conn: conn,
       site: site
@@ -156,6 +172,23 @@ defmodule PlausibleWeb.Api.ExternalStatsController.AggregateTest do
                  "Metric `views_per_visit` cannot be queried with a filter on `event:page`."
              }
     end
+
+    test "validates that views_per_visit cannot be used with an event only filter", %{
+      conn: conn,
+      site: site
+    } do
+      conn =
+        get(conn, "/api/v1/stats/aggregate", %{
+          "site_id" => site.domain,
+          "filters" => "event:name==Something",
+          "metrics" => "views_per_visit"
+        })
+
+      assert json_response(conn, 400) == %{
+               "error" =>
+                 "Session metric `views_per_visit` cannot be queried when using a filter on `event:name`."
+             }
+    end
   end
 
   test "aggregates a single metric", %{conn: conn, site: site} do
@@ -293,6 +326,35 @@ defmodule PlausibleWeb.Api.ExternalStatsController.AggregateTest do
                "visit_duration" => %{"value" => 0, "change" => 0}
              }
     end
+
+    test "can compare conversion_rate with previous period", %{conn: conn, site: site} do
+      today = ~N[2023-05-05 12:00:00]
+      yesterday = Timex.shift(today, days: -1)
+
+      populate_stats(site, [
+        build(:event, name: "Signup", timestamp: yesterday),
+        build(:pageview, timestamp: yesterday),
+        build(:pageview, timestamp: yesterday),
+        build(:event, name: "Signup", timestamp: today),
+        build(:pageview, timestamp: today)
+      ])
+
+      insert(:goal, %{site: site, event_name: "Signup"})
+
+      conn =
+        get(conn, "/api/v1/stats/aggregate", %{
+          "site_id" => site.domain,
+          "period" => "day",
+          "date" => "2023-05-05",
+          "metrics" => "conversion_rate",
+          "filters" => "event:goal==Signup",
+          "compare" => "previous_period"
+        })
+
+      assert json_response(conn, 200)["results"] == %{
+               "conversion_rate" => %{"value" => 50.0, "change" => 50.0}
+             }
+    end
   end
 
   describe "with imported data" do
@@ -1234,4 +1296,111 @@ defmodule PlausibleWeb.Api.ExternalStatsController.AggregateTest do
       assert json_response(conn, 200)["results"] == %{"pageviews" => %{"value" => 3}}
     end
   end
+
+  describe "metrics" do
+    test "conversion_rate when goal filter is applied", %{conn: conn, site: site} do
+      populate_stats(site, [
+        build(:event, name: "Signup"),
+        build(:pageview)
+      ])
+
+      insert(:goal, %{site: site, event_name: "Signup"})
+
+      conn =
+        get(conn, "/api/v1/stats/aggregate", %{
+          "site_id" => site.domain,
+          "metrics" => "conversion_rate",
+          "filters" => "event:goal==Signup"
+        })
+
+      assert json_response(conn, 200)["results"] == %{"conversion_rate" => %{"value" => 50}}
+    end
+
+    test "conversion_rate when goal + custom prop filter applied", %{
+      conn: conn,
+      site: site
+    } do
+      populate_stats(site, [
+        build(:event, name: "Signup"),
+        build(:event, name: "Signup", "meta.key": ["author"], "meta.value": ["Uku"]),
+        build(:event, name: "Signup", "meta.key": ["author"], "meta.value": ["Marko"]),
+        build(:pageview)
+      ])
+
+      insert(:goal, %{site: site, event_name: "Signup"})
+
+      conn =
+        get(conn, "/api/v1/stats/aggregate", %{
+          "site_id" => site.domain,
+          "metrics" => "conversion_rate,visitors,events",
+          "filters" => "event:goal==Signup;event:props:author==Uku"
+        })
+
+      assert %{
+               "conversion_rate" => %{"value" => 25.0},
+               "visitors" => %{"value" => 1},
+               "events" => %{"value" => 1}
+             } = json_response(conn, 200)["results"]
+    end
+
+    test "conversion_rate when goal + visit property filter applied", %{
+      conn: conn,
+      site: site
+    } do
+      populate_stats(site, [
+        build(:event, name: "Signup"),
+        build(:event, name: "Signup", browser: "Chrome"),
+        build(:event, name: "Signup", browser: "Firefox", user_id: 123),
+        build(:event, name: "Signup", browser: "Firefox", user_id: 123),
+        build(:pageview, browser: "Firefox"),
+        build(:pageview)
+      ])
+
+      insert(:goal, %{site: site, event_name: "Signup"})
+
+      conn =
+        get(conn, "/api/v1/stats/aggregate", %{
+          "site_id" => site.domain,
+          "metrics" => "conversion_rate,visitors,events",
+          "filters" => "visit:browser==Firefox;event:goal==Signup"
+        })
+
+      assert %{
+               "conversion_rate" => %{"value" => 50.0},
+               "visitors" => %{"value" => 1},
+               "events" => %{"value" => 2}
+             } =
+               json_response(conn, 200)["results"]
+    end
+
+    test "conversion_rate when goal + page filter applied", %{
+      conn: conn,
+      site: site
+    } do
+      populate_stats(site, [
+        build(:event, name: "Signup"),
+        build(:event, name: "Signup", pathname: "/not-this"),
+        build(:event, name: "Signup", pathname: "/this", user_id: 123),
+        build(:event, name: "Signup", pathname: "/this", user_id: 123),
+        build(:pageview, pathname: "/this"),
+        build(:pageview)
+      ])
+
+      insert(:goal, %{site: site, event_name: "Signup"})
+
+      conn =
+        get(conn, "/api/v1/stats/aggregate", %{
+          "site_id" => site.domain,
+          "metrics" => "conversion_rate,visitors,events",
+          "filters" => "event:page==/this;event:goal==Signup"
+        })
+
+      assert %{
+               "conversion_rate" => %{"value" => 50.0},
+               "visitors" => %{"value" => 1},
+               "events" => %{"value" => 2}
+             } =
+               json_response(conn, 200)["results"]
+    end
+  end
 end

test/plausible_web/controllers/api/external_stats_controller/breakdown_test.exs

@@ -83,6 +83,23 @@ defmodule PlausibleWeb.Api.ExternalStatsController.BreakdownTest do
   end
 
   describe "param validation" do
+    test "does not allow querying conversion_rate without a goal filter", %{
+      conn: conn,
+      site: site
+    } do
+      conn =
+        get(conn, "/api/v1/stats/breakdown", %{
+          "site_id" => site.domain,
+          "property" => "event:page",
+          "metrics" => "conversion_rate"
+        })
+
+      assert json_response(conn, 400) == %{
+               "error" =>
+                 "Metric `conversion_rate` can only be queried in a goal breakdown or with a goal filter"
+             }
+    end
+
     test "validates that property is required", %{conn: conn, site: site} do
       conn =
         get(conn, "/api/v1/stats/breakdown", %{
@@ -2001,6 +2018,308 @@ defmodule PlausibleWeb.Api.ExternalStatsController.BreakdownTest do
   end
 
   describe "metrics" do
+    test "returns conversion_rate in an event:goal breakdown", %{conn: conn, site: site} do
+      populate_stats(site, [
+        build(:event, name: "Signup", user_id: 1),
+        build(:event, name: "Signup", user_id: 1),
+        build(:pageview, pathname: "/blog"),
+        build(:pageview, pathname: "/blog/post"),
+        build(:pageview)
+      ])
+
+      insert(:goal, %{site: site, event_name: "Signup"})
+      insert(:goal, %{site: site, page_path: "/blog**"})
+
+      conn =
+        get(conn, "/api/v1/stats/breakdown", %{
+          "site_id" => site.domain,
+          "period" => "day",
+          "property" => "event:goal",
+          "metrics" => "visitors,events,conversion_rate"
+        })
+
+      assert json_response(conn, 200) == %{
+               "results" => [
+                 %{
+                   "goal" => "Visit /blog**",
+                   "visitors" => 2,
+                   "events" => 2,
+                   "conversion_rate" => 50
+                 },
+                 %{
+                   "goal" => "Signup",
+                   "visitors" => 1,
+                   "events" => 2,
+                   "conversion_rate" => 25
+                 }
+               ]
+             }
+    end
+
+    test "returns conversion_rate alone in an event:goal breakdown", %{conn: conn, site: site} do
+      populate_stats(site, [
+        build(:event, name: "Signup", user_id: 1),
+        build(:pageview)
+      ])
+
+      insert(:goal, %{site: site, event_name: "Signup"})
+
+      conn =
+        get(conn, "/api/v1/stats/breakdown", %{
+          "site_id" => site.domain,
+          "period" => "day",
+          "property" => "event:goal",
+          "metrics" => "conversion_rate"
+        })
+
+      assert json_response(conn, 200) == %{
+               "results" => [
+                 %{
+                   "goal" => "Signup",
+                   "conversion_rate" => 50
+                 }
+               ]
+             }
+    end
+
+    test "returns conversion_rate in a goal filtered custom prop breakdown", %{
+      conn: conn,
+      site: site
+    } do
+      populate_stats(site, [
+        build(:pageview, pathname: "/blog/1", "meta.key": ["author"], "meta.value": ["Uku"]),
+        build(:pageview, pathname: "/blog/2", "meta.key": ["author"], "meta.value": ["Uku"]),
+        build(:pageview, pathname: "/blog/3", "meta.key": ["author"], "meta.value": ["Uku"]),
+        build(:pageview, pathname: "/blog/1", "meta.key": ["author"], "meta.value": ["Marko"]),
+        build(:pageview,
+          pathname: "/blog/2",
+          "meta.key": ["author"],
+          "meta.value": ["Marko"],
+          user_id: 1
+        ),
+        build(:pageview,
+          pathname: "/blog/3",
+          "meta.key": ["author"],
+          "meta.value": ["Marko"],
+          user_id: 1
+        ),
+        build(:pageview, pathname: "/blog"),
+        build(:pageview, "meta.key": ["author"], "meta.value": ["Marko"]),
+        build(:pageview)
+      ])
+
+      insert(:goal, %{site: site, page_path: "/blog**"})
+
+      conn =
+        get(conn, "/api/v1/stats/breakdown", %{
+          "site_id" => site.domain,
+          "period" => "day",
+          "property" => "event:props:author",
+          "filters" => "event:goal==Visit /blog**",
+          "metrics" => "visitors,events,conversion_rate"
+        })
+
+      assert json_response(conn, 200) == %{
+               "results" => [
+                 %{
+                   "author" => "Uku",
+                   "visitors" => 3,
+                   "events" => 3,
+                   "conversion_rate" => 37.5
+                 },
+                 %{
+                   "author" => "Marko",
+                   "visitors" => 2,
+                   "events" => 3,
+                   "conversion_rate" => 25
+                 },
+                 %{
+                   "author" => "(none)",
+                   "visitors" => 1,
+                   "events" => 1,
+                   "conversion_rate" => 12.5
+                 }
+               ]
+             }
+    end
+
+    test "returns conversion_rate alone in a goal filtered custom prop breakdown", %{
+      conn: conn,
+      site: site
+    } do
+      populate_stats(site, [
+        build(:pageview, pathname: "/blog/1", "meta.key": ["author"], "meta.value": ["Uku"]),
+        build(:pageview)
+      ])
+
+      insert(:goal, %{site: site, page_path: "/blog**"})
+
+      conn =
+        get(conn, "/api/v1/stats/breakdown", %{
+          "site_id" => site.domain,
+          "period" => "day",
+          "property" => "event:props:author",
+          "filters" => "event:goal==Visit /blog**",
+          "metrics" => "conversion_rate"
+        })
+
+      assert json_response(conn, 200) == %{
+               "results" => [
+                 %{
+                   "author" => "Uku",
+                   "conversion_rate" => 50
+                 }
+               ]
+             }
+    end
+
+    test "returns conversion_rate in a goal filtered event:page breakdown", %{
+      conn: conn,
+      site: site
+    } do
+      populate_stats(site, [
+        build(:event, pathname: "/en/register"),
+        build(:event, pathname: "/en/register", name: "Signup"),
+        build(:event, pathname: "/en/register", name: "Signup"),
+        build(:event, pathname: "/it/register", name: "Signup", user_id: 1),
+        build(:event, pathname: "/it/register", name: "Signup", user_id: 1),
+        build(:event, pathname: "/it/register")
+      ])
+
+      insert(:goal, %{site: site, event_name: "Signup"})
+
+      conn =
+        get(conn, "/api/v1/stats/breakdown", %{
+          "site_id" => site.domain,
+          "period" => "day",
+          "property" => "event:page",
+          "filters" => "event:goal==Signup",
+          "metrics" => "visitors,events,conversion_rate"
+        })
+
+      assert json_response(conn, 200) == %{
+               "results" => [
+                 %{
+                   "page" => "/en/register",
+                   "visitors" => 2,
+                   "events" => 2,
+                   "conversion_rate" => 66.7
+                 },
+                 %{
+                   "page" => "/it/register",
+                   "visitors" => 1,
+                   "events" => 2,
+                   "conversion_rate" => 50
+                 }
+               ]
+             }
+    end
+
+    test "returns conversion_rate alone in a goal filtered event:page breakdown", %{
+      conn: conn,
+      site: site
+    } do
+      populate_stats(site, [
+        build(:event, pathname: "/en/register"),
+        build(:event, pathname: "/en/register", name: "Signup")
+      ])
+
+      insert(:goal, %{site: site, event_name: "Signup"})
+
+      conn =
+        get(conn, "/api/v1/stats/breakdown", %{
+          "site_id" => site.domain,
+          "period" => "day",
+          "property" => "event:page",
+          "filters" => "event:goal==Signup",
+          "metrics" => "conversion_rate"
+        })
+
+      assert json_response(conn, 200) == %{
+               "results" => [
+                 %{
+                   "page" => "/en/register",
+                   "conversion_rate" => 50
+                 }
+               ]
+             }
+    end
+
+    test "returns conversion_rate in a multi-goal filtered visit:screen_size breakdown", %{
+      conn: conn,
+      site: site
+    } do
+      populate_stats(site, [
+        build(:event, screen_size: "Mobile"),
+        build(:event, screen_size: "Mobile", name: "AddToCart"),
+        build(:event, screen_size: "Mobile", name: "AddToCart"),
+        build(:event, screen_size: "Desktop", name: "AddToCart", user_id: 1),
+        build(:event, screen_size: "Desktop", name: "Purchase", user_id: 1),
+        build(:event, screen_size: "Desktop")
+      ])
+
+      # Make sure that revenue goals are treated the same
+      # way as regular custom event goals
+      insert(:goal, %{site: site, event_name: "Purchase", currency: :EUR})
+      insert(:goal, %{site: site, event_name: "AddToCart"})
+
+      conn =
+        get(conn, "/api/v1/stats/breakdown", %{
+          "site_id" => site.domain,
+          "period" => "day",
+          "property" => "visit:device",
+          "filters" => "event:goal==AddToCart|Purchase",
+          "metrics" => "visitors,events,conversion_rate"
+        })
+
+      assert json_response(conn, 200) == %{
+               "results" => [
+                 %{
+                   "device" => "Mobile",
+                   "visitors" => 2,
+                   "events" => 2,
+                   "conversion_rate" => 66.7
+                 },
+                 %{
+                   "device" => "Desktop",
+                   "visitors" => 1,
+                   "events" => 2,
+                   "conversion_rate" => 50
+                 }
+               ]
+             }
+    end
+
+    test "returns conversion_rate alone in a goal filtered visit:screen_size breakdown", %{
+      conn: conn,
+      site: site
+    } do
+      populate_stats(site, [
+        build(:event, screen_size: "Mobile"),
+        build(:event, screen_size: "Mobile", name: "AddToCart")
+      ])
+
+      insert(:goal, %{site: site, event_name: "AddToCart"})
+
+      conn =
+        get(conn, "/api/v1/stats/breakdown", %{
+          "site_id" => site.domain,
+          "period" => "day",
+          "property" => "visit:device",
+          "filters" => "event:goal==AddToCart",
+          "metrics" => "conversion_rate"
+        })
+
+      assert json_response(conn, 200) == %{
+               "results" => [
+                 %{
+                   "device" => "Mobile",
+                   "conversion_rate" => 50
+                 }
+               ]
+             }
+    end
+
     test "all metrics for breakdown by visit prop", %{conn: conn, site: site} do
       populate_stats(site, [
         build(:pageview,

test/plausible_web/controllers/api/external_stats_controller/timeseries_test.exs

@@ -1093,6 +1093,22 @@ defmodule PlausibleWeb.Api.ExternalStatsController.TimeseriesTest do
   end
 
   describe "metrics" do
+    test "validates that conversion_rate cannot be queried without a goal filter", %{
+      conn: conn,
+      site: site
+    } do
+      conn =
+        get(conn, "/api/v1/stats/timeseries", %{
+          "site_id" => site.domain,
+          "metrics" => "conversion_rate"
+        })
+
+      assert %{"error" => msg} = json_response(conn, 400)
+
+      assert msg ==
+               "Metric `conversion_rate` can only be queried in a goal breakdown or with a goal filter"
+    end
+
     test "shows pageviews,visits,views_per_visit for last 7d", %{conn: conn, site: site} do
       populate_stats(site, [
         build(:pageview,

test/plausible_web/controllers/api/stats_controller/pages_test.exs

@@ -1313,17 +1313,13 @@ defmodule PlausibleWeb.Api.StatsController.PagesTest do
                %{
                  "total_visitors" => 2,
                  "visitors" => 1,
-                 "visits" => 1,
                  "name" => "/page1",
-                 "visit_duration" => 0,
                  "conversion_rate" => 50.0
                },
                %{
                  "total_visitors" => 1,
                  "visitors" => 1,
-                 "visits" => 1,
                  "name" => "/page2",
-                 "visit_duration" => 900,
                  "conversion_rate" => 100.0
                }
              ]
@@ -1508,14 +1504,12 @@ defmodule PlausibleWeb.Api.StatsController.PagesTest do
                  "name" => "/exit1",
                  "visitors" => 1,
                  "total_visitors" => 1,
-                 "visits" => 1,
                  "conversion_rate" => 100.0
                },
                %{
                  "name" => "/exit2",
                  "visitors" => 1,
                  "total_visitors" => 1,
-                 "visits" => 1,
                  "conversion_rate" => 100.0
                }
              ]