Commit Graph

367 Commits

Author SHA1 Message Date
Adam Rutkowski
5de43b758d
Run tests in async mode where applicable (#2542)
* Set pg pool size for MIX_ENV=test

* Include slow tests in CI run

* Exclude slow tests by default

* Mark tests slow/async where applicable

* Restructure captcha mocks

* Revert async where env is relied upon

* Add --max-failures=1 to CI run

* Set warnings as errors

* Disable async where various mocks are used

* Revert "Disable async where various mocks are used"

This reverts commit 2446b72a29.

* Disable async for test using vcr
2022-12-26 10:20:29 -03:00
Vini Brasil
0e87b489f6
Return empty list when breaking down by event:page without events (#2530)
* Return empty list when breaking down by event:page without events

This commit fixes a bug with pagination where breaking down by event:page
would always return results despite pagination.

Closes #2255

* Update CHANGELOG.md
2022-12-19 13:49:03 +01:00
Uku Taht
a37b3433d7
Always show direct traffic in sources reports (#2531)
* Remove show_noref behaviour

Removes query param show_noref which was used from React to control
whether to show Direct / None traffic or not. The show_noref behaviour
was untested previously.

Closes #2523

* Add changelog entry

* Fix tests

* Removed files I did not mean to check in :)
2022-12-19 13:40:21 +01:00
Vini Brasil
478e0c6990
Handle GA authentication errors (#2505) 2022-12-07 23:32:14 -03:00
Vini Brasil
9b32dace3e
Add Content-Security-Policy to favicon Plug (#2489) 2022-12-02 06:33:24 -03:00
Adam Rutkowski
467317f24e
Fix: application startup, event timestamps (#2479)
### Changes

This PR:
- pushes PromEx to the bottom of supervision stack to avoid Endpoint
instrumentation failure
- ensures the site cache is ready by exposing it through the health
check endpoint
- fixes event timestamps being calculated at compile time, with
regression unit and integration tests

### Tests
- [x] Automated tests have been added
- [ ] This PR does not require tests

### Changelog
- [ ] Entry has been added to changelog
- [x] This PR does not make a user-facing change

### Documentation
- [ ] [Docs](https://github.com/plausible/docs) have been updated
- [x] This change does not need a documentation update

### Dark mode
- [ ] The UI has been tested both in dark and light mode
- [x] This PR does not change the UI
2022-11-29 10:46:49 +01:00
Adam Rutkowski
356575ef78
Gatekeep ingestion pipeline (#2472)
* Update Sites.Cache

So it's now capable of refreshing most recent sites.
Refreshing a single site is no longer wanted.

* Introduce Warmer.RecentlyUpdated

This is Sites Cache warmer that runs only for
most recently updated sites every 30s.

* Validate Request creation early

* Rename RateLimiter to GateKeeper and introduce detailed policies

* Update events API tests - a provisioned site is now required

* Update events ingestion tests

* Make limits visible in CRM Sites index

* Hard-deprecate DOMAIN_BLACKLIST

* Remove unnecessary clause

* Fix typo

* Explicitly delegate Warmer.All

* GateKeeper.allwoance => GateKeeper.check

* Instrument Sites.Cache measurments

* Update send_pageview task to output response headers

* Instrument ingestion pipeline

* Credo

* Make event telemetry test a sync case

* Simplify Request.uri/hostname handling

* Use embedded schema, apply action and rely on get_field
2022-11-28 15:50:55 +01:00
Vini Brasil
ffbfb14714
Pass interval param to CSV export (#2469)
This commit adds the interval param to the CSV export client-side
request.
2022-11-25 10:53:22 +02:00
ruslandoga
77cca04417
fix ownership transfer when is_selfhost=true (#2455)
* fix ownership transfer when is_selfhost=true

* add changelog entry
2022-11-24 16:18:29 +02:00
Vignesh Joglekar
497a52c10a
Add support for intervals (#1574)
Co-authored-by: Vini Brasil <vini@hey.com>
Co-authored-by: Vignesh Joglekar <hey@vigneshjoglekar.com>
2022-11-22 09:50:58 -03:00
Vini Brasil
9c98a3f2e8
Add API support for intervals (#2417) 2022-11-14 18:41:51 -03:00
RobertJoonas
47bf003c29
Merge pull request #2431 from plausible/fix-email-links
Fix email links
2022-11-10 16:45:43 +02:00
Robert
27e8d8db3c remove redundant test 2022-11-10 15:55:33 +02:00
ruslandoga
0b7870dc4d
improve first launch experience for self-hosters (#2357)
* first launch

* dynamic children, wait for repo

* remove wait_for_repo and app env manipulations

* don't mention free trial in self-hosted pages

* add changelog

* assigns[:is_selfhost] -> @is_selfhost

* better changelog wording

* rm admin_user, admin_email, admin_pwd from app env

* rm DISABLE_AUTH

* redirect / to /login when not authenticated

* remove TODO

* Update lib/plausible_web/controllers/page_controller.ex

Co-authored-by: Uku Taht <Uku.taht@gmail.com>

* format

Co-authored-by: Uku Taht <Uku.taht@gmail.com>
2022-11-10 12:42:22 +01:00
Robert
d6125ffeb1 add tests for base_email template 2022-11-09 19:56:46 +02:00
RobertJoonas
7ccbc5d432
fix typos (#2393) 2022-10-27 08:57:34 +02:00
Adam Rutkowski
a44b8ff7f5
Improve test coverage (#2392)
Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
2022-10-27 08:39:34 +02:00
Marko Saric
252cecbf25
Correct the spelling of JavaScript (#2380)
* Update README.md

* Update snippet.html.eex

* Update settings_general.html.eex

* Update site_controller_test.exs
2022-10-26 08:34:33 +02:00
Adam Rutkowski
8e75f2fc07
Separate GA/SC scopes (#2372)
* Use separate scopes for GA/SC integrations

* Update tests with google scope expectations
2022-10-25 08:17:17 -03:00
Adam Rutkowski
d29597ed37
Fix random typos (#2375) 2022-10-25 08:17:09 -03:00
Adam Rutkowski
0fa6b688af
Google APIs integration improvements (#2358)
* Make TestUtils module available in all tests

* Add macros patching the application env in tests

Unfortunately a lot of existing functionality relies on
certain application env setup. This isn't ideal because
the app config is a shared state that prevents us from
running the tests in parallel.

Those macros encapsulate setting up new env for test purposes
and make sure the changes are reverted when the test finishes.

* Allow passing request opts to HTTPClient.post/4

We need this to swap custom request building in
Google Analytics import.

* Unify errors when listing sites

* React: propagate backend error messages if available

* React: catch API errors in Search Terms component

* Propagate google API errors on referrer drilldown

* Handle verified properties errors in SC settings

* Add missing tests for SC settings controller

* Unify errors for fetching search analytics queries (list stats)

* Unify errors refreshing Google Auth Token

* Test fetch_stats/3 errors and replace Double with Mox

* Fixup makrup

* s/class/className

* Simplify Search Terms display in case of errors

* Fix warnings
2022-10-24 09:34:02 +02:00
Adam Rutkowski
e3ca3b32db
Include tests for Captcha success/failure scenarios (#2344)
* Include tests for Captcha success/failure scenarios

* DRY
2022-10-17 08:16:59 -03:00
Adam Rutkowski
28cec9d939
Automatically decode JSON payloads from HTTP (#2329) 2022-10-13 09:39:15 -03:00
Adam Rutkowski
ae4ae5d0a0
Handle duplicate invitations gracefully (#2323) 2022-10-11 11:40:20 -03:00
Adam Rutkowski
ec90a264b4
Improve user input validation (#2291)
* Make pagination params parsing ignore bad input

* Remove unused binding

* Don't crash on filter parse error

* Sanitize input date on internal stats API

* Revert Query module changes (ref 55645734)

* Implement simplistic input date validation in stats controller

* Mute bad request logging
2022-10-11 14:42:14 +02:00
RobertJoonas
c0da024b23
Remove static tracker files (#2116)
* remove tracker files from git index

* generate tracker files on npm test

* generate tracker files for elixir tests/dev/CI

* update tracker/package-lock.json

* exclude npm run deploy from mix test + some docs
2022-10-11 12:19:28 +02:00
Adam Rutkowski
523b114682
Allow super-admin Stats API access for locked sites (#2308) 2022-10-05 12:04:37 +02:00
Vinicius Brasil
beabef7269
Redirect user after upgrading only when subscription is created (#2285)
* Create API to check for an active subscription

* Redirect after upgrading only when subscription is created

* Change upgrade page copy

* Add timeout notice to upgrade success page
2022-10-04 15:54:54 +03:00
Adam Rutkowski
bf2b6880c5
Block Stats API for locked sites (#2302)
* Extract Sites.locked? predicate

* Lock Stats API when dashboard is locked

* Tidy tests

* Don't pollute application env from tests

* Add changelog entry

* Revert "Add changelog entry"

This reverts commit 76346074f9.
2022-10-04 15:34:45 +03:00
Uku Taht
0bc762bfdc
Fix svg favicons (#2295)
* Overrides content-type for SVG favicons

* Organize favicon rendering

Make sure the placeholder icon is always requested from
/favicon/sources/placeholder

* Run prettier on site-switcher.js

* Yak Shave: upgrade Heroicons to 2.0

* Use HeroIcons instead of custom svg

* Update lib/plausible_web/plugs/favicon.ex

Co-authored-by: Adam Rutkowski <hq@mtod.org>

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2022-10-04 13:20:51 +03:00
Uku Taht
12a513f3cd
Remove Mimic dependency (#2280) 2022-10-03 11:41:51 -03:00
Uku Taht
65f0ce3061
[Sentry APP-5B] Ensure form errors can be rendered on /settings (#2278)
* Ensure form errors can be rendered on /settings

* Changelog
2022-09-28 08:56:07 -03:00
Uku Taht
d104abb53d
Add fallback for favicon (#2279)
* Add fallback for favicon

* Add Favicon tests

* Changelog

* Move placeholder icon to priv folder
2022-09-28 08:55:46 -03:00
Vinicius Brasil
7489290d11
International Domain Names (IDN) Support (#2034)
* Accept letters from non-Latin alphabets in domain names

* Replace static URLs with Router functions in settings_visibility

* Beautify dashboard URL in visibility tab

* Add IDN support to CHANGELOG
2022-09-28 10:42:15 +03:00
RobertJoonas
155e274150
Stats API: allow escaping | literal character with \| when filtering (#2266)
* add separate module for filter parsing

* add tests for filter parser

* allow escaping pipe character in filter value

* add documentation and doctests

* do not remove escape chars from wildcard values

* changelog update

* change the parse_filters/1 function argument
2022-09-26 16:20:08 +03:00
Vinicius Brasil
1adda42a75
Refresh Google Analytics token before import (#2254)
* Capture refresh and expires from GA callback

* Pass GA refresh token to import worker

* Refresh GA token before import
2022-09-26 12:29:56 +03:00
Uku Taht
e849e03058
Fix favicons (#2257) 2022-09-23 07:22:43 -03:00
RobertJoonas
80e01fedd5
do not show invoices for free subscription (#2249)
* do not show invoices for free subscription

* use refute instead of negative assert

* changed to refute in the other test too
2022-09-22 17:25:24 -03:00
Uku Taht
e16e357dd2
Fix shared link with bad auth (#2225)
* Render 404 when shared link cannot be found

* Add documentation for StatsController and shared link rendering

* Refactor shared_link/2 for more clarity

* Add changelog entry

* Use mermaid graph for sequence diagram

* Use more accurate return value in sequence diagram

* Refactor Ecto query to be more idiomatic

* Remove order dependence in test

* Restore backwards compatibility for older shared links

* Add changelog entry
2022-09-20 15:37:18 +03:00
Uku Taht
669091f2ef
Ignore unknown country in imported data (#2247) 2022-09-20 15:02:14 +03:00
Uku Taht
5d4918b66b
Allow admins to grant admin rights (#2216)
* Make sure admin can grant the admin role

* Use `site` and `current_user_role` from conn.assigns to avoid extra lookups

* Add some missing documentation

* Restrict owners in the `update_role` action

* Move @moduledoc to appropriate location

* Use Ecto.Enum to get role atoms

* Remove unused functionality

* Ensure that owners cannot lock themselves out

* Add question mark to `can_grant_role`

Co-authored-by: Adam Rutkowski <hq@mtod.org>

* Use strict short-circuit operator for booleans

Co-authored-by: Adam Rutkowski <hq@mtod.org>

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2022-09-16 11:03:09 +03:00
Uku Taht
b239f73a6d
Ignore unknown country code (#2223)
* Ignore unknown country code

* Add changelog entry
2022-09-16 11:02:39 +03:00
RobertJoonas
0194d57b38
add upper bound to limit parameter (#2226)
* add upper bound to limit parameter

* add more strict validation for limit parameter + 2 tests

* update changelog
2022-09-16 10:21:44 +03:00
Vinicius Brasil
d31db86b49
List all Google Analytics views during import (#2184)
* List all Google Analytics views during import

This commit fixes a bug where different Google Analytics views with the
same name and URI were not shown. This was caused because GA views were
stored as a map, that naturally doesn't support duplicate keys.

This change updates the GA views list to display view IDs, making it
clearer to know what is being imported. The dropdown is now grouped by
website URL.

* Put Google Analytics API URLs in app env

* Add controller test to GA view list
2022-09-08 21:02:17 +03:00
Adam Rutkowski
2275cd8e22 Respond with 400 when no auth supplied to shared link request 2022-09-05 11:15:09 +02:00
Vinicius Brasil
a10d44a0d7
Refactor event struct creation function (#2098)
* Replace Ingestion.Request headers with user_agent

* Replace generic Ingestion.Request params with specific fields

* Refactor event building function into small functions

* Move Plausible.Ingestion to Plausible.Ingestion.Event

* Add option to override event fields while building

* Rename Ingestion.Request meta to props

* Replace UTM-specific fields with generic query_params

* Remove Map.from_struct/1 call from ingestion pipeline

* Remove stash options from ingestion
2022-08-16 14:43:10 +03:00
Uku Taht
a5757bc61e Allow ownership transfers 2022-08-11 13:16:46 +03:00
ruslandoga
ce461b5192 Auth updates 2022-08-10 10:37:37 +03:00
RobertJoonas
a058cf6240
added a test and changed hashmode condition (#2082) 2022-08-09 14:31:37 +03:00
Vinicius Brasil
b415ebe776
Fix geolocation subdivision pattern matching (#2063)
* Fix geolocation subdivision pattern matching

This commit fixes a bug where regions were not being saved. This was
caused because Geolix response was returning an additional
`:geolocation` map key. It also adds a test case for this.

Closes #2033

* Add geolocation database to .gitignore
2022-07-28 15:59:39 +03:00
Vinicius Brasil
b5ea6ae3dc
Keep user filter when listing cities, countries, and regions stats (#2030)
This commit fixes a bug where location filters were filtering stats but
not the locations list. This was caused by a `Map.put/3` call that
overrides the user filter. This commit rollbacks 5b57143273
changes and removes the `Map.put/3` call.

Closes #1982
2022-07-25 12:19:38 +03:00
Sasha Fonseca
db3a852f26
Fix missing 'pageviews' column from 'pages.csv' (#1988)
* Fix missing 'pageviews' column from 'pages.csv'

* Add CHANGELOG entry for Issue #1878 fix
2022-07-12 23:46:47 +03:00
Uku Taht
292a419473
Add feature flag to block event ingest (#1991)
Event ingest can be blocked using the flag `block_event_ingest`
2022-07-06 12:25:40 +03:00
Uku Taht
3e5695408a
Use new Session.CacheStore in favour of Session.Store (#1934)
* Remove Session.Store in favour of Session.CacheStore

* Add CHANGELOG entry

* Use appropriate enum function
2022-06-06 10:44:33 +03:00
Kurt McAlpine
b92889aaf0
Add API to retrieve site by domain (#1942) 2022-06-03 10:37:14 +03:00
Uku Taht
102ff1885e Trim trailing whitespace from pathname 2022-06-02 11:34:10 +03:00
RobertJoonas
11654ddc07
Script extension additions (#1915)
* added data-include attribute to plausible.exclusions.js

* reorder extensions in filename when serving the plausible script

* fix formatting

* tweaks after review

* changelog update
2022-05-27 10:11:40 +03:00
Uku Taht
948de2b4c9 Mark more tests as async 2022-05-06 10:30:38 +03:00
Uku Taht
cd8df23e5e Do not rely on insertion order in tests 2022-05-06 10:20:35 +03:00
Uku Taht
e4816e8f64 Update Elixir version 2022-05-02 11:56:00 +03:00
Uku Taht
02abbda06d Ignore nil custom prop as well 2022-04-29 11:50:31 +03:00
Uku Taht
c92c548ca8 Do not error on empty string custom prop value 2022-04-29 11:13:36 +03:00
Uku Taht
5e415c2420 Add entry_props back in 2022-04-22 10:58:02 +03:00
Uku Taht
8fb4f3f886 Revert entry props 2022-04-21 19:22:38 +03:00
RobertJoonas
40275b64d4
Pageview custom dimensions (#1816)
* added custom dimension filtering tests for pages

* first filter UI in place

* pages, entry pages and exit pages can be filtered by pageview props

* added tests for expected filtering behaviour

* fix dimension filter for sources + tests

* added is_not filtering functionality

* fixed formatting

* fixed admin_test

* added (none) as filter value + is_not filter type in UI

* added prefilling applied filter values and some UI tweaks

* added fetch options

* Make prop suggestions work with `props` filter

* Fix test

* Track login state internally

* Add CHANGELOG entry

Co-authored-by: Uku Taht <uku.taht@gmail.com>
2022-04-21 11:47:15 +03:00
Uku Taht
66aceda9e9 Fix main graph selection with imported data 2022-04-19 10:50:50 +03:00
Uku Taht
404a5d15a3 Fix tests 2022-04-18 14:51:33 +03:00
Uku Taht
3d231fc094 Redirect to correct change plan page for enterprise customers 2022-04-18 14:18:49 +03:00
Andrea Mazzarella
1128ff4bfa
Update the internal /sites api to paginate results (#1824)
* Update the internal /sites api to paginate results and adapts site-switcher to it

* Update the Changelog

* Format internal controller

* Remove the `+ Add Site` link from the site-switcher in the dashboard

* Change camel to snake case and replace imports with fully qualified calls

* Remove trailing comma from site-switcher
2022-04-18 12:32:01 +03:00
Vignesh Joglekar
3b97ecdc62
Adds Main Graph Metric Selection (#1364)
* First pass bringing in previous graph improvements, and comparsion context

* Swaps issue template to new issue form syntax

* Indentation update

* Indentation update?

* More indentation

* Intendation is hard

* Finalized indentation?

* Github indentation

* Missing fields

* Formatting changes

* Checkbox changes

* Uses new timeseries API, various UI improvements, descopes conversions, ToP from graphing

* Fixes Mobile UI Issues

* Improves point detection and display on hover

* Fixes & adds tests for updated main-graph API route

* Changelog

* Changes to better metric option declaration & minor UI/default fixes

* Fixes top stat tooltips showing unformatted numbers for special (non-rounded) top stats

* Formatting

* Fixes regression with dashed portion not stopping at present_index

* Removes comparison + lint

* Improves top stat active style

* Removes comparison tests

* Splits out tooltip and top stats

Still needs:
- Tests
- Potentially more cleanup

* Adds/moves tests for top stats

* Formatting

* Updates metric LS key, removes console log

* Various fixes + cleanup

* Makes tooltip position & style more consistent

* Fixes test (returns import status on both main graph & top stats)

* Fixes interaction with month dateFormatter

* Fixes edge case tooltip behavior

It was simpler than I thought :/

* Make the entire top stat clickable

* Minor UI improvements

* Fixes another tooltip visibility edge case + cleans up boolean algebra

Co-authored-by: Uku Taht <Uku.taht@gmail.com>
2022-04-13 10:38:47 +03:00
Kian-Meng Ang
70cb3eed54
Fix typos (#1829) 2022-04-11 20:42:40 +03:00
Uku Taht
a359271af2 Fix more tests 2022-04-06 11:34:54 +03:00
Uku Taht
7f58e6be4e Add Year to Date option in datepicker 2022-03-31 13:52:48 +03:00
Uku Taht
8467979051 Add All Time option to datepicker 2022-03-31 11:00:20 +03:00
Uku Taht
8719865a41 Redirect user when they've already upgraded 2022-03-30 10:10:44 +03:00
Uku Taht
ce649966ed Fix display 2022-03-29 13:38:35 +03:00
Uku Taht
ebdd856df7 Fix tests 2022-03-29 13:06:44 +03:00
Andrea Mazzarella
3c93a2d91b
1421 contains filter (#1799)
* small refactors and adds 'contains' to modal

* supports contains filter in the backend

* moves entry and exit page under the page filter

* prettier

* updates the CHANGELOG

* undo package-lock changes

* fixes formatting for elixir

* renames unused parameter to _

* Update changelog

* Use uppercase for constants and update type/prefix lookup
2022-03-29 08:39:16 +03:00
RobertJoonas
e5cf800d4a
Improve stats api (#1759)
* can now query events metric when breaking down on a visit prop

* updated changelog

* fixed formatting

* fixed a bug - querying for bounce rate does not return visits anymore

* removed the changelog update
2022-03-24 16:16:51 +02:00
Uku Taht
51cd24bcaf Allow user to cancel ongoing import 2022-03-23 11:58:36 +02:00
Uku Taht
203f87520b
Ga import improvements (#1784)
* Do not link Google account for import

* Record start date

* Fix tests
2022-03-22 16:09:45 +02:00
Uku Taht
59acff02f8 Fix tests 2022-03-21 12:55:20 +02:00
Uku Taht
e1d6d119b0 Add end date to GA import 2022-03-21 12:47:27 +02:00
Uku Taht
e026148fb0 Show better error message on duplicate domain
Co-authored-by: Marko Saric <me@markosaric.com>
2022-03-10 15:11:19 -06:00
Uku Taht
e27734ed79
[Continued] Google Analytics import (#1753)
* Add has_imported_stats boolean to Site

* Add Google Analytics import panel to general settings

* Get GA profiles to display in import settings panel

* Add import_from_google method as entrypoint to import data

* Add imported_visitors table

* Remove conflicting code from migration

* Import visitors data into clickhouse database

* Pass another dataset to main graph for rendering in red

This adds another entry to the JSON data returned via the main graph API
called `imported_plot`, which is similar to `plot` in form but will be
completed with previously imported data.  Currently it simply returns
the values from `plot` / 2. The data is rendered in the main graph in
red without fill, and without an indicator for the present. Rationale:
imported data will not continue to grow so there is no projection
forward, only backwards.

* Hook imported GA data to dashboard timeseries plot

* Add settings option to forget imported data

* Import sources from google analytics

* Merge imported sources when queried

* Merge imported source data native data when querying sources

* Start converting metrics to atoms so they can be subqueried

This changes "visitors" and in some places "sources" to atoms. This does
not change the behaviour of the functions - the tests all pass unchanged
following this commit. This is necessary as joining subqueries requires
that the keys in `select` statements be atoms and not strings.

* Convery GA (direct) source to empty string

* Import utm campaign and utm medium from GA

* format

* Import all data types from GA into new tables

* Handle large amounts of more data more safely

* Fix some mistakes in tables

* Make GA requests in chunks of 5 queries

* Only display imported timeseries when there is no filter

* Correctly show last 30 minutes timeseries when 'realtime'

* Add with_imported key to Query struct

* Account for injected :is_not filter on sources from dashboard

* Also add tentative imported_utm_sources table

This needs a bit more work on the google import side, as GA do not
report sources and utm sources as distinct things.

* Return imported data to dashboard for rest of Sources panel

This extends the merge_imported function definition for sources to
utm_sources, utm_mediums and utm_campaigns too. This appears to be
working on the DB side but something is incomplete on the client side.

* Clear imported stats from all tables when requested

* Merge entry pages and exit pages from imported data into unfiltered dashboard view

This requires converting the `"visits"` and `"visit_duration"` metrics
to atoms so that they can be used in ecto subqueries.

* Display imported devices, browsers and OSs on dashboard

* Display imported country data on dashboard

* Add more metrics to entries/exits for modals

* make sure data is returned via API with correct keys

* Import regions and cities from GA

* Capitalize device upon import to match native data

* Leave query limits/offsets until after possibly joining with imported data

* Also import timeOnPage and pageviews for pages from GA

* imported_countries -> imported_locations

* Get timeOnPage and pageviews for pages from GA

These are needed for the pages modal, and for calculating exit rates for
exit pages.

* Add indicator to dashboard when imported data is being used

* Don't show imported data as separately line on main graph

* "bounce_rate" -> :bounce_rate, so it works in subqueries

* Drop imported browser and OS versions

These are not needed.

* Toggle displaying imported data by clicking indicator

* Parse referrers with RefInspector

- Use 'ga:fullReferrer' instead of 'ga:source'. This provides the actual
  referrer host + path, whereas 'ga:source' includes utm_mediums and
  other values when relevant.
- 'ga:fullReferror' does however include search engine names directly,
  so they are manually checked for as RefInspector won't pick up on
  these.

* Keep imported data indicator on dashboard and strikethrough when hidden

* Add unlink google button to import panel

* Rename some GA browsers and OSes to plausible versions

* Get main top pages and exit pages panels working correctly with imported data

* mix format

* Fetch time_on_pages for imported data when needed

* entry pages need to fetch bounces from GA

* "sample_percent" -> :sample_percent as only atoms can be used in subqueries

* Calculate bounce_rate for joined native and imported data for top pages modal

* Flip some query bindings around to be less misleading

* Fixup entry page modal visit durations

* mix format

* Fetch bounces and visit_duration for sources from GA

* add more source metrics used for data in modals

* Make sources modals display correct values

* imported_visitors: bounce_rate -> bounces, avg_visit_duration -> visit_duration

* Merge imported data into aggregate stats

* Reformat top graph side icons

* Ensure sample_percent is yielded from aggregate data

* filter event_props should be strings

* Hide imported data from frontend when using filter

* Fix existing tests

* fix tests

* Fix imported indicator appearing when filtering

* comma needed, lost when rebasing

* Import utm_terms and utm_content from GA

* Merge imported utm_term and utm_content

* Rename imported Countries data as Locations

* Set imported city schema field to int

* Remove utm_terms and utm_content when clearing imported

* Clean locations import from Google Analytics

- Country and region should be set to "" when GA provides "(not set)"
- City should be set to 0 for "unknown", as we cannot reliably import
  city data from GA.

* Display imported region and city in dashboard

* os -> operating_system in some parts of code

The inconsistency of using os in some places and operating_system in
others causes trouble with subqueries and joins for the native and
imported data, which would require additional logic to account for. The
simplest solution is the just use a consistent word for all uses. This
doesn't make any user-facing or database changes.

* to_atom -> to_existing_atom

* format

* "events" metric -> :events

* ignore imported data when "events" in metrics

* update "bounce_rate"

* atomise some more metrics from new city and region api

* atomise some more metrics for email handlers

* "conversion_rate" -> :conversion_rate during csv export

* Move imported data stats code to own module

* Move imported timeseries function to Stats.Imported

* Use Timex.parse to import dates from GA

* has_imported_stats -> imported_source

* "time_on_page" -> :time_on_page

* Convert imported GA data to UTC

* Clean up GA request code a bit

There was some weird logic here with two separate lists that really
ought to be together, so this merges those.

* Fail sooner if GA timezone can't be identified

* Link imported tables to site by id

* imported_utm_content -> imported_utm_contents

* Imported GA from all of time

* Reorganise GA data fetch logic

- Fetch data from the start of time (2005)
- Check whether no data was fetched, and if so, inform user and don't
  consider data to be imported.

* Clarify removal of "visits" data when it isn't in metrics

* Apply location filters from API

This makes it consistent with the sources etc which filter out 'Direct /
None' on the API side. These filters are used by both the native and
imported data handling code, which would otherwise both duplicate the
filters in their `where` clauses.

* Do not use changeset for setting site.imported_source

* Add all metrics to all dimensions

* Run GA import in the background

* Send email when GA import completes

* Add handler to insert imported data into tests and imported_browsers_factory

* Add remaining import data test factories

* Add imported location data to test

* Test main graph with imported data

* Add imported data to operating systems tests

* Add imported data to pages tests

* Add imported data to entry pages tests

* Add imported data to exit pages tests

* Add imported data to devices tests

* Add imported data to sources tests

* Add imported data to UTM tests

* Add new test module for the data import step

* Test import of sources GA data

* Test import of utm_mediums GA data

* Test import of utm_campaigns GA data

* Add tests for UTM terms

* Add tests for UTM contents

* Add test for importing pages and entry pages data from GA

* Add test for importing exit page data

* Fix module file name typo

* Add test for importing location data from GA

* Add test for importing devices data from GA

* Add test for importing browsers data from GA

* Add test for importing OS data from GA

* Paginate GA requests to download all data

* Bump clickhouse_ecto version

* Move RefInspector wrapper function into module

* Drop timezone transform on import

* Order imported by side_id then date

* More strings -> atoms

Also changes a conditional to be a bit nicer

* Remove parallelisation of data import

* Split sources and UTM sources from fetched GA data

GA has only a "source" dimension and no "UTM source" dimension. Instead
it returns these combined. The logic herein to tease these apart is:

1. "(direct)" -> it's a direct source
2. if the source is a domain -> it's a source
3. "google" -> it's from adwords; let's make this a UTM source "adwords"
4. else -> just a UTM source

* Keep prop names in queries as strings

* fix typo

* Fix import

* Insert data to clickhouse in batches

* Fix link when removing imported data

* Merge source tables

* Import hostname as well as pathname

* Record start and end time of imported data

* Track import progress

* Fix month interval with imported data

* Do not JOIN when imported date range has no overlap

* Fix time on page using exits

Co-authored-by: mcol <mcol@posteo.net>
2022-03-10 15:04:59 -06:00
RobertJoonas
b4992cedc1
Referrer spam blocklist (#1750)
* integrating blocklist library

* loads blocklist dependency from Github
2022-03-10 13:58:30 -06:00
RobertJoonas
6b22806e1f
Allow admin access to locked dashboards (#1710)
* added super-admin access to locked dashboards

* fixed formatting
2022-02-23 13:48:33 -06:00
Uku Taht
b0b9b9212b Fixes password authentication issue with shared links 2022-02-22 13:46:57 -06:00
Uku Taht
5b893bde39 Revert "Include 0 length pageviews in time_on_page calculation"
This reverts commit cee79e7875.
2022-02-10 14:03:56 -06:00
Uku Taht
cee79e7875 Include 0 length pageviews in time_on_page calculation 2022-02-09 16:53:05 -06:00
Ralf Zimmermann
408d95fe09
Add custom goal props to breakdown endpoint (#1578) 2022-02-01 10:09:45 -06:00
Uku Taht
0505de9b42 Add site limit to enterprise upgrade plan 2022-01-24 15:08:13 -06:00
Uku Taht
0a5f6c5886 Implement wildcard filters for the API 2022-01-21 11:49:23 -06:00
Uku Taht
cd40579740 Ignore unknown country code 2022-01-18 10:23:26 -06:00
Uku Taht
35ee2fe3fb Parse bunny header 2022-01-15 08:57:25 -06:00
Matt Colligan
8148d41e42
Add cities and regions to exported CSV data (#1485) 2022-01-04 11:08:06 +02:00
Uku Taht
711cbdfca2 Remove unused custom domain code 2021-12-20 11:49:50 +02:00
Uku Taht
15931b9346 Add tests for utm_term and content in the API 2021-12-16 16:18:16 +02:00
Uku Taht
4ea0e29620 Add UTM content and term to CSV export 2021-12-16 15:50:37 +02:00
Uku Taht
dc8d20496c Show locked message in shared link 2021-12-14 12:10:34 +02:00