* Add has_imported_stats boolean to Site
* Add Google Analytics import panel to general settings
* Get GA profiles to display in import settings panel
* Add import_from_google method as entrypoint to import data
* Add imported_visitors table
* Remove conflicting code from migration
* Import visitors data into clickhouse database
* Pass another dataset to main graph for rendering in red
This adds another entry to the JSON data returned via the main graph API
called `imported_plot`, which is similar to `plot` in form but will be
completed with previously imported data. Currently it simply returns
the values from `plot` / 2. The data is rendered in the main graph in
red without fill, and without an indicator for the present. Rationale:
imported data will not continue to grow so there is no projection
forward, only backwards.
* Hook imported GA data to dashboard timeseries plot
* Add settings option to forget imported data
* Import sources from google analytics
* Merge imported sources when queried
* Merge imported source data native data when querying sources
* Start converting metrics to atoms so they can be subqueried
This changes "visitors" and in some places "sources" to atoms. This does
not change the behaviour of the functions - the tests all pass unchanged
following this commit. This is necessary as joining subqueries requires
that the keys in `select` statements be atoms and not strings.
* Convery GA (direct) source to empty string
* Import utm campaign and utm medium from GA
* format
* Import all data types from GA into new tables
* Handle large amounts of more data more safely
* Fix some mistakes in tables
* Make GA requests in chunks of 5 queries
* Only display imported timeseries when there is no filter
* Correctly show last 30 minutes timeseries when 'realtime'
* Add with_imported key to Query struct
* Account for injected :is_not filter on sources from dashboard
* Also add tentative imported_utm_sources table
This needs a bit more work on the google import side, as GA do not
report sources and utm sources as distinct things.
* Return imported data to dashboard for rest of Sources panel
This extends the merge_imported function definition for sources to
utm_sources, utm_mediums and utm_campaigns too. This appears to be
working on the DB side but something is incomplete on the client side.
* Clear imported stats from all tables when requested
* Merge entry pages and exit pages from imported data into unfiltered dashboard view
This requires converting the `"visits"` and `"visit_duration"` metrics
to atoms so that they can be used in ecto subqueries.
* Display imported devices, browsers and OSs on dashboard
* Display imported country data on dashboard
* Add more metrics to entries/exits for modals
* make sure data is returned via API with correct keys
* Import regions and cities from GA
* Capitalize device upon import to match native data
* Leave query limits/offsets until after possibly joining with imported data
* Also import timeOnPage and pageviews for pages from GA
* imported_countries -> imported_locations
* Get timeOnPage and pageviews for pages from GA
These are needed for the pages modal, and for calculating exit rates for
exit pages.
* Add indicator to dashboard when imported data is being used
* Don't show imported data as separately line on main graph
* "bounce_rate" -> :bounce_rate, so it works in subqueries
* Drop imported browser and OS versions
These are not needed.
* Toggle displaying imported data by clicking indicator
* Parse referrers with RefInspector
- Use 'ga:fullReferrer' instead of 'ga:source'. This provides the actual
referrer host + path, whereas 'ga:source' includes utm_mediums and
other values when relevant.
- 'ga:fullReferror' does however include search engine names directly,
so they are manually checked for as RefInspector won't pick up on
these.
* Keep imported data indicator on dashboard and strikethrough when hidden
* Add unlink google button to import panel
* Rename some GA browsers and OSes to plausible versions
* Get main top pages and exit pages panels working correctly with imported data
* mix format
* Fetch time_on_pages for imported data when needed
* entry pages need to fetch bounces from GA
* "sample_percent" -> :sample_percent as only atoms can be used in subqueries
* Calculate bounce_rate for joined native and imported data for top pages modal
* Flip some query bindings around to be less misleading
* Fixup entry page modal visit durations
* mix format
* Fetch bounces and visit_duration for sources from GA
* add more source metrics used for data in modals
* Make sources modals display correct values
* imported_visitors: bounce_rate -> bounces, avg_visit_duration -> visit_duration
* Merge imported data into aggregate stats
* Reformat top graph side icons
* Ensure sample_percent is yielded from aggregate data
* filter event_props should be strings
* Hide imported data from frontend when using filter
* Fix existing tests
* fix tests
* Fix imported indicator appearing when filtering
* comma needed, lost when rebasing
* Import utm_terms and utm_content from GA
* Merge imported utm_term and utm_content
* Rename imported Countries data as Locations
* Set imported city schema field to int
* Remove utm_terms and utm_content when clearing imported
* Clean locations import from Google Analytics
- Country and region should be set to "" when GA provides "(not set)"
- City should be set to 0 for "unknown", as we cannot reliably import
city data from GA.
* Display imported region and city in dashboard
* os -> operating_system in some parts of code
The inconsistency of using os in some places and operating_system in
others causes trouble with subqueries and joins for the native and
imported data, which would require additional logic to account for. The
simplest solution is the just use a consistent word for all uses. This
doesn't make any user-facing or database changes.
* to_atom -> to_existing_atom
* format
* "events" metric -> :events
* ignore imported data when "events" in metrics
* update "bounce_rate"
* atomise some more metrics from new city and region api
* atomise some more metrics for email handlers
* "conversion_rate" -> :conversion_rate during csv export
* Move imported data stats code to own module
* Move imported timeseries function to Stats.Imported
* Use Timex.parse to import dates from GA
* has_imported_stats -> imported_source
* "time_on_page" -> :time_on_page
* Convert imported GA data to UTC
* Clean up GA request code a bit
There was some weird logic here with two separate lists that really
ought to be together, so this merges those.
* Fail sooner if GA timezone can't be identified
* Link imported tables to site by id
* imported_utm_content -> imported_utm_contents
* Imported GA from all of time
* Reorganise GA data fetch logic
- Fetch data from the start of time (2005)
- Check whether no data was fetched, and if so, inform user and don't
consider data to be imported.
* Clarify removal of "visits" data when it isn't in metrics
* Apply location filters from API
This makes it consistent with the sources etc which filter out 'Direct /
None' on the API side. These filters are used by both the native and
imported data handling code, which would otherwise both duplicate the
filters in their `where` clauses.
* Do not use changeset for setting site.imported_source
* Add all metrics to all dimensions
* Run GA import in the background
* Send email when GA import completes
* Add handler to insert imported data into tests and imported_browsers_factory
* Add remaining import data test factories
* Add imported location data to test
* Test main graph with imported data
* Add imported data to operating systems tests
* Add imported data to pages tests
* Add imported data to entry pages tests
* Add imported data to exit pages tests
* Add imported data to devices tests
* Add imported data to sources tests
* Add imported data to UTM tests
* Add new test module for the data import step
* Test import of sources GA data
* Test import of utm_mediums GA data
* Test import of utm_campaigns GA data
* Add tests for UTM terms
* Add tests for UTM contents
* Add test for importing pages and entry pages data from GA
* Add test for importing exit page data
* Fix module file name typo
* Add test for importing location data from GA
* Add test for importing devices data from GA
* Add test for importing browsers data from GA
* Add test for importing OS data from GA
* Paginate GA requests to download all data
* Bump clickhouse_ecto version
* Move RefInspector wrapper function into module
* Drop timezone transform on import
* Order imported by side_id then date
* More strings -> atoms
Also changes a conditional to be a bit nicer
* Remove parallelisation of data import
* Split sources and UTM sources from fetched GA data
GA has only a "source" dimension and no "UTM source" dimension. Instead
it returns these combined. The logic herein to tease these apart is:
1. "(direct)" -> it's a direct source
2. if the source is a domain -> it's a source
3. "google" -> it's from adwords; let's make this a UTM source "adwords"
4. else -> just a UTM source
* Keep prop names in queries as strings
* fix typo
* Fix import
* Insert data to clickhouse in batches
* Fix link when removing imported data
* Merge source tables
* Import hostname as well as pathname
* Record start and end time of imported data
* Track import progress
* Fix month interval with imported data
* Do not JOIN when imported date range has no overlap
* Fix time on page using exits
Co-authored-by: mcol <mcol@posteo.net>
* Add `utm_content` and `utm_term`.
Support `utm_content` and `utm_term` as requested in #515.
* Add dropdown for UTM options
* Remove utm_content and term from filter modal for now
Co-authored-by: Blender Defender <defenderblender@gmail.com>
* Merge branch 'plausible_master'
* Add City level details
* Add City level details
* Use ISO codes instead of geoname_id for subdivisions
* Add easier way to configure geolocation database
* Add workflow for dev branch
* Correct clickhouse migration
* Translate subdivision names
* Translate city names
* WIP
* Region and country filters
* Fix region filter
* Remove region_name when removing region filter
* Add modals for regions and cities
* Remove dead code
* WIP
* Revert "WIP"
This reverts commit 3202bf2fe9.
* Feature flag to hide cities when deployed
* Add changelog entry
* Remove unused code
* Remove unused variables
* Fix test
Co-authored-by: AymanTerra <aymanterra@yahoo.com>
and so was just annoying
2. Get rid of all existing ESLint errors.
2a. Turned off `react/display-name` because I couldn't figure out how to
make that wrapped component have a display name. If anyone can figure it
out, that'd be great, because that makes things nicer when using the
React debugger.
2b. The part where it says `plausible()` is undefined in `app.js` is
bothering me. I disabled the check because I can't figure out where that
actually comes from to put in the proper import.
2c. Told ESLint we're using Babel.
3. Added `npm run format` and `npm run lint` commands.
* Add filter groups
* New flow for filters
* Visual consistency
* Mobile improvements to dropdown
* Do not let filters wrap on a new line
* Fix country filter
* Add mix format to pre-commit configuration
* Make eslint happy
* Fix formatting
* Use transition from headlessUI
* First pass on manual filter & path regex/negated filters
Still needs:
- Form structure on filter modal
- Edit filter button
- Filter dropdown UI improvement
- Filter modal mount data collection
- Tests
- Potentially negating other filters
* Second pass - mostly everything user-facing is done
Still needs:
- Tests
- Potentially negating other filters
- Potentially some code cleanup
* Bugfix in realtime view, formatting
* Fixes editing UX on list view
* Adds tests for exclusions and wildcards
* Various UI Improvements
- Makes edit buttons full-length & properly sized
- Adds remove filter button in edit menu
* Changelog
* Makes requested changes, adds different version of filter button
* Makes colorings on top bar elements consistent
* Initial Pass
* Adds support for page visits counting by referrer
* Includes goal selection in entry and exit computation
* Adds goal-based entry and exit page stats, formatting, code cleanup
* Changelog
* Format
* Exit rate, visit duration, updated tests
* I keep forgetting to format :/
* Tests, last time
* Fixes double counting, exit rate >100%, relevant tests
* Fixes exit pages on filter and goal states
* Adds entry and exit filters, fixes various bugs
* Fixes discussed issues
* Format
* Fixes impossible case in tests
Originally, there were only 2 pageviews for `test-site.com`,`/` on `2019-01-01`, but that doesn't make sense when there were 3 sessions that exited on the same site/date.
* Format
* Removes boolean function parameter in favor of separate function
* Adds support for queries that use `page` filter as `entry-page`
* Format
* Makes loader/title interaction in sources report consistent
* Prevent choosing a date before site was added in datepicker
Fixes#446
* change date navigation arrows to real buttons and disable them when date is invalid for site
* simplify. ditch LinkButton and just use button. make lint a little happier
* put back dark mode for date nav buttons
* enforce line length limit
* fix a11y lint errors
* correct event/e
* Initial Implementation
Still needs clear all functionality, various styles and spacing, and overall cleanup
* Second pass
Most of the styling issues are still present, but collapsing/expanding logic should be complete
* Third pass
* Refactors, comments, cleans up filters code
* Flatpickr color+placement changes
* Last(?) pass on UI
* Actually fixes sticky dates without breaking navigation
* Changelog
* Puts this back where it goes
* Updates based on @ukutaht's comments
- Removes Screenclass hook in favor of simple viewport size storage
- Uses class-based component
- Removes recheck variable in favor of tri-state wrapped
- General formatting & code cleanup
This centralizes updating the search parameters
It's needed in order to implement proper history handling
Previously navigating to a new period filter would not revert if going back in history
The new function will update history entry to include the period if a new period is navigated to
This way going back in history will correctly revert the period filter
* Collect browser and OS version
* Display browser version
* Show operating system versions
* Device categorization
* Treat headless chrome like a bot
* Ignore events from automated browsers
* Only take major and minor of the version
* Add tests
* Add CHANGELOG entry
* Add changelog entry for bots
* Store empty value as browser when unknown
* Ability to add event metadata
* Close Dropdown on outside click
* Show (none) value in metadata breakdown
* Allow filtering for metadata key/val pairs
* Use correct clickhouse_ecto
* Better naming for meta filter
* Add tests
* Add changelog entry
* Remove change made for testing
* Filter by screen size
* Filter by browsers
* Add filter for Operating system
* Add changelog entry
* Add filter for countries
* Add tests for filters
* Show utm_medium, utm_source, and utm_campaign in sources modal
* Allow filtering by UTM tags
* Integrate filters with URL bar
* Add CHANGELOG entry
* Refresh modal when filter changes
* Remove Direct / None from campaign results
* Add UTM tabs to top sources report
* Add pagination
* Remove dropdown from top sources popup
* Fix bug in clickhouse_ecto
* Remove referrers_for_goal
* Make sure UTM tags work OK with goals
* Make source tab selection sticky
* Consistent styling for devices and source tabs
* Add noref in realtime to utm tabs
* Fix tests
* WIP
* Add filter option for page
* Use JOIN to get correct top numbers
* Filter google search terms by page
* Allow filtering by page from modal
* More robust base url for google search console integration
* Fix tests
* Referrer drilldown
* Render external links for referrer drilldown modal
* Rename top pages to entry pages
* Add google keywords to referrer drilldown
* Fix link in source modal
* Remove back link from referrer drilldown
* Fix tests
* Mix format
* Remove unused function
* Get top stats from the events table
* Revert "Get top stats from the events table"
This reverts commit 67384e2d35.
* Fix pages modal title
* Improve pages report in referrer drilldown
* Fix label in referrer drilldown
* Auto-updating dashboard with realtime info
* Remove extra route
* Draw list of countries next to the map
* Nice animations
* Do not show bounce rates in realtime modals
* Update countries and devices in realtime
* Remove unused component
* Show total pageviews in the last 30 minutes
* Show proper labels
* Remove unnecessary z-index
* Fix label for main graph
* Fix compiler warnings
* Add tests
* Fix copy pluralizations
* Fix copy in countries modal
* Real-time -> Realtime
* Looser test assertion
* Show last 30 minutes conversions on realtime report
* Remove EventTarget API because it doesn't work on Safari
* Get referrer drilldown from sessions table
* Fix failing tests
* Load dashboard with react
* Rename stast2 to dashboard
* Save timeframe on the frontend
* Implement current visitors
* Implement comparisons
* React to route changes
* Add modals
* Show number of visitors on hover
* Show 'Today' for today
* Add 30 days
* Show referrer drilldown
* Arrow keys to go back and forward
* Improve comparisons UI
* Fix dropdown when clicking on it
* Verify API access in a memoized fashion
* Test API access
* Test stats through controller
* Move map formatting from stats controller to stats
* Remove unused code
* Remove dead code from query
* Remove dead code from stats templates
* Add stats view test back in
* Render modal inside the modal component
* Implement google search terms
* Add explanation for screen sizes
* Separate dashboard JS from the app js
