Commit Graph

3057 Commits

Author SHA1 Message Date
ruslandoga
8ee4827fea
Run migrations in order across repos (#4466)
* order migrations

* migrate in 'streaks'

* cleanup

* docs

* tests

* add comment

* changelog

* continue

* run migrations tests in ci

* add pending_streams cmd

* fix ci

* add docs

* fix test

* simplify test

* only test v2+

* fix test

* return async: true

* cleanup

* add 'no pending migration'

* drop migrate/0 and pending_migrations/0

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-09 11:27:11 +02:00
ruslandoga
9ba9e2abc5
Make ClickhouseRepo task timeouts configurable in CE (#4494)
* remove ch timeouts from ce

* just infinity

* make timeout configurable

* quadruple

* add test

* improve test

* fix test filtering in ci

* just don't compile the test in ee

* fix test in ee

* maybe this would work

* rm test

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-09 11:20:06 +02:00
hq1
0031d1487d
Verification: Add super-admin report displaying diagnostics (#4554)
* Verification: Add super-admin report displaying diagnostics

* Revert LSP not catching up with formatter rules on time

* Remove redundant condition
2024-09-09 10:49:29 +02:00
Karl-Aksel Puulmann
604dde99fd
APIv2: Regex operations, consistent operators (#4488)
* Rename matches/does_not_match filters internally

These have never been exposed to the frontend/user directly, only via
APIv1 filtering syntax. As such we are free to rename these without
breaking things

* Rename function arguments for consistency, simplify

* Add support for `match`/`not_match` operators for query apiv2

These match the string against a regular expression, as defined in
https://github.com/google/re2/wiki/Syntax

* not_match -> match_not

* does_not_contain -> contains_not

Note that for backwards compatibility:
- Browser handles does_not_contain in URL
- Backend will handle does_not_contain in queries for a day where we will remove it for better autocompletion

* not_matches_wildcard -> matches_wildcard_not

* prettier

* match -> matches

* Fix and test fix for matches_wildcard against prop when prop is missing

* Custom properties support for matches/matches_not

* Restore contains_not

* Test contains and contains_not behavior for custom properties
2024-09-09 10:05:24 +03:00
ruslandoga
67d7c6522c
Change SMTP defaults (#4538)
* default smtp config updates

* tests

* improve SMTP_MIDDLEBOX_COMP_MODE decoding

* changelog

* format

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-06 10:31:23 +02:00
dependabot[bot]
4864a2c30e
Bump req from 0.5.0 to 0.5.6 (#4537)
Bumps [req](https://github.com/wojtekmach/req) from 0.5.0 to 0.5.6.
- [Release notes](https://github.com/wojtekmach/req/releases)
- [Changelog](https://github.com/wojtekmach/req/blob/main/CHANGELOG.md)
- [Commits](https://github.com/wojtekmach/req/compare/v0.5.0...v0.5.6)

---
updated-dependencies:
- dependency-name: req
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-06 10:22:52 +02:00
ruslandoga
b6859352cb
Fix tzdata (#4539)
* fix tzdata 2024b

* 2024b is buggy, use 2024a instead

* fix comma
2024-09-06 10:06:38 +02:00
hq1
b9493f687c
Replace native tooltips with simplified, generic component (#4528)
* Replace native tooltips with simplified, generic component

* Make tooltip a button?

* Revert "Make tooltip a button?"

This reverts commit adacc560f9.
2024-09-05 12:18:16 +02:00
ruslandoga
e7ac60c067
Return domain-less cookies in CE (#4482) 2024-09-05 12:05:40 +02:00
ruslandoga
654558dbf9
Remove Timex from Plausible.Timezones (#3899)
* use stdlib for timezones

* Remove unused `Timezones.to_utc_datetime/2`

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-05 11:48:07 +02:00
dependabot[bot]
3deca86201
Bump mjml from 1.5.0 to 3.1.0 (#4484)
Bumps [mjml](https://github.com/adoptoposs/mjml_nif) from 1.5.0 to 3.1.0.
- [Release notes](https://github.com/adoptoposs/mjml_nif/releases)
- [Changelog](https://github.com/adoptoposs/mjml_nif/blob/main/CHANGELOG.md)
- [Commits](https://github.com/adoptoposs/mjml_nif/compare/v1.5.0...v3.1.0)

---
updated-dependencies:
- dependency-name: mjml
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-05 11:20:30 +02:00
Uku Taht
d56d6998df
Acquisition channel (#4489)
* WIP

* Add acquisition channel

* Add detection for gclid and msclkid

* Add GA4 source categories file as external resource
2024-09-05 12:02:15 +03:00
Uku Taht
90b81b615f
Add migration for acquisition channel (#4531) 2024-09-05 11:51:16 +03:00
Karl-Aksel Puulmann
8fa3a83129
APIv2: and/or/not support (#4480)
* First approximation of AND/OR/NOT support

Broken by this:
- Goal filtering
- Table deciding
- Imports

* TableDecider handle nesting

* Query.remove_top_level_filters

* Plausible.Stats.Imported.SQL.Expression

* Handle AND/OR/NOT with imported data, create Plausible.Stats.Imported.SQL.WhereBuilder

* Add parser validations for event:goal, event:hostname and event:props:x filters top level constraints

* Move module around

* Query.get_filter -> Filters.filtering_on_dimension? in some callsites

* Filters.get_toplevel_filter

* TableDecider.sessions_join_events?, remove old method

* Transforming filters in query_optimizer

* Query API tests for and/or/not

* Reorder parser steps

* Post-merge test fixups

* Solve merge issue

* Simplify filtering_on_dimension?

* Update transformer code

* dimensions_used_in_filters min_depth option, simplify parser validations

* rename_dimensions_used_in_filter

* fix rename_dimensions_used_in_filter

* Rename a test
2024-09-04 15:44:03 +03:00
Karl-Aksel Puulmann
3310006337
Update 20240801091615_capitalize_known_sources.exs migration (#4525)
Previous migration took forever on prod, likely because Map lookups are linear time in complexity.
`transform/3` helps achieve the same functionality with the help of a hash table and updated
WHERE clause allows skipping most rows which dont need updating

Co-authored-by: Uku Taht <Uku.taht@gmail.com>
2024-09-04 13:57:28 +03:00
hq1
09999f8b0c
Disguise dogfooding LV URLs (#4529) 2024-09-04 11:36:00 +02:00
Karl-Aksel Puulmann
059b5e0cdd
Dont include imports for time:hour and time:minute dimensions (#4504)
* Dont include imports for time:hour and time:minute dimensions

Also include more information about import warnings in query results

* Update lib/plausible/stats/query_result.ex

Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>

* Revert patch

* Imported disabled graph notice (#4522)

* add explicit skip_imported_reason for unsupported interval

* stop returning information about imports from main_graph

* return warning about interval in Stats API Timeseries

* display warning bubble about interval too short for imported data

* update changelog

* improve styling of the exclamation circle icon

* return tuple from timeseries instead of map

* rename variable

* Update CHANGELOG.md

---------

Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
2024-09-04 10:53:00 +03:00
Adrian Gruntkowski
d0619aaea0
Redirect to login on expired legacy session (#4523)
After refactor, it turned out that when the _legacy_ session times out,
the conn is halted but no body is sent. This results in 500 error (`Plug.Conn.NotSentError`).

This PR fixes it by redirecting to login page.
2024-09-03 21:48:05 +02:00
ruslandoga
530d290678
Adapt site_setup_success email for CE (#4496)
* adapt site_setup_success email for ce

* add tests

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-03 14:50:08 +02:00
ruslandoga
47c2aec2e8
Update bamboo_mua to fix headers issue (#4474)
* update bamboo_mua to fix headers issue

* changelog

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-03 14:09:57 +02:00
Karl-Aksel Puulmann
3eee780557
Handle more special characters in parseSearchFragment (#4521)
Some old filters with spaces and | characters broke
2024-09-03 15:04:36 +03:00
hq1
acefd792fb
Installation: don't crash when socket reconnects and finds no changes (#4520) 2024-09-03 13:59:29 +02:00
ruslandoga
6c7f123d2c
Add better assets compression to CE (#4476)
* add better assets compression to ce

* changelog

* fix conditional compilation

* improve changelog message

* don't need zstd
2024-09-03 13:54:11 +02:00
ruslandoga
4a36148f9b
Better Google error messages in CE (#4485) 2024-09-03 13:44:34 +02:00
ruslandoga
2634ff7673
Fix HTML-in-JSON in ErrorView (#4500)
* fix html-in-json errors

* add tests

---------

Co-authored-by: hq1 <hq@mtod.org>
2024-09-03 13:35:07 +02:00
hq1
53d94b5b1b
Remove custom background from "Choose Plan" page (#4516)
* Remove custom background on "Choose Plan" page

* Avoid `UserAuth` test failing due to bad timing

---------

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-03 13:17:26 +02:00
Adrian Gruntkowski
10629d13a1
Fix faulty UserSessionTouchTest setup (#4517) 2024-09-03 12:53:06 +02:00
Adrian Gruntkowski
373d4dd665
Implement token-based sessions (#4463)
* Turn `Plausible.Auth.UserSession` into full schema

* Implement token based sessions and use them as default

* Ignore expired user sessions during retrieval from DB

* Implement plug bumping user session last used and timeout timestamps

* Implement Oban worker removing expired user sessions with grace period

* Implement legacy session conversion on touch, when applicable

* Update `UserAuth` moduledoc

* Extend `UserAuth` tests to account for db-backed session tokens

* Update CHANGELOG

* Add tests for `UserSessionTouch` plug

* Add test for `CleanUserSessions` worker

* Add logging of legacy session retrievals

* Use single update permitting stale records  when touching user session

* Don't fetch session and user for external API endpoints (/api/event too)

* Refactor `Users.with_subscription/1` and expose helper query

* Skip fetching session in legacy `SessionTimeoutPlug`

* Rely on user session assign from `AuthContext` in `SentryContext`

* Silence legacy session warnings in `UserSessionTouchTest`

* Rely on session assign from `AuthPlug` in `SuperAdminOnlyPlug`

* Change `UserAuth` to get session, user and last subscription in one go

* Avoid refetching user session in `AuthorizeSiteAccess` plug

* Fix code formatting

* Refactor `UserAuth.get_user_token/1` (h/t @aerosol)

* Remove bogus empty opts from `scope` declarations in router

* Only touch session once an hour and keep `user.last_seen` in sync

* Bring back logging of legacy token use
2024-09-03 11:34:37 +02:00
Adrian Gruntkowski
533bf90329
Create user_sessions table (#4511) 2024-09-03 10:02:43 +02:00
Karl-Aksel Puulmann
f18d004584
Fix property filter suggestions 500 error when property hasn't been selected (#4503)
Broken by APIv2 refactors.

Relevant sentry error: https://sentry.plausible.io/organizations/plausible/issues/448/?query=is%3Aunresolved&referrer=issue-stream&statsPeriod=14d&stream_index=4
2024-09-03 10:37:17 +03:00
Adrian Gruntkowski
8a1c6e0913
Enforce sequential processing of session events (#4493)
* Create a regression demonstration test for race condition

* Use `ConCache.isolated/1` to force sequential processing of session events

* Revise comment in regression test

* Put lock call behind cache adapter API

* Add more explicit handling of failing lock

NOTE: Apparent double execution of lock function needs to be investigated.

* Improve slow lock cases tests

* Reduce number of session cache locks and instrument them w/ telemetry

* Format

---------

Co-authored-by: Adam Rutkowski <hq@mtod.org>
2024-09-03 09:29:32 +02:00
RobertJoonas
93c9061ab2
Proper fix for realtime timeseries queries (#4509)
* enforce same timezones in date_range param

* use query.date_range.first.timezone as query.timezone
2024-09-03 10:12:50 +03:00
hq1
caef1e80f4
Allow installation lookups by superadmins (#4510) 2024-09-03 07:56:32 +02:00
Joris Drenth
49ff00252e
Fix documentation link for Google integration (#4472)
* Fix documentation link for Google integration

* Fix documentation link for Google integration (test update)

* Update lib/plausible_web/templates/site/settings_search_console.html.heex

---------

Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
2024-09-02 21:19:43 +02:00
hq1
979764d010
Onboarding improvements improvements (#4507)
* Make it clear(er) that info links are clickable

* Don't pollute browser history with checkbox states
2024-09-02 20:06:27 +02:00
RobertJoonas
5e26e2c97d
Fix realtime periods returning wrong date range for non-UTC timezone sites (#4508)
* fix realtime

* mix format
2024-09-02 19:21:17 +03:00
Artur Pata
9b59e95c33
Support order_by in URL params of breakdown endpoints (#4502)
* Support order_by in URL params of breakdown endpoints
2024-09-02 15:00:18 +03:00
hq1
07d3582746
Fix password reset link (#4505) 2024-09-02 13:06:50 +02:00
Uku Taht
77248c8800
Add data migration for capitalizing sources (#4418) 2024-09-02 13:59:58 +03:00
hq1
e3af1a317d
Onboarding improvements (#4459)
* Migration: add installation meta

* Update site schema with installation meta

* Remove VERIFICATION_ENABLED env var

* Add context API to create/remove special goals

* Add context api to update installation meta

* Remove verification enabled check

* Update new progress flow definitions

* Update generic components

* Remove internal /status API

* Implement installation live view

* Update traffic change notifier link

* Update verification, no more modal

* Update routes

* Remove focus.html - will unify everything under app layout

* Fix broken link

* Update templates with focus_box mostly

* Update controller tests

* Update controllers and stop using the focus layout

* copy changes

* Update verification.ex

* Remove dead template

* Update settings_general.html.heex

* Update copy in tests

* Update installation.ex

* Remove dangling dot

* Fix link

* Update installation.ex

* Update installation.ex

* Better tooltips?

* Simpler labels

* Revert "Simpler labels"

This reverts commit 797560ef82f2067458b03b884be5aecc8fdc72bc.

* Add copy to clipboard link and fix snippet's dark mode

* Offer installation detection skip only if ws connected

* Put COPY link at the bottom with background

* Make tooltips link to docs

* Fix cherry-pick gone wrong

* Hide tooltips on mobile screens

* WIP: 404 tracking wizard

* Revert "WIP: 404 tracking wizard"

This reverts commit a9c9c79bbd.

* Update lib/plausible_web/live/components/verification.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Update lib/plausible_web/live/installation.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Use current_user from socket.assigns

* Update lib/plausible_web/live/installation.ex

Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>

* Use current_user from socket.assigns

* Use conn.private to steer verification tests

* Drop non-sticky tooltip in favour of component parametrization

Co-authored-by: Artur Pata <artur.pata@gmail.com>

* Reapply "WIP: 404 tracking wizard"

This reverts commit 3ba81671d7.

* Fix installation tests including 404 tracking

* Fixup the tooltip component

* Format

* Update installation.ex

* Put flash whenever installation option changes

* Use last known installation type on domain change

* Extract user flow definition to provide compile-time checks

* See if this helps running CE migrations successfully

* Use `styled_link` on registration/login views

* Don't crash when there's no conn.private carried over

* Format

* Push "Determining installation type" message a bit lower

* Use links and footer lists uniformly

This commit introduces a `<.focus_list/>` component
for rendering focus box footer links with colored
discs. It also equips generic link components
with the ability of sending non-GET requests
along with CSRF token, so we can apply uniform
styling and stop using legacy Phoenix link tags.

cc @zoldar @apata

* ws 👾

* Render more descriptive flashes on script config change

---------

Co-authored-by: Marko Saric <34340819+metmarkosaric@users.noreply.github.com>
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
Co-authored-by: Artur Pata <artur.pata@gmail.com>
2024-09-02 12:49:54 +02:00
RobertJoonas
f04c47f881
Support realtime periods in API v2 (#4469)
* add realtime date_ranges into the private API schema

This commit starts parsing date ranges into a new NaiveDateTimeRange
struct, rather than a simple Date.Range.

* transform realtime labels into negative integers + test

* move schema type argument to last position in helper functions

* allow passing a date param + tests

* Update test/plausible/stats/query_parser_test.exs

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* Update test/plausible/stats/query_parser_test.exs

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* Update test/plausible/stats/query_parser_test.exs

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* Update test/plausible/stats/query_parser_test.exs

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>

* keep test file structure consistent

* Turn NaiveDateTimeRange into DateTimeRange

* change 'now' field from NaiveDateTime to DateTime in v2 query

* fix minute interval labels + add missing tests

* return query_result.date_range as iso8601 timestamps with timezone

* allow timestamps with tz as date_range arguments in API v2

* delete Plausible.Timezones.to_utc_datetime

* simplify returning comparison periods

* add comment about realtime not supported in comparisons

* pass only now instead of test_opts

* drop redundant else branch

* separate tests

* stick to a single check_date_range function in tests

* fix credo error

---------

Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
2024-09-02 12:56:58 +03:00
Adrian Gruntkowski
a91df10bbf
Fix invalid parameter destructuring in navigate calls (#4495) 2024-09-02 08:58:42 +02:00
ruslandoga
19ecd3d0ee
Update Postgrex SSL config (#4460)
* update postgrex config

* enable ssl only if DATABASE_CACERTFILE is set

* update tests

* changelog

---------

Co-authored-by: Cenk Kücük <cenk@plausible.io>
2024-08-30 11:11:36 +01:00
Adrian Gruntkowski
e9dd895d6c
Do not try querying imported data for unsupported props (#4467)
* Do not try querying imported data for unsupported props

* Remove unnecessary `unquote`s

Co-authored-by: hq1 <hq@mtod.org>

* Add respective regression test to APIv2 tests

---------

Co-authored-by: hq1 <hq@mtod.org>
2024-08-30 10:55:31 +02:00
hq1
05136398cf
Migration: add installation meta (#4486) 2024-08-29 11:05:36 +02:00
ruslandoga
b73bdf932f
add /var/lib/plausible (#4275)
Co-authored-by: Cenk Kücük <cenk@plausible.io>
2024-08-28 13:44:53 +02:00
ruslandoga
3a72bb8250
update ecto (#4475)
Co-authored-by: hq1 <hq@mtod.org>
Co-authored-by: Cenk Kücük <cenk@plausible.io>
2024-08-28 13:28:09 +02:00
ruslandoga
7efcd9621f
update alpine version to 3.20.1 (#4478)
Co-authored-by: hq1 <hq@mtod.org>
2024-08-28 13:11:37 +02:00
Artur Pata
51fba1c2cf
Add tests for FE date calculations, fix niggles with site context (#4471)
* Adds tests for FE date calculations during DST changeover times
* Fixes unparsed offset in SiteContext
* Fixes unparsed embedded in SiteContext
* Adds tests for SiteContext
2024-08-28 11:14:29 +03:00
ruslandoga
8b06c91b0c
fix dockerfile warning (#4477) 2024-08-28 08:16:06 +02:00