no issue
- The sleep method has been used in 8 modules reimplementing the same thing over and over again. It's usually a sign of async event processing outside of the request/response loop. It's good to have a single point of implementation for a "hack" like this, so we could track it easier and address the even processing delay in a more optimal way centrally if it ever becomes a bottleneck
fixes https://github.com/TryGhost/Team/issues/2346
- Adds email batch browse endpoint
- Adds email recipient failures browse endpoint
- Adds new fixtures and E2E tests for the new API
- Added support for snapshot tests to have 'nullable' types.
refs https://github.com/TryGhost/Team/issues/2339
- Includes a new pattern in the job manager that allows us to properly
await jobs.
- Added new convenience mocking methods to stub settings
- Tests the main flows for bulk sending:
- Sending in multiple batches
- Sending to multiple segments
- Handling a failed batch and retrying that batch
- Fixes bug in batch generation (ordering not working)
In a different PR I'll add more detailed tests.
closes https://github.com/TryGhost/Team/issues/2324
- It seemed like the "limit" query parameter did not work properly returning multiple entries from the endpoint. In reality the whole query string was ignored because of an error in the "filter" part of the query ^_^
closesTryGhost/Team#2313
- Added Sent event to Post analytics and Members feed. Now post can be
Sent or Received or Bounced.
- Excluded Delivered event from Sent filter on backend.
refs https://github.com/TryGhost/Ghost/security/advisories/GHSA-9gh8-wp53-ccc6
refs https://github.com/TryGhost/Toolbox/issues/465
- Bookshelf relations allows us to edit relational records by default, which was used liberally in the codebase.
- Not having a clear track record of editable relations left the model layer prone to triggering unwanted nested saves and created a vulnerability where members were able to edit newsletter settings.
- With explicit editable relations it's easier to keep track of relations having editable access to related records. Makes the relational data modification pattern safer to use too.
- Anyone running 5.x should update to 5.24.1
Credits: Dave McDaniel and other members of [Cisco Talos](https://talosintelligence.com/vulnerability_reports)
fixes https://github.com/TryGhost/Team/issues/2284
New batch sending flow (still WIP). Logs the sent emails instead of actually sending them. Unit tests are coming in later commits.
refs https://github.com/TryGhost/Team/issues/2280
We are moving away from storing html and plaintext on email and instead will store the email data in source and source_type columns which allows us to store the email in other formats like mobiledoc and lexical. Storing in those formats allows greater flexibility for later html generation
- adds new `source` column that stores `mobiledoc`/`lexical`/`html` data for a newsletter
- adds new `source_type` column that stores one of `mobiledoc`/`lexical`/`html` to identify type of source
fixes https://github.com/TryGhost/Team/issues/2282
Added a new email service package that is used when the email stability
flag is enabled. Currently not yet implemented so will throw an error
for all entry points (if flag enabled).
Removed usage of `labs.isSet.bind` across the code, because that breaks
the stubbing of labs by `mockManager.mockLabsEnabled` and
`mockManager.mockLabsDisabled`. `flag => labs.isSet(flag)` should be
used instead.
All email depending tests now disable the `emailStability` feature flag
to keep the tests passing + make sure we still run all the tests for the
old flow while the email stability package is being built.
refs https://github.com/TryGhost/Team/issues/2268
The approach of using the service to lead email suppression data as
opposed to bookshelf relations allows us to wire things up without
having implemented the database. The getBulkSuppressionData allows us to
do this without much of a DB performance hit.
closes https://github.com/TryGhost/Team/issues/2126
- Cleaned up the following GA flags: `newsletterPaywall`, `freeTrial`, `compExpiring`, `searchHelper`, `emailAlerts`, `fixNewsletterLinks`.
- updated the cover image to be simpler
- made the change in text fixtures as well, just to keep the fixtures in sync
Co-authored-by: Hannah Wolfe <github.erisds@gmail.com>
fixes https://github.com/TryGhost/Team/issues/2085
Don't load relations we don't need anymore for the posts table. And
reload the individual post when we open the analytics page with more
relations that we actually need.
fixes https://github.com/TryGhost/Team/issues/1903
MembersAgent.loginAs sends email, asynchronously via events. Which
conflicts with tests that also test emails. We cannot properly await
these events, so this is currently fixed with a timeout of 200ms. But
this was too random and unreliable.
closes https://github.com/TryGhost/Team/issues/2222
Whilst we were checking for Stripe objects being active, we were not
checking for them existing in Stripe. This adds handling to all read
request to Stripe in the payment link flow, so that we can gracefully
handle deleted objects.
We've also included an automated test which fails without this fix.
We've also improved the query to find Stripe Prices which will result
in less request to the Stripe API to check if it is valid.
refs https://github.com/TryGhost/Ghost/commit/1f300fb781f0
The full customer object was not being passed to the StripeAPI service
when it already exists, this was resulting in inconsistent behaviour when
sending the customerEmail param to the API, causing `invalid_email`
errors to be thrown from Stripe and breaking the checkout.
closes https://github.com/TryGhost/Team/issues/2196
We were incorrectly assuming that all requests would have the
`customerEmail` passed in the body. Instead we were incorrectly
passing `undefined` or `''` as the `customerEmail` property to stripe,
which resulted in a validation error.
We've updated the code to pass `null` in the case of a falsy value,
which the Stripe API handles without error.
closes https://github.com/TryGhost/Team/issues/2195
The issue here is two-fold, and specific to using Offers so was not
caught by any automated tests. First, we were incorrectly comparing
the tier.id to the offer.tier.id - this is because the Tier objects id
property is an instance of ObjectID rather than a string.
Secondly we were passing through the cadence parameter from the
request body, but when using Offers this is not including in the
request, so we must pull the data off of the Offer object instead and
pass that to the payments service.
closes https://github.com/TryGhost/Ghost/issues/15740
The validation function for a Tier description was not returning the
validated value, which meant we were unable to set the Tier
description.
- These tests are very slow, and make the build fail about 2/3 times
- Temporarily skipping until we can fix, as I want to get all our outstanding hacktoberfest PRs merged
fixes https://github.com/TryGhost/Team/issues/2175
- New event type `aggregated_click_event` that is disabled by default in all the existing activity feeds
- This returns click events, but only the first click events for each member/post combination.
- It includes the total count of unique link clicks for that member on that post combination
- Had to resort to some custom knex queries to make this work easily
- Requires `@tryghost/bookshelf-pagination@0.1.31`, included in `@tryghost/bookshelf-plugins@0.6.1` (this fixes an issue with custom selects breaking the total count query of pages)
- Went a bit overboard with the pagination tests to cover as much unknown edge cases as possible
fixes https://github.com/TryGhost/Team/issues/2129
- This changes how the activity feed API parses the filter.
- We now parse the filter early to a MongoDB filter, and split it in two. One of the filters is applied to the pageActions, and the other one is used individually for every event type. We now allow to use grouping and OR's inside the filters because of this change. As long as we don't combine filters on 'type' with other filters inside grouped filters or OR, then it is allowed.
- We make use of mongoTransformer to manually inject a mongo filter without needing to parse it from a string value again (that would make it a lot harder because we would have to convert the splitted filter back to a string and we currently don't have methods for that).
- Added sorting by id for events with the same timestamp (required for reliable pagination)
- Added id to each event (required for pagination)
- Added more tests for filters
- Added test for pagination
- Removed unsued getSubscriptions and getVolume methods
Used new mongo utility methods introduced here: https://github.com/TryGhost/NQL/pull/49
refs https://github.com/TryGhost/Team/issues/2077
- Members and Posts test suites were using a broad tiers property matcher, which is an anti-pattern for snapshot tests. Without more specific snapshots it would be very hard to track down tier-related breaking changes!
- This change is groundwork for a refactor coming in tier usage at API's output serializers
closesTryGhost/Team#2159
- Added column to email table
- Hide the feedback tab on frontend depending on the column value
Co-authored-by: Daniel Lockyer <daniellockyer@fastmail.com>
fixes https://github.com/TryGhost/Team/issues/2137
For the analytics page, we need the sent events to show up immediately
after sending an email. Otherwise we need to wait for emails to be
marked as received (which takes too long) before being able to show them
on the analytics page.
This adds the email_sent_event, which is hidden by default everywhere
and used on the analytics page.
refs https://github.com/TryGhost/Team/issues/2116
- allows site owners to edit a link in a post that has already been sent out, fixing any typos or other mistakes
- resets click counter for the edited link back to 0 so site owners can see the clicks on new link, doesn't change the overall click count
refs https://github.com/TryGhost/Team/issues/2135
The email link redirects on Pro are cached as 302 redirects in Varnish, so we're missing further clicks after the first one for each member, until the cache is invalidated. This change invalidates cache on link edits to ensure that we correctly redirect members to updated link everytime
fixes https://github.com/TryGhost/Team/issues/2090
- This changes how sentiment is exposed in the API. Now it is exposed as a `sentiment` relation, directly on the model (no longer in counts). Internally we still use `count.sentiment`.
- Content API users (and themes) can include the 'sentiment' relation and order by sentiment.
- Updated Admin to use sentiment instead of count.sentiment
fixes https://github.com/TryGhost/Team/issues/2091
fixes https://github.com/TryGhost/Team/issues/2089
- Added new fixtures to make testing easier for the activity feed
- Improved E2E test coverage of activity feed with separate test file
- Added data.post_id filter to enable filtering by events related to a
given post
- Fixed return types in JSDoc of test agents (TypeScript interprets
these as `typeof Agent` if we don't add `InstanceType<Agent>`)
- Added total pagination metadata to activity feed API (to allow a basic
type of pagination using filters)
fixes https://github.com/TryGhost/Team/issues/2084
- When audience feedback is enabled, we use a single 'conversions' count instead of having separate ones for signups and paid conversions.
- The analytics component is separated so we can change it without breaking the existing page.
refs https://github.com/TryGhost/Team/issues/1765
In order to better handle deleted objects in Stripe we want to decouple
Members from Stripe.
These changes allow us to have the Tier concept completely independent
of the Stripe tables, such that the Stripe data can be generated as/when
it's needed - which will help to protect against missing data.
closes: https://github.com/TryGhost/Ghost/issues/14973
- When fetching content using a non-standard charset, characters were notproperly decoded to utf-8 resulting in mangled text in the editor -> Detect charset and use iconv to decode the page text
- When requesting a non bookmark card, if no oembed data could be foundand we fallback to bookmark, a second network request to fetch the content was issued. This seemed unnecessary -> refactored to avoid that
fixes https://github.com/TryGhost/Team/issues/2054
This change adds the sentiment and positive_feedback counts to the posts models. This change isn't really ideal because there are some problems here:
- sentiment isn't really a count
- we don't need to include the sentiment and positive_feedback as a default for posts (but the same is true for attribution)
It would make sense to move this to separate endpoints that only fetch the analytics for a given post when the analytics page is opened. But for our initial skateboard version of audience feedback this should be a good start to already see the data.
refs https://github.com/TryGhost/Team/issues/2047
- We anticipate upcoming changes in the PUT /members/:id/subscriptions/:subscription_id endpoint , so covered it with a snapshot test to track the differences more precisely.
- Note, the test case contains a more explicit outgoing HTTP request mocking.
fixes https://github.com/TryGhost/Ghost/issues/14508
This change requires the frontend to send an explicit `emailType` when sending a magic link. We default to `subscribe` (`signin` for invite only sites) for now to remain compatible with the existing behaviour.
**Problem:**
When a member tries to login and that member doesn't exist, we created a new member in the past.
- This caused the creation of duplicate accounts when members were guessing the email address they used.
- This caused the creation of new accounts when using an old impersonation token, login link or email change link that was sent before member deletion.
**Fixed:**
- Trying to login with an email address that doesn't exist will throw an error now.
- Added new and separate rate limiting to login (to prevent user enumeration). This rate limiting has a higher default limit of 8. I think it needs a higher default limit (because it is rate limited on every call instead of per email address. And it should be configurable independent from administrator rate limiting. It also needs a lower lifetime value because it is never reset.
- Updated error responses in the `sendMagicLink` endpoint to use the default error encoding middleware.
- The type (`signin`, `signup`, `updateEmail` or `subscribe`) is now stored in the magic link. This is used to prevent signups with a sign in token.
**Notes:**
- Between tests, we truncate the database, but this is not enough for the rate limits to be truly reset. I had to add a method to the spam prevention service to reset all the instances between tests. Not resetting them caused random failures because every login in every test was hitting those spam prevention middlewares and somehow left a trace of that in those instances (even when the brute table is reset). Maybe those instances were doing some in memory caching.
fixes https://github.com/TryGhost/Ghost/issues/14508
This change requires the frontend to send an explicit `emailType` when sending a magic link. We default to `subscribe` (`signin` for invite only sites) for now to remain compatible with the existing behaviour.
**Problem:**
When a member tries to login and that member doesn't exist, we created a new member in the past.
- This caused the creation of duplicate accounts when members were guessing the email address they used.
- This caused the creation of new accounts when using an old impersonation token, login link or email change link that was sent before member deletion.
**Fixed:**
- Trying to login with an email address that doesn't exist will throw an error now.
- Added new and separate rate limiting to login (to prevent user enumeration). This rate limiting has a higher default limit of 8. I think it needs a higher default limit (because it is rate limited on every call instead of per email address. And it should be configurable independent from administrator rate limiting. It also needs a lower lifetime value because it is never reset.
- Updated error responses in the `sendMagicLink` endpoint to use the default error encoding middleware.
- The type (`signin`, `signup`, `updateEmail` or `subscribe`) is now stored in the magic link. This is used to prevent signups with a sign in token.
**Notes:**
- Between tests, we truncate the database, but this is not enough for the rate limits to be truly reset. I had to add a method to the spam prevention service to reset all the instances between tests. Not resetting them caused random failures because every login in every test was hitting those spam prevention middlewares and somehow left a trace of that in those instances (even when the brute table is reset). Maybe those instances were doing some in memory caching.
no issue
- All content-length snapshots should be using the same matcher for consistency - anyContentLength. It's more explicit about what the matcher is all about and might be useful to have content-length matchers in one place if it ever changes (the header value should be a damn digit after all, not a string!) (ref. https://www.rfc-editor.org/rfc/rfc7230#section-3.3.2)
refs https://github.com/TryGhost/Team/issues/2024
Without validation it was possible to send a string of comma separated
email addresses to the endpoint, and an email would be sent to each
address, bypassing any rate limiting.
This bug does not allow for an authentication bypass exploit. It is purely a
spam email concern.
Credit: Sandip Maity <maitysandip925@gmail.com>
closesTryGhost/Team#2007
- uses request context to add referrer source and medium for a new member
- uses integration name as referrer medium if exists
fixes https://github.com/TryGhost/Team/issues/2008
- New column that stores email click tracking at the time it was created
- Improved frontend side checks for when to show analytics
refs https://github.com/TryGhost/Toolbox/issues/425
refs https://github.com/TryGhost/Toolbox/issues/280
- The versioned API responses vary based on requested version (passed in request's 'accept-version' header). shared caches that sit between Ghost's origin server and the browser would be putting responses with same Vary into the same caching bucket, which is incorrect.
- This change makes response's Vary more granular and tells caching mechanisms to take 'Accept-Version' request header into account when caching.
- Informative read on the topic - https://www.fastly.com/blog/getting-most-out-vary-fastly
- renames `refSource`, `refMedium` and `refUrl` to `referrerSource`, `referrerMedium` and `referrerUrl` respectively for consistent naming across files and usages
refs https://github.com/TryGhost/Toolbox/issues/410
- Private cache control was preventing browser or shared caches from storing Content APIs response. The type of data served through the Content API is very much of a "public" nature, so should be cacheable.
- Right now the 'max-age' value of 'cache-control' header is hardcoded to '0', without 'must-revalidate' value, to allow browsers to cache content slightly more aggressively. In the future the 'max-age' value will most-likely become configurable to allow even more aggressive HTTP caching.
refs https://github.com/TryGhost/Toolbox/issues/410
- The 'private' value in 'Cache-Control' response header for all errors made it impossible for shared caches (e.g.: Fastly, Cloudflare) to cache 404 responses efficiently.
- The change substitutes 'max-age=0' which should not effect the browser cache behavior but would allow shared caches to process such requests efficiently.
- A more loose caching logic only applies to 404 responses from GET requests that are not user-specific (non-authenticated, non-cookie containing requests)
closes https://github.com/TryGhost/Team/issues/1942
- Added data fixtures for referrers
- Added new endpoint to fetch referrer stats for a given post: `/stats/referrers/posts/:id`
- Added new ReferrersStatsService, responsible for calculating referrer stats
refs TryGhost/Team#1931
- referrer source, medium and url will be stored in the events table along with rest of attribution data
- stores referrer information on two tables
- `members_created_events` for signups
- `members_subscription_created_events` for paid conversions
closes https://github.com/TryGhost/Team/issues/1933
- Added click_events to activity feed
- Added support for parsing click_events in the frontend
- Moved url parsing (transform ready) to model layer of LinkRedirect
- Moved `getEventTimeline` method to the top of the event repository
- Added description field to parsed events in the frontend (because we need a second line)
- Fixed: member email not returned in comment_event
refs d5f03ec0b1
- underlying error message varies across node versions so the content-length can't be fixed
- applied any-content-length matcher to the right test this time
no issue
- The explore endpoint needs to expose the total amount of published posts
- To be more consistent, this PR creates a PostStats class which is exposed as `stats` method within the PostService; just like it's done with the MemberService
- Moved existing method to return the date of the most recently published post into the stats service
- Updated the explore service test to reflect the new return property
no issue
- added `PostRevsion` model
- duplicated `mobiledoc_revision` creation routine in Post model's onSaving hook to create `post_revision` when model's `lexical` field has changed
- updated `mobiledoc_revision` creation to skip when `lexical` field is populated
no issue
- added `@tryghost/kg-lexical-html-renderer` dependency
- added `lexical` lib following the same pattern as our `mobiledoc` lib
- updated the Post model's `onSaving` hook to generate the `html` value from `lexical` when present
fixes https://github.com/TryGhost/Team/issues/1900
refs https://github.com/TryGhost/Team/issues/1901
- Defaults to the same value as the current email_track_opens setting for existing installations, otherwise defaults to true
- Had to use a custom migration because the `addSetting` helper doesn't support using an existing setting as current value
- Added a minimal UI to change the setting, but this still needs some design magic 🪄✨
- Link replacement is disabled if `email_track_clicks` is disabled. In the future we might consider to still do parial additions, such as source attribution and maybe redirects (to discuss).
no issue
- fixed API returning "Invalid mobiledoc structure" errors when `mobiledoc:null` is sent in the payload alongside `lexical: '{...}'`
- updated Admin's `posts` and `pages` adapters to always add `?formats=mobiledoc,lexical` because the API doesn't return `lexical` by default
- added `lexical` attribute to Admin's Post model
- updated `lexical-editor` controller and related components to work with `lexical` always being a JSON string rather than a parsed object
- updated `<KoenigLexicalEditor>` to pass through the lexical state string as initial state and wired up the `onChange` prop
no issue
- bumped `@tryghost/admin-api-schema` to allow passthrough of the `lexical` property on post and page API endpoints
- prevented saving of blank document in the `mobiledoc` field if `lexical` is provided
- prevented API input containing both `mobiledoc` and `lexical` fields to avoid issues when both are present:
- not possible to know which content is latest/has precedence
- not possible to know which editor should be displayed in Admin
refs 9471384020
- previously added tests (any subsequent matcher updates) for browse endpoint were not using matchers that sufficiently covered the dynamic portions of the body
no issue
- left `mobiledoc` as the only default format added in the post/page input serializers for now to minimize API/test churn during these early stages of lexical development
- tested that the `lexical` field is not returned by default but can be requested via `?formats=lexical`
no issue
- similar to the `mobiledoc` field, the Content API should not return the source `lexical` field if requested via `?formats=`
- renamed `removeMobiledocFormat()` to `removeSourceFormats()` to better match it's behaviour
closes https://github.com/TryGhost/Team/issues/1864
refs https://github.com/TryGhost/Team/issues/1881
- triggers free member email alert via event dispatch from member create method
- passes subscription/stripe data to member creation for paid members so free member alert can be ignored for them
- moves subscription created event being called from webhook controller to `linkSubscription`, allows creating subscription events for all new subscriptions instead of ones just via webhooks
no issue
- Bumped into these tests when doing cleanup in the notifications service. Having full snapshot of requests is useful to have as a sanity check, so migrated this test suite quickly.
closes https://github.com/TryGhost/Team/issues/1772
- The user facing side of comments recently replaced `bio` with `expertise`.
- To remain consistent we replaced all the references of `bio` with `expertise` throughout the codebase.
- This includes a database column name changing migration, within the `members` table.
- Bumped up the comments-ui version to a new minor (0.10.x) as its a breaking change.
fixes https://github.com/TryGhost/Toolbox/issues/356
- this feature allows site Administrators to view a history log of staff
actions on their site so they can audit when and by whom that something happened
- this commit promotes the History log to GA
