refs https://github.com/TryGhost/Toolbox/issues/501
- this reverts commit 48dda23554
- also includes a resolution for `@elastic/elasticsearch` so we don't
run a version that is potentially problematic - see referenced issue
for context
- this was all getting terribly behind so I've done several things:
- majority of `@tryghost/*` except Lexical packages
- gscan + knex-migrator to remove old `@tryghost/errors` usage
- bumped lockfile
refs https://github.com/TryGhost/Team/issues/2235
We found some cases which can cause a site to have member emails that have invalid characters like `member@example.com�`. This happened due to the `validator` version used by Ghost not able to catch some specific cases as invalid email, allowing members to be created with them either via Admin or Importer or direct signup. Portal UI already blocked these email as invalid. This change:
- updates `@tryghost/validator` to include a latest version of email validator that catches these invalid cases
- doesn't allow member creation with invalid email like above
- doesn't allow existing member emails to be edited to invalid
fixes https://github.com/TryGhost/Team/issues/2386
**Issue:**
- When trying to import a member that already exists, and has
'subscribed' set to 'true' in the CSV, the newsletters the member is
subscribed to are reset to the default newsletters.
- When ediging a member with the API and setting `subscribed` to true,
the same happens.
**Cause:**
A faulty check for the `status` property of a newsletter.
Fixed and added a new E2E test.
refs https://github.com/TryGhost/Team/issues/2367
We're not releasing this immediately so need to put it behind the flag
so that we continue to send emails to the correct members.
refs https://github.com/TryGhost/Team/issues/2367
This ensures that a Member is not considered subscribed to any emails, so that
counts for newsletter recipients are correct. Eventually we will filter members
on their email suppression status but this is not implemented yet.
refs https://ghost.slack.com/archives/C02G9E68C/p1670215917451249
When a member is deleted, and we receive an opened event for an email to
that member. We threw an uncaught Bookshelf EmptyResponse error.
- This change makes fetching the member not a requirement when handling
that event in the last seen at updater.
- It also adds try catches for all event listeners in the last seen at
updater
closesTryGhost/Team#2313
- Added Sent event to Post analytics and Members feed. Now post can be
Sent or Received or Bounced.
- Excluded Delivered event from Sent filter on backend.
refs https://github.com/TryGhost/Ghost/security/advisories/GHSA-9gh8-wp53-ccc6
refs https://github.com/TryGhost/Toolbox/issues/465
- Bookshelf relations allows us to edit relational records by default, which was used liberally in the codebase.
- Not having a clear track record of editable relations left the model layer prone to triggering unwanted nested saves and created a vulnerability where members were able to edit newsletter settings.
- With explicit editable relations it's easier to keep track of relations having editable access to related records. Makes the relational data modification pattern safer to use too.
- Anyone running 5.x should update to 5.24.1
Credits: Dave McDaniel and other members of [Cisco Talos](https://talosintelligence.com/vulnerability_reports)
refs https://github.com/TryGhost/Team/issues/2268
The approach of using the service to lead email suppression data as
opposed to bookshelf relations allows us to wire things up without
having implemented the database. The getBulkSuppressionData allows us to
do this without much of a DB performance hit.
closes https://github.com/TryGhost/Team/issues/2126
- Cleaned up the following GA flags: `newsletterPaywall`, `freeTrial`, `compExpiring`, `searchHelper`, `emailAlerts`, `fixNewsletterLinks`.
fixesTryGhost/Team#2266
The getCreatedEvents method was removed but was still used in the verification trigger. This commit adds the method again as a quick fix. A better fix will be commited later: https://github.com/TryGhost/Ghost/pull/15831, which includes tests that also have been ran on this change (but keeping it out of this commit to keep changes minimal).
refs https://github.com/TryGhost/Team/issues/2216
This change removes old analytics code which was added under `membersActivity` flag as an experimental alpha feature to test the first versions of member analytics, and is no longer active or in use.
This change removes the remaining services and its usage that were created to manage this version of analytics but is no longer active or maintained.
- removes `members-analytics-ingress` service that was used to ingest events from Portal in this experimental feature
- removes `member-analytics-service` service that managed the events from this experimental feature
- removes usages of the 2 services and their dependency in `members-api`
- removes `member-analytic-event` model as the corresponding table for it does not exist anymore and was dropped in 5.0
fixes https://github.com/TryGhost/Team/issues/2238
**Issue**
When viewing the 'conversions' tab on the analytics page, you could
sometimes see more listed events than the total number of conversions.
This is because other subscription events are also shown in the list.
E.g., if a new member became a paid subscriber that is attributed to a
given post, and later that subscrption has been canceled, that canceled
event would also be shown on the analytics page. This isn't really
desirable.
**Fix**
Now only 'created' subscription events are shown when the activity feed
is filtered by post_id. The other subtypes aren't related to that given
post and should be excluded.
refs https://github.com/TryGhost/Team/issues/2204
This was found during Tiers flows testing, the logic for fetching
price information from Tiers had not been updated to use the new Tiers
package and Payments service. This only affects Tiers created since 5.22.x
closes https://github.com/TryGhost/Team/issues/2195
The issue here is two-fold, and specific to using Offers so was not
caught by any automated tests. First, we were incorrectly comparing
the tier.id to the offer.tier.id - this is because the Tier objects id
property is an instance of ObjectID rather than a string.
Secondly we were passing through the cadence parameter from the
request body, but when using Offers this is not including in the
request, so we must pull the data off of the Offer object instead and
pass that to the payments service.
closes https://github.com/TryGhost/Team/issues/2184
- when using the old legacy method of `comped:true` to add complimentary subs to a member along with a label, the API call failed with `Internal Server error` and the member was added as free on the site.
- patches the options sent for fetching default product to only pick the relevant keys, as it was picking up the `withRelated` for `labels` that caused the API failure
fixes https://github.com/TryGhost/Team/issues/2175
- New event type `aggregated_click_event` that is disabled by default in all the existing activity feeds
- This returns click events, but only the first click events for each member/post combination.
- It includes the total count of unique link clicks for that member on that post combination
- Had to resort to some custom knex queries to make this work easily
- Requires `@tryghost/bookshelf-pagination@0.1.31`, included in `@tryghost/bookshelf-plugins@0.6.1` (this fixes an issue with custom selects breaking the total count query of pages)
- Went a bit overboard with the pagination tests to cover as much unknown edge cases as possible
fixes https://github.com/TryGhost/Team/issues/2129
- This changes how the activity feed API parses the filter.
- We now parse the filter early to a MongoDB filter, and split it in two. One of the filters is applied to the pageActions, and the other one is used individually for every event type. We now allow to use grouping and OR's inside the filters because of this change. As long as we don't combine filters on 'type' with other filters inside grouped filters or OR, then it is allowed.
- We make use of mongoTransformer to manually inject a mongo filter without needing to parse it from a string value again (that would make it a lot harder because we would have to convert the splitted filter back to a string and we currently don't have methods for that).
- Added sorting by id for events with the same timestamp (required for reliable pagination)
- Added id to each event (required for pagination)
- Added more tests for filters
- Added test for pagination
- Removed unsued getSubscriptions and getVolume methods
Used new mongo utility methods introduced here: https://github.com/TryGhost/NQL/pull/49
refs https://github.com/TryGhost/Team/issues/2078
This adds handling to the Tier events to create Stripe Products & Prices when
Tiers are created and have their pricing changed. We also update the Stripe
Product names when the Tier name is edited.
We also add a method to generate a payment URL which will replace our current
implementation of creating a Stripe Checkout Session
fixes https://github.com/TryGhost/Team/issues/2137
For the analytics page, we need the sent events to show up immediately
after sending an email. Otherwise we need to wait for emails to be
marked as received (which takes too long) before being able to show them
on the analytics page.
This adds the email_sent_event, which is hidden by default everywhere
and used on the analytics page.
refs 82ed10473b
refs https://github.com/TryGhost/Team/issues/1869
- getDefaultProduct has unified logic across different places (see refed commit). It is recommended to use instead of writing custom queries prone to mistakes.
- Also added more readable name to the possible error message thrown by setComplimentarySubscription
refs https://github.com/TryGhost/Team/issues/1869
- There are multiple places in the codebase fetching "default product". The code is slightly divergent in each one of them and has been a source of bugs (like the one referenced). Having the logic captured in one place will allow reducing the code duplication, making code less bug prone, and making testing the modules dependent on the "setDefaultProduct" method easier
fixes https://github.com/TryGhost/Team/issues/2091
fixes https://github.com/TryGhost/Team/issues/2089
- Added new fixtures to make testing easier for the activity feed
- Improved E2E test coverage of activity feed with separate test file
- Added data.post_id filter to enable filtering by events related to a
given post
- Fixed return types in JSDoc of test agents (TypeScript interprets
these as `typeof Agent` if we don't add `InstanceType<Agent>`)
- Added total pagination metadata to activity feed API (to allow a basic
type of pagination using filters)
refs https://jsdoc.app/tags-param.html#optional-parameters-and-default-values
- using an equals sign in the type definition is part of the Google
Closure syntax but we use the JSDoc syntax in all other places, and
tsc detects the different syntax
- this commit standardizes the syntax ahead of enforcing a certain style
down the line
refs https://github.com/TryGhost/Team/issues/2029
This will allow us to start decoupling the Stripe side of things once we've got
the core data stored. We've also add some integrity checks on the incoming
monthly_price and yearly_price to ensure they are the same currency.
refs/closes https://github.com/TryGhost/Team/issues/2004
- for imports, members are created inside a transaction, which causes the member created events to be dispatched.
- its possible that transactions for import can be rolled back if for some reason there is an error down the line while inserting other member properties. The rollback doesn't commit the member to DB, but the event dispatched earlier will still try to create the member created event which fails due to missing member id.
- knex transactions resolve the `executionPromise` both in case of explicit commit or rollback from the user, so just the transaction end check will not be good enough to make sure the member exists in DB
- adds explicit config to knex to reject transaction in case of rollback, which is then caught and event is not dispatched
closes https://github.com/TryGhost/Toolbox/issues/399
- The MemberCreatedEvent event is more accurate representation of the limit nature - counting the number of members created. The previous MemberSubscribeEvent was slightly hacky solution because a member could be subscribed/unsubscribed multiple times and distorting the limit counts.
fixes https://github.com/TryGhost/Ghost/issues/14508
This change requires the frontend to send an explicit `emailType` when sending a magic link. We default to `subscribe` (`signin` for invite only sites) for now to remain compatible with the existing behaviour.
**Problem:**
When a member tries to login and that member doesn't exist, we created a new member in the past.
- This caused the creation of duplicate accounts when members were guessing the email address they used.
- This caused the creation of new accounts when using an old impersonation token, login link or email change link that was sent before member deletion.
**Fixed:**
- Trying to login with an email address that doesn't exist will throw an error now.
- Added new and separate rate limiting to login (to prevent user enumeration). This rate limiting has a higher default limit of 8. I think it needs a higher default limit (because it is rate limited on every call instead of per email address. And it should be configurable independent from administrator rate limiting. It also needs a lower lifetime value because it is never reset.
- Updated error responses in the `sendMagicLink` endpoint to use the default error encoding middleware.
- The type (`signin`, `signup`, `updateEmail` or `subscribe`) is now stored in the magic link. This is used to prevent signups with a sign in token.
**Notes:**
- Between tests, we truncate the database, but this is not enough for the rate limits to be truly reset. I had to add a method to the spam prevention service to reset all the instances between tests. Not resetting them caused random failures because every login in every test was hitting those spam prevention middlewares and somehow left a trace of that in those instances (even when the brute table is reset). Maybe those instances were doing some in memory caching.
We're planning to change this from a warning to an error and need to
clean the codebase up before we do so.
In all of these cases the shadowing was known about and was not
causing unexpected behaviour, so the refactor consists entirely of
renaming, rather than refactoring/bug fixes.
- I want to upgrade no-shadow to an error, but to do this I need to resolve the outstanding warnings
- This is not all the warnings, just the ones that were easy to fix
closesTryGhost/Team#2007
- uses request context to add referrer source and medium for a new member
- uses integration name as referrer medium if exists
- renames `refSource`, `refMedium` and `refUrl` to `referrerSource`, `referrerMedium` and `referrerUrl` respectively for consistent naming across files and usages
closes https://github.com/TryGhost/Team/issues/1933
- Added click_events to activity feed
- Added support for parsing click_events in the frontend
- Moved url parsing (transform ready) to model layer of LinkRedirect
- Moved `getEventTimeline` method to the top of the event repository
- Added description field to parsed events in the frontend (because we need a second line)
- Fixed: member email not returned in comment_event
refs https://github.com/TryGhost/Team/issues/1899
- Added `addEmailAttributionToUrl` method to MemberAttributionService. This adds both the source attribution (`rel=newsletter`) and member attribution (`?attribution_id=123&attribution_type=post`) to a URL.
- The URLHistory can now contain a new sort of items: `{type: 'post', id: 'post-id', time: 123}`.
- Updated frontend script to read `?attribution_id=123&attribution_type=post` from the URL and add it to the URLHistory + clear it from the URL.
- Wired up some external dependencies to LinkReplacementService and added some dummy code.
- Increased test coverage of attribution service
- Moved all logic that removes the subdirectory from a URL to the UrlTranslator instead of the AttributionBuilder
- The UrlTranslator now parses a URLHistoryItem to an object that can be used to build an Attribution instance
- Excluded sites with different domain from member id and attribution tracking
closes https://github.com/TryGhost/Team/issues/1864
refs https://github.com/TryGhost/Team/issues/1881
- triggers free member email alert via event dispatch from member create method
- passes subscription/stripe data to member creation for paid members so free member alert can be ignored for them
- moves subscription created event being called from webhook controller to `linkSubscription`, allows creating subscription events for all new subscriptions instead of ones just via webhooks
refs https://github.com/TryGhost/Team/issues/1865
- refactors staff service to listen to member and subscription events
- triggers email alerts based on events instead of directly calling the service
- removes staff service dependency for members api
closes https://github.com/TryGhost/Team/issues/1772
- The user facing side of comments recently replaced `bio` with `expertise`.
- To remain consistent we replaced all the references of `bio` with `expertise` throughout the codebase.
- This includes a database column name changing migration, within the `members` table.
- Bumped up the comments-ui version to a new minor (0.10.x) as its a breaking change.
fixes https://github.com/TryGhost/Ghost/issues/15190
refs https://github.com/TryGhost/framework/pull/76
- log output always uses UTC timestamps, but it may be desirable to
configure logs to use the local machine timezone
- a new config option has been added to `@tryghost/logging` so you can
switch the logs to the local timezone
- this commit bumps the package and sets the default config option to
`false`, so it doesn't suddenly change the timezone of the logs
- docs will be updated soon but if you'd like to use the
timezone-altered timestamps, you can set `logging.useLocalTime` to
`true`
- credits to https://github.com/levee223 for the implementation and PR
fixes https://github.com/TryGhost/Team/issues/1859
**Problem:**
When for some reason a member has an active subscription (or legacy comped subscription) for product A, and a comped subscription for product B. You cannot remove comped subscription B.
**Fixed by:**
Updating the API to allow more flexible product changes on members.
- Allow the removal of (comped) products on a member, as long as that product doesn't have a related subscription
- (still) allow the addition of comped products to a member, as long as that member doesn't have other active subscriptions. This matches the existing behaviour, but now this is only checked for added products.
- Includes tests for these edge cases
fixes https://github.com/TryGhost/Team/issues/1859
**Problem:**
When for some reason a member has an active subscription (or legacy comped subscription) for product A, and a comped subscription for product B. You cannot remove comped subscription B.
**Fixed by:**
Updating the API to allow more flexible product changes on members.
- Allow the removal of (comped) products on a member, as long as that product doesn't have a related subscription
- (still) allow the addition of comped products to a member, as long as that member doesn't have other active subscriptions. This matches the existing behaviour, but now this is only checked for added products.
- Includes tests for these edge cases
no issue
- Return was missing for `res.end` if an invalid subscription_id was passed
- Added explicit `text/plain` `Content-Type` headers to error messages to avoid MIME sniffing
Signed-off-by: Elijah Conners <business@elijahpepe.com>
Co-authored-by: Simon Backx <simon@ghost.org>
refs https://github.com/TryGhost/Team/issues/1826
Geolocation was prev. loaded after member was created and updated on existing member. this was mostly due to historical context where we couldn't store data on magic link token.
Since email alerts go out at the time of member creation, this flow missed out on attaching member's location to email.
This change -
- stores request ip when a member asks for magic link in the token
- loads request ip from token when member uses magic link, and for new members loads their geolocation and stores it with member creation
refs TryGhost/Team#1826
- triggers paid subscription cancellation alert for staff users
- passes tier and subscription information for the email - loads tier info from DB for the subscription tier
refs https://github.com/TryGhost/Toolbox/issues/387
- There will three distinct verification limits soon. To keep the naming clear "configThreshold" would be too generic/confusing to use.
- Introduced jsdoc descriptions for the "source" parameter, which will be corelating with each new config parameter ("apiTriggerThreshold", "importTriggerThreshold", "adminTriggerThreshold", etc.). This should give a better visibility into parameters we are dealing in this area.
refs https://github.com/TryGhost/Team/issues/1833
refs https://github.com/TryGhost/Team/issues/1834
We've added the attribution property to subscription and signup events when the
flag is enabled. The attributions resource is fetched by creating multiple relations
on the model, rather than polymorphic as we ran into issues with that as they can't
be nullable/optional.
The parse-member-event structure has been updated to make it easier to work with,
specifically `getObject` is only used when the event is clickable, and there is now a
join property which makes it easier to join the action and the object.
refs https://github.com/TryGhost/Team/issues/1833
refs https://github.com/TryGhost/Team/issues/1834
We've added the attribution property to subscription and signup events when the
flag is enabled. The attributions resource is fetched by creating multiple relations
on the model, rather than polymorphic as we ran into issues with that as they can't
be nullable/optional.
The parse-member-event structure has been updated to make it easier to work with,
specifically `getObject` is only used when the event is clickable, and there is now a
join property which makes it easier to join the action and the object.
closes https://github.com/TryGhost/Toolbox/issues/386
- When the API request was made using staff token the source attribution was "user" instead of "api". Misattribution caused ripple effects in limit service.
- The fix also adds a new combination of data available on the `req` object - both `user` and `api_key` can be present when the request is done using a staff (user) token. Having both pieces of data on the request object gives more context for business logic, did not find a good reason to keep it "pure" with either `api_key` or `user` property.
refs https://github.com/TryGhost/Toolbox/issues/386
- Reusing tontext mapping logic to improve maintainability. It seems like the `update` method was not updated properly or intentionally was left out from 'import' source as that should not ever happen theoretically. Probably the latter is most likely.
- My reasoning on reusing same context to source mapping is: it is better to attribute an appropriate "import" source here. Who knows, maybe we'll have logic in the future where the importer updates instead of skipping existing members. It would not make sense to attribute the source to 'member' in that case, amirite?
- This refactor also makes maintainability of this code way easier
refs https://github.com/TryGhost/Team/issues/1728
- previously, we allowed a member to be mapped to multiple tiers simultaneously as an edge case, in case they managed to signup via another subscription
- since this was always an edge case and not supported, to simplify the flows going forward now that complimentary members can also upgrade, in case of an active subscription we'll always just attach the associated tier to member and remove all other tiers mapped to it
refs https://github.com/TryGhost/Team/issues/1808
refs https://github.com/TryGhost/Team/issues/1809
refs https://github.com/TryGhost/Team/issues/1820
refs https://github.com/TryGhost/Team/issues/1814
### Changes in `member-events` package
- Added MemberCreatedEvent (event, not model)
- Added SubscriptionCreatedEvent (event, not model)
### Added `member-attribution` package (new)
- Added the AttributionBuilder class which is able to convert a url history to an attribution object (exposed as getAttribution on the service itself, which handles the dependencies)
```
[{
"path": "/",
"time": 123
}]
```
to
```
{
"url": "/",
"id": null,
"type": "url"
}
```
- event handler listens for MemberCreatedEvent and SubscriptionCreatedEvent and creates the corresponding models in the database.
### Changes in `members-api` package
- Added urlHistory to `sendMagicLink` endpoint body + convert the urlHistory to an attribution object that is stored in the tokenData of the magic link (sent by Portal in this PR: https://github.com/TryGhost/Portal/pull/256).
- Added urlHistory to `createCheckoutSession` endpoint + convert the urlHistory to attribution keys that are saved in the Stripe Session metadata (sent by Portal in this PR: https://github.com/TryGhost/Portal/pull/256).
- Added attribution data property to member repository's create method (when a member is created)
- Dispatch MemberCreatedEvent with attribution
### Changes in `members-stripe-service` package (`ghost/stripe`)
- Dispatch SubscriptionCreatedEvent in WebhookController on subscription checkout (with attribution from session metadata)
- cleaned up unused dependencies
- adds missing dependencies that are used in the code
- this should help us be more explicit about the dependencies a package
uses
- because of how the npm scripts were set up, we were running the full
Admin integration tests during the unit tests phase of CI
- this commit renames the majority of `test` to `test:unit` in the
package.json files, and aliases `test` to `test:unit`
- special packages like Admin have no-op'd `test:unit` scripts so we
don't end up running its tests
refs https://github.com/TryGhost/Team/issues/1726
Free trial offers don't have a Stripe coupon created for them, as the trial is directly added to checkout session. So for mapping a subscription to offer, we pass the offer id directly from checkout metadata to link the subscription in backend with right offer data. This also handles the case where the offer id against a subscription can get overwritten for a subsequent subscription event, as the sub event from Stripe doesn't has the trial offer info.
- handles storing an offer id for a subscription
- updates member detail in Admin to show the offer info for a subscription
refs https://github.com/TryGhost/Team/issues/1726
- free trial offers don't need a stripe coupon created for them
- checkout sessions for free trial offers ignore stripe coupon and directly pass the trial days value
- trial days of an offer take precedence over trial days added as default to a tier
