Commit Graph

3211 Commits

Author SHA1 Message Date
Ghost CI
3b6d3b1a02 v5.84.0 2024-06-06 08:34:57 +00:00
Ghost CI
5c29bb6bb9 🎨 Updated Source to v1.2.3 2024-06-06 08:34:57 +00:00
Kevin Ansfield
a7a1166310
Added beta feature toggle for moving excerpt field into editor (#20341)
closes https://linear.app/tryghost/issue/MOM-179

Full details coming soon to https://ghost.org/changelog

- Adds a toggle to labs setting that moves the excerpt input below the post title in the editor
2024-06-05 20:14:34 +01:00
Kevin Ansfield
4a94b5efc9 Added newsletter design setting to display excerpt as subtitle
no issue

Full details coming soon to https://ghost.org/changelog/

- when enabled in newsletter design settings a post's custom excerpt will be displayed as a subtitle in the email
2024-06-05 19:46:50 +01:00
Steve Larson
e49021b7ea
🐛 Fixed default sort for the content API posts endpoint with included relations (#20333)
ref https://linear.app/tryghost/issue/CFR-31/
- when relations were included, erroneous logic resulted in the model's
default sort being applied
- the model default sort is not intended for the content API and
needlessly slowed down responses
- there's a change for users here that should be incredibly unlikely to
be hit; default sort is `published_at desc` which will be secondarily
sorted by `id desc` instead of `published_at desc, updated_at desc, id
desc`

This is a very significant performance improvement for content API
requests with includes for sites with a significant amount of data,
which will primarily impact those using Ghost as a CMS or theme {{#get}}
helpers.
2024-06-05 13:26:10 -05:00
Kevin Ansfield
9ca1f3ce24
Renamed subtitle to excerpt (#20334)
no issue

We've settled on using "excerpt" naming in place of "subtitle" to better reflect the underlying property name and tie in with themes and historical usage.

- added migration to rename the `show_subtitle` newsletter setting to `show_excerpt`
- renamed all places in the codebase that referenced subtitle
2024-06-05 17:59:30 +01:00
Michael Barrett
0f283da8eb
🐛 Fixed Slack integration using member content in excerpt (#20328)
- refs
[ONC-63](https://linear.app/tryghost/issue/ONC-63/discordslack-webhook-integration)
- fixes [#20304](https://github.com/TryGhost/Ghost/issues/20304)

When a post is published and sent to Slack via webhook, the excerpt
generated could contain member content. This change ensures that the
excerpt does not contain member content. This change also ensures that
the author for the post is correctly shown in Slack
2024-06-05 17:46:21 +01:00
Kevin Ansfield
b447a26832
Added custom excerpt to post revisions (#20323)
closes https://linear.app/tryghost/issue/MOM-170

When the subtitle field is included in the editor it creates a disconnect with post revisions if the underlying custom excerpt data is not included so we'd like to both preview and restore the subtitle when the in-editor subtitle field is enabled.

- added `post_revisions.custom_excerpt` column to schema
- added migration to add `post_revisions.custom_excerpt` to existing databases
- added migration to populate `post_revisions.custom_excerpt` with the current `post.custom_excerpt` value from the associated record
  - ensures no data is inadvertently lost when restoring an old version
- using current data matches what would have happened previously where custom_excerpt was never overwritten when restoring an old version
- updated post revisions handling to accept the `custom_excerpt` field
- updated Admin's revision preview and restoration to display and set the `custom_excerpt` field
2024-06-05 14:47:33 +01:00
renovate[bot]
d40ef32ca8
🐛 Fixed mailto: not being recognised in internal-linking search (#20331)
closes https://linear.app/tryghost/issue/MOM-195

- bumped Koenig packages with updated URL-matching
2024-06-05 14:30:24 +01:00
Kevin Ansfield
952a89f673
Renamed show_subhead to show_subtitle (#20320)
closes https://linear.app/tryghost/issue/MOM-194

- whilst working on the feature our naming changed from "Subhead" to "Subtitle"
- this rename of the newsletter design setting column brings naming back into a consistent state before public release
2024-06-04 10:57:14 +00:00
Sag
d751d648c7
Fixed offer not found case during Stripe checkout (#20322)
fixes https://linear.app/tryghost/issue/SLO-135

- handles edge cases when an invalid `offerId` is provided during Stripe
checkout
2024-06-04 10:27:45 +00:00
Ghost CI
46cb819feb v5.83.0 2024-05-31 16:04:36 +00:00
renovate[bot]
418316959c Update dependency mysql2 to v3.10.0 2024-05-30 19:28:02 +00:00
Steve Larson
65b929d1d8
Added JSDoc to stripe-mocker (#20282)
no ref
2024-05-29 17:38:24 -05:00
renovate[bot]
d507c5c6c2 Update Koenig packages 2024-05-29 22:28:20 +01:00
Kevin Ansfield
55015ccbcc
Added '@' internal link shortcut to editor (#20272)
closes https://linear.app/tryghost/issue/MOM-83

- added additional labs flag to allow internal testing prior to private beta release
- bumped Koenig packages containing support for @-link feature
2024-05-29 18:13:52 +01:00
Sanne de Vries
acac7801f4
Split subhead feature into two feature flags (#20279)
REF MOM-119
- Split subhead feature flag into two: editorSubtitle and
newsletterSubtitle
- Updated UI copy, feature flag names and class names from subhead to
subtitle
2024-05-29 17:59:25 +02:00
renovate[bot]
4ef1dd2146 Update dependency mysql2 to v3.9.9 2024-05-29 15:14:00 +00:00
Ronald Langeveld
fddcf3ffee
Added Subheads behind a flag (#20265)
refs MOM-152 MOM-148 MOM-151

- Added Subheads behind a flag + toggle in settings.
- Removes Excerpt fields from post settings if flag is enabled.
- Added subhead toggle in newsletter settings.
- Loads of styling

---------

Co-authored-by: Sanne de Vries <sannedv@protonmail.com>
2024-05-29 09:53:40 +00:00
Steve Larson
e6fcbf45a1
Added sanitization for svg uploads (#20264)
ref https://linear.app/tryghost/issue/ENG-856
- svgs were not previously sanitized and could contain scripts
2024-05-28 08:58:16 -05:00
Ronald Langeveld
d799f2ecb1
Added show_subhead column to newsletters table (#20268)
refs MOM-152

Adds `show_subhead` column to newsletter table. False by default.
2024-05-28 16:29:20 +07:00
Fabien 'egg' O'Carroll
6a8ae57a24
Used a base64 encoded string for hmac secret (#20269)
We want to use a randomly generated 64 byte secret for the hmac, and
utf8 encoding isn't nice to work with for this, so we're going to use a
base64 string and decode it into a buffer for the secret.
2024-05-28 14:12:48 +07:00
renovate[bot]
3ebe206ea7 Update TryGhost packages 2024-05-27 16:58:32 +02:00
renovate[bot]
6ff08c5c14 Update dependency mysql2 to v3.9.8 2024-05-27 08:04:34 +00:00
Daniel Lockyer
ecc6053e5d Cleaned up jsdom dependency usage
- we don't need this in `ghost/core` as it's not used in there
- we need to declare this dependency for the apps, as they use it for
  running tests
- this doesn't change the lockfile but it means we're declaring the
  dependency in the right places now
2024-05-27 09:49:43 +02:00
renovate[bot]
3e3bcb81fd Update dependency jsdom to v24 2024-05-27 09:07:18 +02:00
Ghost CI
43e25f44dc v5.82.12 2024-05-27 06:39:20 +00:00
Chris Raible
98d49f57d5
Added experimental headers to allow caching members content (#20200)
ref
https://linear.app/tryghost/issue/KTLO-45/deploy-members-caching-solution-to-a-single-site-to-validate-and-test

Currently we only cache publicly available content. Any content that is
accessed by a logged in member is only cached for that specific member
based on their cookie. As a result, almost all requests from logged in
members bypass our caching layer and reach Ghost, which adds unnecessary
load to Ghost and its database.

This change adds experimental headers that allow our CDN to understand
which tier to cache the content against, and securely tell the CDN which
tier a logged in member has access to. With these changes, we can cache
the member content against the tier, rather than the individual member,
which should result in a higher cache HIT ratio and reduce the load on
Ghost.

For requests to the frontend of the site, Ghost will set a custom
`X-Member-Cache-Tier` header to the ID of the tier of the member who is
accessing the content. This tells the CDN which tier to cache the
content against.

For requests to either `/members/?token=...` endpoint (the magic link
endpoint) or `/members/api/member`, Ghost will set a `ghost-access` and
`ghost-access-hmac` cookie with the ID of the tier of the logged in
member. With these two pieces of information, our CDN can serve cached
content to logged in members.

These headers are experimental, and can only be enabled via Ghost's
config. To enable these headers, set `cacheMembersContent:enabled` to
`true` and provide an HMAC key in `cacheMembersContent:hmacSecret`.
2024-05-23 19:06:45 -07:00
renovate[bot]
d302d4049c Update Koenig packages 2024-05-23 19:13:38 +01:00
Daniel Lockyer
4a02938774 Reverted "Enabled includeLocalVariables option in Sentry"
refs https://app.incident.io/ghost/incidents/73
refs https://blog.sentry.io/local-variables-for-nodejs-in-sentry/

- this reverts commit cc76fda3e8
- it turns out that enabling this causes the entire program to pause
  whilst it collects local variables
- this was only added to aid with debugging, so it's not critical to
  have and can be disabled
2024-05-23 12:04:10 +02:00
renovate[bot]
ad48d8eb25 Update sentry-javascript monorepo to v7.116.0 2024-05-21 14:24:23 +01:00
Steve Larson
9d9a421b54
Added a column disallow list in the content API posts serializer (#20207)
ref https://linear.app/tryghost/issue/CFR-29
- Removed the mobiledoc and lexical columns from the posts input
serializer, meaning they will no longer be queried for.

Get helpers are essentially a gateway to the Content API. We already
strip out the mobiledoc and lexical fields in the output
serializer/returned response, but this means we're passing the mobiledoc
and lexical fields back from the db. This is pointless and these fields
are substantial in size - by far the largest fields in the whole ghost
db - leading to slowed performance.

I've updated the posts input serializer to strip out the lexical and mobiledoc
columns so we stop doing a `select *` with every query.
2024-05-20 08:25:20 -05:00
Ronald Langeveld
e5056d8d9d
🐛 Fixed External Image URLs being incorrectly prefixed (#20226)
ref ENG-824

- the bug is causing resize prefixes being added to images served from
outside of Ghost.
- this now would only append the prefex to images served by Ghost and
other images urls' would get served as is.
- we can determine that by checking whether imageName doesn't exist,
meaning the source is a third party.
- this mostly affect edge case users, eg where a feature image url was
passed in via the API and doesn't get served by Ghost.
2024-05-20 18:06:03 +08:00
Ronald Langeveld
9a60254cd2
🐛 Fixed redundant member data loading for static assets (#20031)
refs CFR-21

Reorganised middleware execution so that member data is not redundantly loaded for static assets or the sitemap.

---------

Co-authored-by: Michael Barrett <mike@ghost.org>
2024-05-20 13:55:45 +08:00
Ghost CI
589793f0c5 v5.82.11 2024-05-17 16:04:53 +00:00
renovate[bot]
239b5b6369
🐛 Fixed certain snippets not inserting correctly (#20129)
ref https://linear.app/tryghost/issue/ONC-26/support-escalation-template-bug
ref https://github.com/TryGhost/Ghost/issues/18948

- bumps `@tryghost/koenig-lexical` to version that doesn't crash during snippet insertion when last node doesn't match expectations for inserting a trailing paragraph
- bumps all Koenig packages to ensure internal dependencies are pinned to prevent potential issues from mixed versions caused by yarn resolution
2024-05-16 11:39:03 +00:00
Ghost CI
b7efe4ff8e Merged v5.82.10 into main 2024-05-16 02:19:02 +00:00
Ghost CI
f6f04792ea v5.82.10 2024-05-16 02:19:00 +00:00
Galdin Raphael
93eaee7e08
Added missing <html> opening tag to the maintenance page (#20203)
- this was previously missing
2024-05-15 09:10:37 +02:00
Laurent Goderre
d67fc8c353
Added jackspeak resolution to core to fix ESM + CJS compat issue (#20199)
refs #20197

- adds a jackspeak resolution to Ghost core so we can try and ensure the compatible version of jackspeak/string-width is used when the lockfile is regenerated
2024-05-15 08:47:04 +02:00
Fabien O'Carroll
e98f505ae3 Added body parsing to the frontend
ref https://linear.app/tryghost/issue/MOM-73

We need to add body parsing middleware here, so that NestJS has access to it.
We also attach the rawBody which is used to validate the HTTP Signatures
2024-05-15 12:41:34 +07:00
Sodbileg Gansukh
60da243d0c
Updated signup and private page forms for consistency (#20208)
ref TRI-92
2024-05-15 04:47:02 +00:00
Kevin Ansfield
d2da9d3c17
Updated metadata shown on hover in internal link lists (#20204)
closes https://linear.app/tryghost/issue/MOM-80

- updated internal linking search results items
  - removed visibility text from meta data
  - added additional icon for paid/specific tier visibility
  - added titles to icons
- bumped `@tryghost/koenig-lexical` to include support for meta icon titles
- bumped other Koenig packages due to sub-dependency updates
2024-05-14 16:07:04 +00:00
Steve Larson
842290cbef
Improved performance for filter strings with multiple neq statements (#20198)
ref https://linear.app/tryghost/issue/CFR-27
- updated packages to include performance improvement for NQL filter
strings including multiple neq filters for the same resource
- bumped `bookshelf-plugins`
- bumped NQL versions

We identified a performance fix that allows us to combine not equal
(neq) filters for the same resource in a logically-equivalent way that
also has far more performant resulting SQL.

We're effectively automatically combining strings like
'tag:-tag1+tag:-tag2` into 'tag:-[tag1,tag2]'.
2024-05-13 10:35:27 -05:00
Daniel Lockyer
e3fa095c80 Fixed misc JSDoc types
- nothing crazy, but cleans up a few editor warnings
2024-05-13 14:53:53 +02:00
Ghost CI
b1c5a2985c v5.82.9 2024-05-13 07:01:15 +00:00
renovate[bot]
fc41947738 Update dependency semver to v7.6.2 2024-05-10 08:30:32 +02:00
renovate[bot]
512c3e1413 Update CSS preprocessors 2024-05-09 10:36:59 +02:00
renovate[bot]
4c1ecabfc7 Update sentry-javascript monorepo to v7.114.0 2024-05-09 10:16:37 +02:00
Ghost CI
88ad62558a Merged v5.82.8 into main 2024-05-08 19:30:39 +00:00
Ghost CI
f303432ba9 v5.82.8 2024-05-08 19:30:37 +00:00
Daniel Lockyer
7e9d82655e Added extra validation for reset_password endpoint
fix https://linear.app/tryghost/issue/SLO-104/cannot-read-properties-of-undefined-reading-0-an-unexpected-error

- if the request body didn't contain the correct keys, it'd just HTTP
  500 out of there
- this adds some optional chaining so we end up with undefined if
  anything isn't as expected, and the following if-statement does the
  rest of the check for us
- this also adds a breaking test (the first E2E test for authentication, yay!)
2024-05-08 18:05:04 +02:00
Daniel Lockyer
76c6e92006
Handled invalid files when uploading DB zips (#20165)
fix
https://linear.app/tryghost/issue/SLO-103/invalid-comment-length-expected-7-found-0-an-unexpected-error-occurred

- similar to
e8e3447f15,
this captures a specific error from yauzl and throws a user-friendly
error
- perhaps in the future we can just look for yauzl errors and always
return user-friendly errors, but let's monitor that first
- also includes a breaking test
2024-05-08 14:59:34 +02:00
Daniel Lockyer
f276abf9e8 Condensed logic for determining whether to send Sentry events
refs https://docs.sentry.io/platforms/javascript/configuration/filtering/#using--1%20

- this simplifies our logic to determine whether we should send events
  by moving the code to `beforeSend`
- `errorHandler` is going away in Sentry v8 so this results in a shorter
  diff in the future
- the logic should be the same, always send non-Ghost errors, and only
  send HTTP 500 Ghost errors
2024-05-08 14:25:00 +02:00
Daniel Lockyer
77fc66340a Refactored Sentry beforeSend code
- this tidies up the implementation by removing the very verbose ternary
  operator, and also uses the optional chaining operator
2024-05-08 14:25:00 +02:00
Daniel Lockyer
c4ad593a5a Added JSDoc types to Sentry
- this helps editors with identifying what the types of parameters are
2024-05-08 14:25:00 +02:00
Daniel Lockyer
e8e3447f15 Fixed handling empty zip file uploads
fix https://linear.app/tryghost/issue/SLO-102/end-of-central-directory-record-signature-not-found-an-unexpected

- previously, uploading an empty zip would result in a HTTP 500 error
  because yauzl would error and we'd bubble that up as an
  InternalServerError
- now, we catch the specific error message and return a more user
  friendly error
- also includes tests and sample zip file
2024-05-08 11:19:47 +02:00
Daniel Lockyer
d82b136a6a Handled uploads with invalid form bodies
fix https://linear.app/tryghost/issue/SLO-101/http-500-with-invalid-multipart-data

- previously, busboy would error out if we supplied a body that was
  invalid (such as an empty FormData)
- we would then return a HTTP 500 to the user, which causes all manner
  of problems
- now we catch errors from busboy and return a nice BadRequestError
2024-05-08 11:19:47 +02:00
Daniel Lockyer
ae88dc8548 Handled invalid timestamp format in filters
fix https://linear.app/tryghost/issue/SLO-85/fix-http-500-on-contentposts

- in the event we give the incorrect format in a filter, MySQL will
  throw an error and we'll throw a HTTP 500 error
- we can capture this error and return a more useful error to the user
- ideally we'd do this in a validation step before attempting the query,
  but parsing this out of NQL and detecting which columns are DATETIME
  could be quite tricky
2024-05-08 09:28:56 +02:00
Daniel Lockyer
82c612bad9 Rolled out API framework JSDoc typing to more places
- this updates a bunch of places where we're just using Object to cheat
  the system
- doing this means editor autocomplete and basic type checking is better
  because we now have proper types in place
- functionality should not change, these are just comments
2024-05-08 09:28:31 +02:00
renovate[bot]
1fea2fc616 Update dependency semver to v7.6.1 2024-05-08 09:04:44 +02:00
Ghost CI
7201006f05 Merged v5.82.7 into main 2024-05-07 18:13:36 +00:00
Ghost CI
000616ac02 v5.82.7 2024-05-07 18:13:34 +00:00
Daniel Lockyer
2659e5aa40 Added handling for parsing errors with user-submitted HTML
fix https://linear.app/tryghost/issue/SLO-87/cannot-read-properties-of-undefined-reading-createimpl-an-unexpected
refs https://github.com/jsdom/jsdom/issues/3709

- in the event we are given some HTML to parse, and that fails, we
  currently return a HTTP 500 because it's unhandled
- the instance we saw was due to `<constructor>` crashing jsdom, we've
  opened an issue for that
- in terms of handling the error gracefully, we can surround the code
  in a try-catch and return a more suitable error. I've gone for a
  ValidationError for now - you could debate whether a different one is
  more appropriate
- also added Sentry error capturing so we're not blind to these,
  ultimately we should make sure the parser can handle all
  user-submitted data
2024-05-07 17:25:48 +02:00
Daniel Lockyer
dccb4ac84e Cleaned up unused controller method
- this isn't being used anywhere, so we can clean it up
2024-05-07 11:44:07 +02:00
Daniel Lockyer
1fd155d56a Fixed extra arguments being supplied to function calls
- identified by tsc, this shouldn't change any functionality because
  there were extra arguments being supplied and were unused
2024-05-07 11:44:07 +02:00
Daniel Lockyer
29cc3003c7 Fixed misc JSDoc types
- nothing crazy, just fixing minor typing issues that I've come across
2024-05-07 11:44:07 +02:00
Ronald Langeveld
4f3bfebfea
Revert "Added latest_event_timestamp to email table (#20118)" (#20149)
This reverts commit 3246a8d2c9.
2024-05-07 08:58:42 +00:00
Daniel Lockyer
c298db912c Added JSDoc types for API controllers
- this adds a simple set of types to the @tryghost/api-framework
  package that should describe all of the keys available on a
  controller, and then rolls it out to all API controllers
- unfortunately, due to https://github.com/microsoft/TypeScript/issues/47107, we have
  to split apart `module.exports` into a variable assignment in order for type-checking
  to be done
- the main benefit of this is that `frame` is now typed, and editors understand what keys
  are available, so intellisense works properly
2024-05-07 10:49:44 +02:00
Ronald Langeveld
3246a8d2c9
Added latest_event_timestamp to email table (#20118)
ref ENG-832

- Added migrations for `latest_event_timestamp` column in emails table.
- updated schema
- updated emails model
2024-05-07 16:12:51 +08:00
Daniel Lockyer
de435238f4 Fixed wrong status code type
- `statusCode` should be a number, but we were passing a string
- this doesn't really affect anything, but tsserver was flagging it up
  as the wrong type
2024-05-07 08:29:48 +02:00
Daniel Lockyer
b29c897da3 Fixed passing error to IncorrectUsageError
- we should pass it as `err` and not `error`
- this probably slipped in because the catch parameter is called
  `error`, so I've updated that and fixed the references
2024-05-07 08:29:48 +02:00
Daniel Lockyer
efc59dd315 Fixed extra parameter to function
- `serializeTier` only takes 2 parameters
- flagged by tsserver
2024-05-07 08:29:48 +02:00
Daniel Lockyer
f72d7b77ac Fixed miscellaneous jsdoc comments
- this helps tsserver figure out what the type of things is around our
  codebase
- nothing crazy, mostly Express types for the middleware, application and router levels
2024-05-07 08:29:48 +02:00
Daniel Lockyer
265a8dd16f Added function names to more middleware
refs 319f251ad2

- this helps debugging because all middleware in the stack will have a
  function name, so it'll show up instead of `<anonymous>`
2024-05-06 17:51:39 +02:00
Daniel Lockyer
a50bb8d79e Added missing error messages for members uploader
fix https://linear.app/tryghost/issue/SLO-97/missing-messages-for-members-file-upload

- these were missing, so if the members importer wasn't given a file, it
  would crash with an HTTP 500 error
- also added a test to ensure we get a 422 back
2024-05-06 15:17:25 +02:00
Daniel Lockyer
5a8145139a Fixed handling cutoff boundary data in image + media upload
fix https://linear.app/tryghost/issue/SLO-95/unexpected-end-of-multipart-data-for-broken-image-upload-request

- in the event the client sends an invalid body to the image or media
  upload endpoints, Dicer will throw an error if the boundary data is
  malformed
- previously, we've just been bubbling that up as an InternalServerError
  and that results in an HTTP 500
- we can capture errors produced by dicer and return a handled
  BadRequestError, as it's the client's fault
- also includes breaking tests
2024-05-06 13:41:25 +02:00
Daniel Lockyer
3e79712466 Fixed handling malformed image + media upload requests
fix https://linear.app/tryghost/issue/SLO-94/unexpected-field-when-given-broken-image-upload-request

- in the event the body of an image or media upload request is malformed
  (broken metadata / blob or something), we get a MulterError and this
  bubbles up as an InternalServerError and spits out a HTTP 500
- we can capture this and return a BadRequestError, as it's the client's
  fault for not providing the correct body
- this implements that and adds breaking tests
2024-05-06 13:24:26 +02:00
Ghost CI
f34c33f330 v5.82.6 2024-05-03 16:04:14 +00:00
Chris Raible
d8b629c713
Added an optional timeout parameter to AdapterCacheRedis (#20131)
ref
https://linear.app/tryghost/issue/ENG-902/add-an-optional-timeout-in-the-redis-cache-adapter-in-case-redis

- Added an optional timeout parameter to AdapterCacheRedis, so that the
`get(key)` method will return `null` after the timeout if it hasn't
received a response from Redis
- When load testing the `LinkRedirectRepository` with the Redis cache
enabled on staging, we noticed that for some reason Redis stopped
responding to commands for ~30 seconds.
- The `LinkRedirectRepository` was waiting for the Redis cache to
respond and resulted in a drastic increase in response times for link
redirects
- This change will allow us to set a timeout on the `get(key)` method,
so that if Redis doesn't respond within the timeout, the method will
return `null` as if it were a cache miss.
- Then the `LinkRedirectRepository` will fall back to the database and
return the link redirect from the database instead of waiting
indefinitely for Redis to respond
2024-05-02 20:39:23 -07:00
Chris Raible
b9f7ea65e9
Revert "Added new member signup flow behind labs flag (#19986)" (#20130)
ref https://linear.app/tryghost/issue/KTLO-1/members-spam-signups

This reverts commit 01d0b2b304.

- Removed the new member signup flow because it didn't solve the
problems with spam signups
2024-05-02 13:02:32 -07:00
Daniel Lockyer
7950122ffe Protected against deleting non-existent image during upload
fix https://linear.app/tryghost/issue/SLO-93/undefined-path-error-with-bad-image-upload

- in the event we receive a request to upload an image, that doesn't
  contain an image, we still try and unlink the files
- this is a dangling promise, so it doesn't cause an explicit HTTP
  error, but it does show up as a console error
- fixed it by checking for the path, and early returning if it doesn't
  exist
- also added a test that would fail without this
2024-05-02 17:29:03 +02:00
Sag
e996213122
Moved POST /members/api/member behind alpha flag (#20124)
ref https://linear.app/tryghost/issue/SLO-78

- the `POST /members/api/member` endpoint is solely used by the alpha
feature `membersSpamPrevention` and should not be available otherwise
2024-05-02 16:34:32 +02:00
Daniel Lockyer
cc76fda3e8 Enabled includeLocalVariables option in Sentry
ref https://linear.app/tryghost/issue/SLO-92/enable-extra-sentry-integrations
refs https://docs.sentry.io/platforms/javascript/guides/node/configuration/options/#include-local-variables

- this will capture the local stack variables when producing a stack
  trace, which would be super useful when debugging
2024-05-02 13:52:40 +02:00
Daniel Lockyer
5b28dc9246 Enabled Sentry's ExtraErrorData integration
refs https://docs.sentry.io/platforms/javascript/guides/node/configuration/integrations/extraerrordata/
ref https://linear.app/tryghost/issue/SLO-92/enable-extra-sentry-integrations

- this enables the ExtraErrorData integration, which should help us
  capture more of the properties of the errors we're producing, which
  _may_ help with debugging
2024-05-02 13:52:40 +02:00
renovate[bot]
a33dccf8cd Update TryGhost packages 2024-05-01 17:01:41 +02:00
Daniel Lockyer
3f7a7fff44 Fixed HTTP 500 when adding unknown products to member
fix https://linear.app/tryghost/issue/SLO-89/cannot-read-properties-of-null-reading-get-an-unexpected-error

- if we pass an invalid ID when updating the products on a member, we
  throw a HTTP 500 error because `product` is `null`
- we can check for this and return a BadRequestError, because the user
  supplied an incorrect ID
2024-05-01 16:54:35 +02:00
Ghost CI
6b32548e5c Merged v5.82.5 into main 2024-05-01 13:22:28 +00:00
Ghost CI
a4c478958b v5.82.5 2024-05-01 13:22:26 +00:00
renovate[bot]
746df63307
Update dependency gscan to v4.43.1 (#20109)
ref https://linear.app/tryghost/issue/ENG-842/gluster-file-name-length-limit

- gscan v4.43.1 contains a bug fix to throw an large filenames
2024-05-01 11:26:21 +00:00
Daniel Lockyer
ddac3a9e8b Removed Sentry error capturing for failed URL decoding
fix https://linear.app/tryghost/issue/SLO-79/incorrectusageerror-the-url-httpsblogkongregatecompercentc0-couldnt-be

- we added this Sentry captureException whilst fixing a bug where
  decodeUrl could fail, and throw a 500 exception
- we added handling for that case and returned an empty string, but we
  also added Sentry error capturing
- at this point, I don't think we need to be capturing errors in Sentry,
  because the issue is already handled, and it only usually happens with
  malicious/incorrect URLs
- this is our #2 cause of Sentry alerts, so it's good to clean it up
2024-05-01 10:15:13 +02:00
renovate[bot]
60a3b5a913 Update TryGhost packages 2024-05-01 08:49:04 +02:00
Daniel Lockyer
7109743282 Fixed overly verbose DatabaseInfo require
- we don't need to deep require into the library as it exports what we
  need on the surface
- this should unblock https://github.com/TryGhost/Ghost/pull/19002, as
  it's randomly failing with this require
2024-04-30 21:39:56 +02:00
Michael Barrett
4cd85ab8b7
Added timeout when resizing an image (#20087)
refs
[ENG-827](https://linear.app/tryghost/issue/ENG-827/🐛-crash-on-resizing-animated-gif)

Added a timeout to the image resizing middleware to prevent crashes when
an image is taking too long to resize. When the timeout is reached and
the image has not been resized, the middleware will return the original
image
2024-04-30 08:39:30 +01:00
Ronald Langeveld
b2970cb4e0
Added integrity test for flags (#20094)
ref
https://ghost.slack.com/archives/C02G9E68C/p1714047709694639?thread_ts=1713956576.497899&cid=C02G9E68C
    
    - Ensures unique feature flags, avoiding configuration conflicts.
    - Enhances code reliability and simplifies feature tracking.
    - Prevents bad rebases was the reason for the initial duplication.
2024-04-29 02:39:15 +00:00
renovate[bot]
305029bc38 Update dependency ws to v8.17.0 2024-04-29 01:23:51 +00:00
renovate[bot]
f18f08928f Update dependency yjs to v13.6.15 2024-04-29 01:22:59 +00:00
Ghost CI
cf9ba60524 v5.82.4 2024-04-26 16:04:30 +00:00
Chris Raible
dcd65bfa4f
Added caching to the LinkRedirectRepository (#20036)
ref
https://linear.app/tryghost/issue/ENG-851/implement-a-minimal-but-complete-version-of-redirect-caching-to
ref https://app.incident.io/ghost/incidents/55

Often immediately after sending an email, sites receive a large volume
of requests to LinkRedirect endpoints from members clicking on the links in
the email.

We currently don't cache any of these requests in our CDN, because we
also record click events, update the member's `last_seen_at` timestamp,
and send webhooks in response to these clicks, so Ghost needs to handle
each of these requests itself. This means that each of these LinkRedirect requests
hits Ghost, and currently all these requests hit the database to lookup
where to redirect the member to.

Each one of these requests can make up to 11 database queries, which can
quickly exhaust Ghost's database connection pool. Even though the
LinkRedirect lookup query is fairly cheap and quick, these queries aren't
prioritized over the "record" queries Ghost needs to handle, so they can
get stuck behind other queries in the queue and eventually timeout.

The result is that members are unable to actually reach the destination
of the link they clicked on, instead receiving a 500 error in Ghost, or
it can take a long time (60s+) for the redirect to happen.

This PR uses our existing `adapterManager` to cache the redirect lookups
either in-memory or in Redis (if configured — by default there is no caching). This only removes 1 out of
11 queries per redirect request, so it won't reduce the load on the DB
drastically, but it at least decouples the serving of the LinkRedirect from
the DB so the member can be redirected even if the DB is under heavy
load.

Local load testing results have shown a decrease in response times from
60 seconds to ~50ms for the redirect requests when handling 500 requests
per second, and reduced the 500 error rate to 0.
2024-04-25 19:17:25 -07:00
Daniel Lockyer
892b9ab397 Moved internalLinking and stripeAutomaticTax flags to private beta
refs https://ghost.slack.com/archives/C02G9E68C/p1714051665654659?thread_ts=1713970812.191919&cid=C02G9E68C

- this enables us to enable the flags on sites without the need to
  enable developer experiments
- added `(private beta)` to the end of the relevant UI flags, because
  they're still in alpha in the UI despite being in beta in backend
2024-04-25 16:37:55 +02:00
Ronald Langeveld
601d05de0d
Removed duplicated ActivityPub labs key (#20089)
no issue

- rebased from main without seeing there's an existing flag.
- this removes the duplicated flag
2024-04-25 14:15:47 +00:00
Steve Larson
a0b7476794
Updated staff deletion logic (#20069)
ref https://linear.app/tryghost/issue/ENG-826

- Changed staff deletion logic to do a bulk insert when adding a tag to
the users' associated posts

Staff deletion logic has really poor performance at scale because we do
individual updates for every post. If a user has dozens+ posts
(especially in a large db with thousands of posts), this can take >60s
and look like a timeout. Ultimately this should probably be a jobbed off
process, but for the time being we can improve this by doing a bulk
insert.

Note that this update uses the pattern for the bulk tagging of posts
from the right click (bulk) actions in the posts lists in Admin. With
bulk actions, **we do not trigger web hooks or the post.edited events**.
We will document this and follow up on this separately.
2024-04-25 08:19:11 -05:00
Djordje Vlaisavljevic
7a3bbfde10
Added ActivityPub playground (#20081)
ref MOM61

- Adds admin-x react app we’ll use as ActivityPub playground to the
sidebar nav behind the feature flag.
- Wired up routing to Ember
- Setup the project as `admin-x-activitypub`

---------

Co-authored-by: Ronald Langeveld <hi@ronaldlangeveld.com>
2024-04-25 16:44:29 +08:00
Fabien O'Carroll
af02ca7044 Initial wire up of Posts -> Outbox flow
ref https://linear.app/tryghost/issue/MOM-29

This is very rough, and all still behind a flag. The idea is that any public
post which is published gets added to the Outbox of the site Actor. We also
dispatch an event, which will be used to deliver the Activity to any relevant
inboxes, but that is outside the scope of this commit.
2024-04-25 11:10:17 +07:00
renovate[bot]
25657b6a99 Update sentry-javascript monorepo to v7.112.1 2024-04-23 16:39:50 +02:00
renovate[bot]
3960e8caa1 Update sentry-javascript monorepo to v7.112.0 2024-04-23 14:11:30 +02:00
Daniel Lockyer
fda8aa5bfe Removed Sentry logging from XMLRPC service
refs https://ghost-foundation.sentry.io/issues/5135326925/

- the service tends to 503 all the time, and we don't really care enough
  for it to ping us in Sentry, as it's not something we control
- we can still keep logging the errors in case we need to go and look at
  what went wrong
2024-04-23 12:48:33 +02:00
renovate[bot]
d69a0aa1e6 Update dependency knex-migrator to v5.2.1 2024-04-22 12:29:48 +02:00
renovate[bot]
e4ffc7b8c0 Update dependency mysql2 to v3.9.7 2024-04-22 01:37:24 +00:00
renovate[bot]
69372d9018 Update dependency @sentry/profiling-node to v7.111.0 2024-04-19 16:56:32 +00:00
renovate[bot]
7fea4151d8 Update dependency @sentry/profiling-node to v7 2024-04-19 18:43:59 +02:00
Ghost CI
cb838c4bb6 v5.82.3 2024-04-19 16:04:16 +00:00
renovate[bot]
80fe672525 Update sentry-javascript monorepo to v7.111.0 2024-04-19 08:54:26 +02:00
Daniel Lockyer
f31814c804 Updated mysql2 dependency
refs #20050

- Renovate seems to be unable to bump the package past the security
  release, but unfortunately this release contains a breaking bug
- this commit manually bumps the package so we can get things flowing
  again
- the security release doesn't really affect us, but we should still try
  and keep on the latest
2024-04-19 08:48:24 +02:00
Chris Raible
a10b13916a
🐛 Fixed admin error when deleting an unsaved or imported post (#20053)
ref
https://linear.app/tryghost/issue/ENG-845/error-attempted-to-set-lexical-on-the-deleted-record
ref
[https://linear.app/tryghost/issue/ENG-854/🐛-deleting-imported-posts-makes-ghost-unresponsive](https://linear.app/tryghost/issue/ENG-854/%F0%9F%90%9B-deleting-imported-posts-makes-ghost-unresponsive)

- When deleting a post in the editor's Post Settings Menu, if the post
has unsaved changes (indicated by the hasDirtyAttributes property in the
editor), Admin will crash because it tries to save a post revision
before leaving the editor, but the post has already been deleted so
saving fails.
- This can occur when editing a post and quickly deleting it from the
Post Settings Menu before saving is completed.
- It can also occur when attempting to delete an imported post, as the
editor will parse the lexical from the server and may make some minor,
invisible-to-the-user changes to the lexical string locally (e.g. JSON
formatting, or updating the JSON to use extended version of base lexical
nodes), which triggers the same error.
- This fix bypasses the attempt to save a post revision when leaving the
editor if the post is already deleted, which allows the transition back
to the Posts route to succeed.
2024-04-18 10:02:02 -07:00
Kevin Ansfield
182fa62759
Bumped Koenig packages (#20047)
closes https://linear.app/tryghost/issue/MOM-39/

- fixes clicking on search results in internal linking PoC
- adds support for Node 20
2024-04-18 14:58:01 +00:00
Daniel Lockyer
10e81aeed8
ℹ️ Added support for Node 20
ref https://linear.app/tryghost/issue/ENG-765/add-support-for-node-20

- this adds support for Node 20 to Ghost and CI, as Node 20 is an LTS
  version and we should pick it up
2024-04-18 13:17:21 +02:00
Daniel Lockyer
8e0ad1a6fb
Fixed test on Node 20
refs f39d1d3aa3

- similar to the commit above, the JSON parser changed between Node 18
  and Node 20, so the error message changed too
- we actually just want to check the error is forwarded to the user, so
  we can do that by getting the error message from JSON.parse and check
  against that
2024-04-18 13:17:16 +02:00
renovate[bot]
ef6f79c272 Update dependency gscan to v4.43.0 2024-04-18 12:48:42 +02:00
Fabien O'Carroll
d34884fc6d Moved ActivityPub API to frontend URL
ref https://linear.app/tryghost/issue/MOM-48

This required some structural changes to our NestJS setup so that we can mount
it on multiple parts of the Ghost express app.

We've used the RouterModule to allow adding submodules that are mounted on
different paths, and we've had to be explicit about the base path for each
module. We've also had to switch back to using the Module decorator, because
RouterModule doesn't work with DynamicModule definitions.

Now that the NestJS app has knowledge of the full path, we need to "reset" the
url & baseUrl when passing the request into NestJS so that it can correctly
match the path. This is probably needed for the frontend too, for subdirs, but
that causes further issues - as this in prototype stage, we'll look later

Another issue is that NestJS replaces the express app instance with its own,
which isn't an issue for the Admin API (though we've fixed it anyway for
consistency), but did cause problems for the frontend, because the express app
is where view engine and directory information is stored.

The fix for this is to save a reference to the original ghost express
application, and reattach it to the request if it is not handled by Nest

Now that we have the Nest app mounted on the frontend, we're able to have it
handle the /.well-known/webfinger route with a proper controller, which is nice!
2024-04-18 13:26:24 +07:00
Fabien O'Carroll
885dc537d5 Added initial support for /.well-known/webfinger
ref https://linear.app/tryghost/issue/MOM-26
ref https://linear.app/tryghost/issue/MOM-27
ref https://webfinger.net/spec/

WebFinger is the protocol which allows different ActivityPub implementers to
find information about Actors, it's kinda like the entrypoint.

Given a username like @user@site.com, we can look up the URL for the Actor at

   https://website.com/.well-known/webfinger?resource=acct:user@site.com

This would then give us the info needed to discover the Actor's Inbox & Outbox
2024-04-17 16:12:57 +07:00
Fabien O'Carroll
ffbf2680ef Enabled Nest when ActivityPub flag is set
ref MOM-31
ref https://linear.app/tryghost/issue/MOM-31

We'll be building a lot of the new code for ActivityPub in Nest, so we'll need
to have it enabled in Ghost to work.
2024-04-17 14:00:07 +07:00
Fabien O'Carroll
0fb290a841 Added Feature Flag for ActivityPub
ref MOM-31
ref https://linear.app/tryghost/issue/MOM-31

Ronseal - let's goooooooo
2024-04-17 14:00:07 +07:00
renovate[bot]
96d0883928
🐛 Fixed file card button not being linked in emails (#20023)
ref https://linear.app/tryghost/issue/DES-202/

- bumped Koenig packages to include fix for incorrectly wrapped download image link in email rendering of file card
2024-04-16 10:37:28 +00:00
renovate[bot]
2af9c04477 Update CSS preprocessors 2024-04-16 09:44:36 +02:00
renovate[bot]
0802b9533c Update dependency semver to v7.6.0 2024-04-16 09:41:01 +02:00
Nicholas Mizoguchi
d6b7ebb517
Enforced more Mocha lint rules (#19720)
ref https://github.com/TryGhost/Ghost/issues/11038

1. Enforced lint rule
**[ghost/mocha/no-identical-title](https://github.com/lo1tuma/eslint-plugin-mocha/blob/main/docs/rules/no-identical-title.md)**
- Fixed relevant tests
2. Enforced lint rule
**[ghost/mocha/max-top-level-suites](https://github.com/lo1tuma/eslint-plugin-mocha/blob/main/docs/rules/max-top-level-suites.md)**
- No required fixes, as tests are compliant already

#### Additional details
Specifically for `ghost/mocha/no-identical-title` most fixes were simple
test description updates. Added comments to aid the PR review for the
ones that had relevant changes, and might require more attention. They
are as follows:
*
[e2e-api/admin/invites.test.js](https://github.com/TryGhost/Ghost/pull/19720#discussion_r1496397548):
Removed duplicated test (exact same code on both);
*
[e2e-api/admin/members.test.js](https://github.com/TryGhost/Ghost/pull/19720#discussion_r1496399107):
From the[ PR this was
introduced](73466c1c40 (diff-4dbc7e96e356428561085147e00e9acb5c71b58d4c1bd3d9fc9ac30e77c45be0L236-L237))
seems like author based his test on an existing one but possibly forgot
to rename it;
*
[unit/api/canary/utils/serializers/input/pages.test.js](https://github.com/TryGhost/Ghost/pull/19720#discussion_r1496400143):
The [page filter](https://github.com/TryGhost/Ghost/pull/14829/files)
was removed, so changed the description accordingly;
*
[unit/api/canary/utils/serializers/input/posts.test.js](https://github.com/TryGhost/Ghost/pull/19720#discussion_r1496400329):
The [page filter](https://github.com/TryGhost/Ghost/pull/14829/files)
was removed, so changed the description accordingly;
*
[unit/frontend/services/rendering/templates.test.js](https://github.com/TryGhost/Ghost/pull/19720#discussion_r1496402430):
Removed duplicated test
*
[unit/server/models/post.test.js](https://github.com/TryGhost/Ghost/pull/19720#discussion_r1496403529):
the change in [this
PR](https://github.com/TryGhost/Ghost/pull/14586/files#diff-c351cb589adefbb886570cfadb33b33eb8fdc12bde1024d1188cd18c165fc5e8L1010)
made three tests here mostly the same. Deduplicated them and kept only
one.
2024-04-16 09:37:06 +02:00
renovate[bot]
39a9a9bfd8 Update sentry-javascript monorepo to v7.110.1 2024-04-16 09:09:22 +02:00
renovate[bot]
b008c00feb Update sentry-javascript monorepo to v7.110.0 2024-04-15 09:47:44 +02:00
Ghost CI
ff03b2eedb v5.82.2 2024-04-15 01:06:07 +00:00
Ghost CI
bc0496e5d3 🎨 Updated Source to v1.2.2 2024-04-15 01:06:06 +00:00
Ghost CI
5e6cc0d746 🎨 Updated Casper to v5.7.2 2024-04-15 01:06:06 +00:00
Hannah Wolfe
145a184967 Added logging to RSS cache
- The RSS cache has lived for a really long time, but I'm not sure it's useful
- Want to be able to determine if it gets used much, and if not, then we can remove it
2024-04-11 10:33:53 +01:00
Michael Barrett
9e78412268
Added queue depth to requests (#19987)
refs
[CFR-14](https://linear.app/tryghost/issue/CFR-14/ensure-queue-depth-is-always-set-on-req)

Added queue depth to any request that passes through the request queue
middleware instead of only adding it to the request if it is queued.
This makes it easier to report on the queue depth within Elastic.
2024-04-11 09:24:04 +01:00
Kevin Ansfield
ebd36f2503
Added internalLinking labs flag (#20006)
ref https://linear.app/tryghost/issue/MOM-1
2024-04-10 12:13:37 +01:00
Ghost CI
f046442bd8 v5.82.1 2024-04-08 14:42:52 +00:00
Kevin Ansfield
5c05ebe6cb
Fixed browser tests broken by onboarding changes (#19998)
ref 78311591d0

- updated tests to not click a button on the setup/done screen that is no longer shown
- fixed setup flow showing an alert bar due to not handling the `TransitionAborted` error that is thrown by the setup/done->dashboard redirect
2024-04-08 15:15:04 +01:00
Kevin Ansfield
78311591d0
🎨 Improved post-setup onboarding flow (#19996)
ref https://linear.app/tryghost/issue/IPC-66/onboarding-checklist-v1

- replaced the setup/done screen with a new onboarding checklist shown on the dashboard
2024-04-08 13:03:41 +01:00
Ghost CI
b6195d204c v5.82.0 2024-04-05 16:05:12 +00:00
Chris Raible
01d0b2b304
Added new member signup flow behind labs flag (#19986)
ref https://linear.app/tryghost/issue/KTLO-1/members-spam-signups

- Some customers are seeing many spammy signups ("hundreds a day") — our
hypothesis is that bots and/or email link checkers are able to signup by
simply following the link in the email without even loading the page in
a browser.
- Currently new members signup by clicking a magic link in an email,
which is a simple GET request. When the user (or a bot) clicks that link, Ghost
creates the member and signs them in for the first time.
- This change, behind an alpha flag, requires a new member to click the
link in the email, which takes them to a new frontend route `/confirm_signup/`, then submit a form on the page which sends a POST request to the
server. If JavaScript is enabled, the form will be submitted
automatically so the only change to the user is an extra flash/redirect
before being signed in and redirected to the homepage.
- This change is behind the alpha flag `membersSpamPrevention` so we can
test it out on a few customer's sites and see if it helps reduce the
spam signups. With the flag off, the signup flow remains the same as
before.
2024-04-04 15:25:41 -07:00
Fabien O'Carroll
a262a64eea Moved additional payment methods into beta
ref ENG-812
ref https://linear.app/tryghost/issue/ENG-812

We want to roll this out as a beta feature
2024-04-04 23:54:01 +07:00
Fabien O'Carroll
c1b72b3997 Wired up additional payment types with labs flag
ref ENG-812
ref https://linear.app/tryghost/issue/ENG-812
2024-04-04 23:13:10 +07:00
Fabien O'Carroll
781599de62 Added a feature flag for additional payment methods
ref ENG-812
ref https://linear.app/tryghost/issue/ENG-812
2024-04-04 23:13:10 +07:00
Fabien O'Carroll
0bd4a800a5 Fixed error message for custom theme settings
ref ENG-809
ref https://linear.app/tryghost/issue/ENG-809

This includes the fix to the error message from gscan
2024-04-04 01:39:30 +07:00
Kevin Ansfield
d5a9731845
Fixed email_recipients indexes to match query usage (#19918)
closes https://linear.app/tryghost/issue/ENG-791/migration-to-fix-email-recipients-indexes

Our indexes over single columns (`delivered_at`, `opened_at`, `failed_at`) were ineffective because the only time we query those is alongside `email_id` meaning we were frequently performing full table scans on very large tables during our email analytics jobs.

- added migration to add new indexes covering `email_id` and the respective columns
- added migration to drop the old indexes that weren't being used in any query plans

Local runtime with ~2M email_recipient rows:
- before: 1.7s
- after: 99ms

Explain output...

before:
```
+----+-------------+------------------+------------+-------+----------------------------------------------------------------------------------+----------------------------------------------+---------+-------+--------+----------+------------------------------------+
| id | select_type | table            | partitions | type  | possible_keys                                                                    | key                                          | key_len | ref   | rows   | filtered | Extra                              |
+----+-------------+------------------+------------+-------+----------------------------------------------------------------------------------+----------------------------------------------+---------+-------+--------+----------+------------------------------------+
|  1 | UPDATE      | emails           | NULL       | index | NULL                                                                             | PRIMARY                                      | 98      | NULL  |      1 |   100.00 | Using where                        |
|  4 | SUBQUERY    | email_recipients | NULL       | range | email_recipients_email_id_member_email_index,email_recipients_failed_at_index    | email_recipients_failed_at_index             | 6       | NULL  |   2343 |     7.76 | Using index condition; Using where |
|  3 | SUBQUERY    | email_recipients | NULL       | ref   | email_recipients_email_id_member_email_index,email_recipients_opened_at_index    | email_recipients_email_id_member_email_index | 98      | const | 159126 |    50.00 | Using where                        |
|  2 | SUBQUERY    | email_recipients | NULL       | ref   | email_recipients_email_id_member_email_index,email_recipients_delivered_at_index | email_recipients_email_id_member_email_index | 98      | const | 159126 |    50.00 | Using where                        |
+----+-------------+------------------+------------+-------+----------------------------------------------------------------------------------+----------------------------------------------+---------+-------+--------+----------+------------------------------------+
```

after:
```
+----+-------------+------------------+------------+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------+---------+------+--------+----------+--------------------------+
| id | select_type | table            | partitions | type  | possible_keys                                                                                                                                                                 | key                                          | key_len | ref  | rows   | filtered | Extra                    |
+----+-------------+------------------+------------+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------+---------+------+--------+----------+--------------------------+
|  1 | UPDATE      | emails           | NULL       | index | NULL                                                                                                                                                                          | PRIMARY                                      | 98      | NULL |      1 |   100.00 | Using where;             |
|  4 | SUBQUERY    | email_recipients | NULL       | range | email_recipients_email_id_member_email_index,email_recipients_email_id_delivered_at_index,email_recipients_email_id_opened_at_index,email_recipients_email_id_failed_at_index | email_recipients_email_id_failed_at_index    | 104     | NULL |     60 |   100.00 | Using where; Using index |
|  3 | SUBQUERY    | email_recipients | NULL       | range | email_recipients_email_id_member_email_index,email_recipients_email_id_delivered_at_index,email_recipients_email_id_opened_at_index,email_recipients_email_id_failed_at_index | email_recipients_email_id_opened_at_index    | 104     | NULL | 119496 |   100.00 | Using where; Using index |
|  2 | SUBQUERY    | email_recipients | NULL       | range | email_recipients_email_id_member_email_index,email_recipients_email_id_delivered_at_index,email_recipients_email_id_opened_at_index,email_recipients_email_id_failed_at_index | email_recipients_email_id_delivered_at_index | 104     | NULL | 146030 |   100.00 | Using where; Using index |
+----+-------------+------------------+------------+-------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------+---------+------+--------+----------+--------------------------+
```
2024-04-03 17:52:52 +01:00
Kevin Ansfield
bd93bf0dea Optimised email stats aggregation query for typical column usage
ref https://linear.app/tryghost/issue/ENG-790/remove-use-of-sub-queries-in-email-analytics

- the `delivered_at` column is typically entirely/nearly entirely filled with values meaning the `IS NOT NULL` query matches a huge number of rows that MySQL has to fetch from the index to count
- using `IS NULL` switches that behaviour around as it will now match very few rows which has been shown in testing to be considerably quicker
- after switching to `IS NULL` the query returns an "undelivered" count rather than a "delivered" count, in order to keep the rest of the system behaviour the same we can calculate the delivered count by subtracting the query result from the total number of emails sent which we can fetch using a very fast primary key lookup query on the `emails` table
2024-04-03 16:27:23 +01:00
Kevin Ansfield
4f5a7b420e Removed use of subqueries in email analytics queries
closes https://linear.app/tryghost/issue/ENG-790/remove-use-of-sub-queries-in-email-analytics

Avoiding sub queries means we don't have a process tied up for longer than necessary and we can more easily see if one of the queries is non-performant.

- extracted the count queries into separate queries and used the retrieved values in the final update query
- removed a query by moving the email open rate calculation into JS as we've already fetched the necessary data before that point
2024-04-03 16:27:23 +01:00
renovate[bot]
541395434d Update sentry-javascript monorepo to v7.109.0 2024-04-01 14:10:07 +02:00
Ghost CI
d4d69e2ac5 v5.81.1 2024-04-01 06:16:31 +00:00
Steve Larson
78d2a5e3c0
🐛 Fixed flaky browser tests (#19929)
ref https://linear.app/tryghost/issue/CFR-13
- enabled saving traces on browser test failure; this makes troubleshooting a lot easier
- updated handling in offers tests to ensure the tier has fully loaded in the UI (not just `networkidle`)
- updated publishing test to examine the publish button reaction to the save action response instead of a 300ms pause

In general, our tests use a lot of watching for 'networkidle' - and sometimes just raw timeouts - which do not scale well into running tests on CI. In particular, 'networkidle' does not work if we're expecting to see React components' state updates propagate and re-render. We should always instead look to the content which encapsulates the response and the UI updates. This is something we should tackle on a larger scale.
2024-03-27 13:57:53 -05:00
renovate[bot]
dcbbfbba70 Update dependency express to v4.19.2 [SECURITY] 2024-03-27 11:18:44 +01:00
Sag
5c4a4e812c
Removed Powered by Ghost clicks in publisher analytics (#19926)
fixes https://linear.app/tryghost/issue/TRI-65/add-powered-by-ghost-badge-tracking

- clicks on the "Powered by Ghost" badge were unintentionally surfaced
in publisher analytics, under Newsletter Clicks
2024-03-26 17:51:23 +01:00
renovate[bot]
ec25aed06f fix(deps): update dependency mysql2 to v3.9.3 2024-03-26 07:46:09 +00:00
Fabien 'egg' O'Carroll
3d3b3ff701
Fixed Editors being able to invite Editors (#19904)
ref ENG-774
ref https://linear.app/tryghost/issue/ENG-774

Staff Tokens will have both a `user` and an `apiKey` present on the
`loadedPermissions`.

The check here for `apiKey` was written when we could assume that an
`apiKey` was an Admin Integration - so it completely overwrote the
previous `allowed` list. When we added the concept of Staff Tokens -
this resulted in a privilege escalation.

This is a good lesson in not using proxies or indicators for data, as
changes elsewhere can invalidate them - if we had been specific and
checked the role of the current actor we wouldn't've had this bug!
2024-03-26 00:45:08 +07:00
Daniël van der Winden
3fa363f944
Fixed design issue DES-4 (#19662)
Fixed inconsistencies in typography for footer and featured images, on desktop and mobile.
2024-03-25 12:08:34 +01:00
renovate[bot]
97c63e1735 chore(deps): update sentry-javascript monorepo to v7.108.0 2024-03-25 08:34:53 +00:00
Ghost CI
5d714f8a05 v5.81.0 2024-03-22 16:05:44 +00:00
Steve Larson
a1c4e64994
Added queueing middleware to handle high request volume (#19887)
ref https://linear.app/tryghost/issue/CFR-4/
- added request queueing middleware (express-queue) to handle high
request volume
- added new config option `optimization.requestQueue`
- added new config option `optimization.requestConcurrency`
- added logging of request queue depth - `req.queueDepth`

We've done a fair amount of investigation around improving Ghost's
resiliency to high request volume. While we believe this to be partly
due to database connection contention, it also seems Ghost gets
overwhelmed by the requests themselves. Implementing a simple queueing
system allows us a simple lever to change the volume of requests Ghost
is actually ingesting at any given time and gives us options besides
simply increasing database connection pool size.

---------

Co-authored-by: Michael Barrett <mike@ghost.org>
2024-03-21 09:25:07 -05:00
renovate[bot]
dfdd4e5cfa Update dependency express to v4.19.1 2024-03-21 11:50:48 +01:00
Ghost CI
b88ef5f816 Merged v5.80.5 into main 2024-03-21 09:51:52 +00:00
Ghost CI
40277465ba v5.80.5 2024-03-21 09:51:50 +00:00
Sag
c2320cd2ea Revert "Added referral tracking to the powered-by-ghost newsletter badge" (#19899)
refs https://ghost.slack.com/archives/CTH5NDJMS/p1710976281912809

- this reverts commit 9869d9adb6
- the referral query parameter is unintentionally surfacing in publisher
analytics
2024-03-21 10:37:42 +01:00
Sag
5477d70a0c
Revert "Added referral tracking to the powered-by-ghost newsletter badge" (#19899)
refs https://ghost.slack.com/archives/CTH5NDJMS/p1710976281912809

- this reverts commit 9869d9adb6
- the referral query parameter is unintentionally surfacing in publisher
analytics
2024-03-21 10:02:17 +01:00
renovate[bot]
28c851be6a Update dependency sanitize-html to v2.13.0 2024-03-20 19:10:32 +00:00
Fabien O'Carroll
cb72835af1 Removed support for id specific permissions
ref ENG-728
ref https://linear.app/tryghost/issue/ENG-728

This is not used anywhere, and makes the code more complicated, it's a good
step toward simplifying permissions and pulling them out of the database.
2024-03-21 00:21:40 +07:00
Fabien O'Carroll
e29c653ef6 Removed permissible method from setting model
ref ENG-728
ref https://linear.app/tryghost/issue/ENG-728

This implementation is essentially a no-op so we can remove it completely
2024-03-21 00:21:24 +07:00
Daniel Lockyer
27cc32ec25 Added comments count endpoint to robots.txt disallow list
fix https://linear.app/tryghost/issue/ENG-771/add-comments-count-endpoint-to-robotstxt-ignorelist

- we've seen web scrapers hitting this endpoint a lot, but the value to
  be taken from it is minimal for SEO purposes
- adding it to robots.txt should encourage web scrapers to ignore it,
  and we should see less traffic as a result
2024-03-20 14:48:54 +01:00
Fabien 'egg' O'Carroll
7cc65c18cc
Added missing permissions to Contributor & Editor (#19881)
ref ENG-728
ref https://linear.app/tryghost/issue/ENG-728

This is NOT a functionality change. The Post#permissible method unit
tests have been updated to pass `true` as `hasUserPermission` and we can
see that the permission functionality remains the same.

The permissible method of the post model is responsible for removing
permission based on the data that is being modified, but the permissions
module is setup to allow the permissible method to grant permission -
this means that we call permissible, even if the current actor doesn't
have permission, this results in code that is hard to understand and
manage.

We are going to be instead returning early if an actor does not have
permission, this will allow permissible method signatures to be greatly
simplified (removing the need for hasUserPermission, hasApiKeyPermission
& hasMemberPermission arguments).
2024-03-20 20:36:07 +07:00
renovate[bot]
38f8e05a3e Update dependency knex-migrator to v5.2.0 2024-03-20 10:54:21 +01:00
renovate[bot]
3ff2eecb76 Update dependency bookshelf-relations to v2.7.0 2024-03-19 11:20:22 +00:00
Daniel Lockyer
134c33cef5
🐛 Fixed missing source + resized images producing rendered 404 (#19869)
fixes https://linear.app/tryghost/issue/ENG-746/http-500-responses-when-handle-image-sizes-middleware-hits-missing

- in the event a request comes in for a resized image, but the source
image does not exist, we return a rendered 404 page
- we do this because we pass the NotFoundError to `next`, which skips
over the static asset code where we return a plaintext 404
- also included a breaking test that ensure we go to the next middleware
without an error
2024-03-18 18:32:10 +01:00
Fabien 'egg' O'Carroll
3f27ca5c00
Cached api controller pipelines (#19880)
ref ENG-761
ref https://linear.app/tryghost/issue/ENG-761

Creating these pipelines is expensive, and we don't want to do it
repeatedly for the same controller. Adding caching should reduce the
amount of time spent setting up pipelines for each usage of the `get`
helper.
2024-03-19 00:29:41 +07:00
Djordje Vlaisavljevic
a67342b06a
Add WIP onboarding checklist behind the flag (#19801)
ref https://linear.app/tryghost/issue/IPC-66/onboarding-checklist-v1

- Adds a basic version of a new onboarding checklist behind the feature
flag, without incomplete/complete state logic
- Links to Design settings, Members screen and new post
- Opens amodal that we’ll use as Share modal

---------

Co-authored-by: Daniël van der Winden <danielvanderwinden@ghost.org>
2024-03-18 14:53:01 +00:00
Daniel Lockyer
48ae822b9f Removed NewRelic from Ghost
fix https://linear.app/tryghost/issue/SLO-47/remove-newrelic-from-ghost

- we no longer need this in Ghost because we've doubled down on Sentry
  Performance
- we can remove this because we've seen it costs 200-300ms to include
  the dependency
2024-03-18 15:22:33 +01:00
Michael Barrett
949b157775
Fixed naming of connection pool metrics (#19874)
no refs
2024-03-18 09:59:16 +00:00
Ghost CI
715ff56032 v5.80.4 2024-03-18 09:42:10 +00:00
Ghost CI
6849aa9891 v5.80.3 2024-03-15 16:04:57 +00:00
Ghost CI
a52c645c44 🎨 Updated Source to v1.2.1 2024-03-15 16:04:57 +00:00
Ghost CI
8b438fa015 🎨 Updated Casper to v5.7.1 2024-03-15 16:04:57 +00:00
renovate[bot]
566a065f36 Update dependency socket.io to v4.7.5 2024-03-14 17:30:47 +00:00
Fabien 'egg' O'Carroll
6a35f6e4cc
Fixed get helper cache optimizations (#19865)
ref [ENG-747](https://linear.app/tryghost/issue/ENG-747/)
ref https://linear.app/tryghost/issue/ENG-747

H'okay - so what we're trying to do here is make get helper queries more
cacheable. The way we're doing that is by modifying the filter used when
we're trying to remove a single post from the query.

The idea is that we can remove that restriction on the filter, increase
the number of posts fetched by 1 and then filter the fetched posts back
down, this means that the same query, but filtering different posts,
will be updated to make _exactly_ the same query, and so share a cache!

We've been purposefully restrictive in the types of filters we
manipulate, so that we only deal with the simplest cases and the code is
easier to understand.
2024-03-15 00:18:15 +07:00
Fabien 'egg' O'Carroll
e607556913
Stopped us from rendering no html when error occurs
no-issue

We need to make sure that we return, otherwise we'll end up skipping the
error handler middleware and trying to render.
2024-03-14 16:34:13 +00:00
Fabien 'egg' O'Carroll
aaa19a535a
🐛 Fixed 500 errors for invalid theme layouts (#19848)
ref ENG-742
ref https://linear.app/tryghost/issue/ENG-742

We don't do any parsing of layouts in gscan, which means themes can be
uploaded which use non-existent files for their layout.

We can catch the error in the res.render call, and wrap it, just like we
do for missing templates (e.g. the StaticRoutesRouter)
2024-03-14 23:12:26 +07:00
Aileen Booker
f16d9802d0 Added ability to pass minThreshold for Milestone Slack notifications
closes ENG-632

- This listens to a new property in the `milestones` config to set a minimum value of Milestones we wanna use the Slack notification service for
2024-03-14 12:06:43 -04:00
Michael Barrett
60d81b2003
🐛 Fixed /p/ redirects not being indexed by search engines (#19864)
ref
[ENG-741](https://linear.app/tryghost/issue/ENG-741/🐛-our-robotstxt-config-causes-indexing-issues-for-customers-who-have)

`/p/` has been dropped from the `robots.txt` file so that search engines
can index the pages at these locations. In the event that the page at
the location is a preview page, the existing robots meta tag on the page
will prevent indexing.
2024-03-14 14:44:54 +00:00
Fabien O'Carroll
39da5a1f88 Revert "Optimised queries made by get helper for posts"
no-issue

This was incorrectly merged - reverting until the work is complete
2024-03-14 20:26:01 +07:00
Sag
7a40ab52fb
🐛 Fixed adding recommendation when oembed fails (#19861)
refs https://linear.app/tryghost/issue/ENG-750

- when adding a recommendation, we fetch the recommended site's metadata
- before this change, if the metadata fetch failed for some reason, we'd show an error and block the recommendation from being added
- after this change, we use fallback values if the metadata fails to fetch, instead of blocking the recommendation from being added. We use the site domain as the title and leave the rest empty (no favicon, no description)
- this change also means we are not checking whether a site exists or not for the publisher anymore. It’s then up to the publisher to make sure they don’t enter broken URLs
2024-03-14 11:36:28 +01:00
Fabien 'egg' O'Carroll
52a28c0059
Optimised queries made by get helper for posts (#19859)
ref ENG-747
ref https://linear.app/tryghost/issue/ENG-747

H'okay - so what we're trying to do here is make get helper queries more
cacheable. The way we're doing that is by modifying the filter used when
we're trying to remove a single post from the query.

The idea is that we can remove that restriction on the filter, increase
the number of posts fetched by 1 and then filter the fetched posts back
down, this means that the same query, but filtering different posts,
will be updated to make _exactly_ the same query, and so share a cache!

We've been purposefully restrictive in the types of filters we
manipulate, so that we only deal with the simplest cases and the code is
easier to understand.
2024-03-13 19:27:27 +00:00
renovate[bot]
48782df301 Update dependency newrelic to v11.13.0 2024-03-13 17:14:53 +00:00
Kevin Ansfield
47e6911ca0
Added cache-control header back to /auth-frame/ response (#19858)
ref https://linear.app/tryghost/issue/ENG-721

- when changing the response to a `204` for requests with no cookie we'd lost the `Cache-Control: public, max-age: 0` header which meant some cache systems weren't caching as efficiently as possible
2024-03-13 16:00:46 +00:00
Sag
9869d9adb6
Added referral tracking to the powered-by-ghost newsletter badge (#19850)
ref https://linear.app/tryghost/issue/TRI-65

In the context of referrals, we want to understand how useful our
“Powered by Ghost” badges are.

There are currently a few places where the “Powered by Ghost” badge can
be rendered:
- in newsletters (enabled/disabled by publisher, on a newsletter basis)
- in Portal popups, e.g. member signup/signin/account settings
- in the footer of some themes, including Source & Casper

We're adding the query param ?via to evaluate the usage of the badge in
newsletters.
2024-03-13 16:03:13 +01:00
Fabien "egg" O'Carroll
0fb0c6c2b5 Added NestJS Playground
no-issue

This adds the barebones of a NestJS application wired up to the Admin API
behind a feature flag, so that we can experiement with how to use Nest in the
context of Ghost
2024-03-13 19:44:06 +07:00
Fabien "egg" O'Carroll
d2620171ea Refactored auth services so they can be used in Nest
no-issue

This decouples the business logic from the express middleware so that it can be
used inside of a NestJS application.
2024-03-13 19:44:06 +07:00
Sag
59bbade630
Fixed browser tests (#19852)
no issue

- browser tests were failing due to the renaming of a button
2024-03-13 12:54:19 +01:00
Fabien 'egg' O'Carroll
5a5ddcb609
🐛 Fixed Tiers API erroring when invalid filter passed (#19845)
closes ENG-730
closes https://linear.app/tryghost/issue/ENG-730/

We've updated the input serializer to parse the filter, and responded
with an error if it cannot be parsed correctly.

Now that it's parsed, we can pass a mongo query object through the
stack, which will lend itself to better typing for this code, which is a
direction we want to go in anyway. We've had to update all the internal
usages of the `browse` method to use mongo query objects.
2024-03-13 00:25:42 +07:00
Daniel Lockyer
55791a8c64 Switched to throwing error upon failed image processing
ref https://linear.app/tryghost/issue/ENG-740/http-500-error-when-image-processing-fails
refs 4aad551c72

- upon further discussion, we've decided it's better to throw an error
  in this case because the uploaded image is deemed invalid and storing
  it on the filesystem might cause more issues with resizing/further
  processing in the future
- this commit implements that and alters the tests
2024-03-12 16:24:29 +01:00
Daniel Lockyer
4aad551c72 🐛 Fixed HTTP 500 error when image processing fails during upload
fixes ENG-740
fixes https://linear.app/tryghost/issue/ENG-740/http-500-error-when-image-processing-fails

- in the event the image transform library throws (which can happen for
  many reasons; sharp/libvips can come across a number of errors), we
  currently return this as a HTTP 500 error to the user
- in this case, we should just try-catch the call and jump to the
  non-processing flow where it just saves the original image
- also added breaking test
2024-03-12 15:33:17 +01:00
Kevin Ansfield
ef143978e7
🎨 Reduced requests and 403 responses for comments auth check (#19840)
closes https://linear.app/tryghost/issue/ENG-721
ref https://linear.app/tryghost/issue/ENG-708

Comments-UI loads `/ghost/admin-frame/` in an iframe to check if a Staff User is authenticated in order to  show moderation options. That iframe request loads a HTML page which in turn contains a script that fires off an API request that attempts to fetch the logged-in user details, resulting in a 403 "error" showing up when not authenticated. In the vast majority of cases there will be no staff user authenticated so lots of extra requests and "errors" are seen unnecessarily.

- adjusted the `/ghost/auth-frame/` endpoint to check if the request contains an Admin session cookie
  - if it does, continue as before with rendering the HTML page so the script is loaded
  - if it doesn't, return an empty 204 response avoiding the script request and subsequent 403-generating API request
- eliminates the 403 error being generated for all typical visitor traffic, the error should only be seen when an Admin was previously logged in but their cookie is no longer valid (either from logging out, or going past the 6month validity period)
2024-03-12 12:27:18 +00:00
Daniel Lockyer
5fa4496d52 🐛 Fixed HTTP 500 responses when oembed endpoint receives error
fixes https://github.com/TryGhost/Product/issues/4237

- this fixes the fact that we return a HTTP 500 response when the oembed
  library receives an error, such as a 401 or 403
- includes special handling for cases where we want to return a slightly
  different error message
- also adds unit tests for @tryghost/oembed-service package
2024-03-12 12:31:44 +01:00
Daniel Lockyer
6842d599e9 🐛 Fixed handling of image uploads with overly long filenames
fixes ENG-733
ref https://linear.app/tryghost/issue/ENG-733/handle-image-uploads-where-name-is-too-long

- filesystems usually have a filename length limit; ie. on macOS it is
  255 characters
- if a file is uploaded with a longer filename, we'll return a HTTP 500
- we shouldn't do this as it is user error, so we can just catch the
  error code and return BadRequest
- this implements that, and adds a breaking test
2024-03-12 12:31:44 +01:00