Commit Graph

3012 Commits

Author SHA1 Message Date
Ghost CI
88ad62558a Merged v5.82.8 into main 2024-05-08 19:30:39 +00:00
Ghost CI
f303432ba9 v5.82.8 2024-05-08 19:30:37 +00:00
Daniel Lockyer
7e9d82655e Added extra validation for reset_password endpoint
fix https://linear.app/tryghost/issue/SLO-104/cannot-read-properties-of-undefined-reading-0-an-unexpected-error

- if the request body didn't contain the correct keys, it'd just HTTP
  500 out of there
- this adds some optional chaining so we end up with undefined if
  anything isn't as expected, and the following if-statement does the
  rest of the check for us
- this also adds a breaking test (the first E2E test for authentication, yay!)
2024-05-08 18:05:04 +02:00
Daniel Lockyer
76c6e92006
Handled invalid files when uploading DB zips (#20165)
fix
https://linear.app/tryghost/issue/SLO-103/invalid-comment-length-expected-7-found-0-an-unexpected-error-occurred

- similar to
e8e3447f15,
this captures a specific error from yauzl and throws a user-friendly
error
- perhaps in the future we can just look for yauzl errors and always
return user-friendly errors, but let's monitor that first
- also includes a breaking test
2024-05-08 14:59:34 +02:00
Daniel Lockyer
f276abf9e8 Condensed logic for determining whether to send Sentry events
refs https://docs.sentry.io/platforms/javascript/configuration/filtering/#using--1%20

- this simplifies our logic to determine whether we should send events
  by moving the code to `beforeSend`
- `errorHandler` is going away in Sentry v8 so this results in a shorter
  diff in the future
- the logic should be the same, always send non-Ghost errors, and only
  send HTTP 500 Ghost errors
2024-05-08 14:25:00 +02:00
Daniel Lockyer
77fc66340a Refactored Sentry beforeSend code
- this tidies up the implementation by removing the very verbose ternary
  operator, and also uses the optional chaining operator
2024-05-08 14:25:00 +02:00
Daniel Lockyer
c4ad593a5a Added JSDoc types to Sentry
- this helps editors with identifying what the types of parameters are
2024-05-08 14:25:00 +02:00
Daniel Lockyer
e8e3447f15 Fixed handling empty zip file uploads
fix https://linear.app/tryghost/issue/SLO-102/end-of-central-directory-record-signature-not-found-an-unexpected

- previously, uploading an empty zip would result in a HTTP 500 error
  because yauzl would error and we'd bubble that up as an
  InternalServerError
- now, we catch the specific error message and return a more user
  friendly error
- also includes tests and sample zip file
2024-05-08 11:19:47 +02:00
Daniel Lockyer
d82b136a6a Handled uploads with invalid form bodies
fix https://linear.app/tryghost/issue/SLO-101/http-500-with-invalid-multipart-data

- previously, busboy would error out if we supplied a body that was
  invalid (such as an empty FormData)
- we would then return a HTTP 500 to the user, which causes all manner
  of problems
- now we catch errors from busboy and return a nice BadRequestError
2024-05-08 11:19:47 +02:00
Daniel Lockyer
ae88dc8548 Handled invalid timestamp format in filters
fix https://linear.app/tryghost/issue/SLO-85/fix-http-500-on-contentposts

- in the event we give the incorrect format in a filter, MySQL will
  throw an error and we'll throw a HTTP 500 error
- we can capture this error and return a more useful error to the user
- ideally we'd do this in a validation step before attempting the query,
  but parsing this out of NQL and detecting which columns are DATETIME
  could be quite tricky
2024-05-08 09:28:56 +02:00
Daniel Lockyer
82c612bad9 Rolled out API framework JSDoc typing to more places
- this updates a bunch of places where we're just using Object to cheat
  the system
- doing this means editor autocomplete and basic type checking is better
  because we now have proper types in place
- functionality should not change, these are just comments
2024-05-08 09:28:31 +02:00
renovate[bot]
1fea2fc616 Update dependency semver to v7.6.1 2024-05-08 09:04:44 +02:00
Ghost CI
7201006f05 Merged v5.82.7 into main 2024-05-07 18:13:36 +00:00
Ghost CI
000616ac02 v5.82.7 2024-05-07 18:13:34 +00:00
Daniel Lockyer
2659e5aa40 Added handling for parsing errors with user-submitted HTML
fix https://linear.app/tryghost/issue/SLO-87/cannot-read-properties-of-undefined-reading-createimpl-an-unexpected
refs https://github.com/jsdom/jsdom/issues/3709

- in the event we are given some HTML to parse, and that fails, we
  currently return a HTTP 500 because it's unhandled
- the instance we saw was due to `<constructor>` crashing jsdom, we've
  opened an issue for that
- in terms of handling the error gracefully, we can surround the code
  in a try-catch and return a more suitable error. I've gone for a
  ValidationError for now - you could debate whether a different one is
  more appropriate
- also added Sentry error capturing so we're not blind to these,
  ultimately we should make sure the parser can handle all
  user-submitted data
2024-05-07 17:25:48 +02:00
Daniel Lockyer
dccb4ac84e Cleaned up unused controller method
- this isn't being used anywhere, so we can clean it up
2024-05-07 11:44:07 +02:00
Daniel Lockyer
1fd155d56a Fixed extra arguments being supplied to function calls
- identified by tsc, this shouldn't change any functionality because
  there were extra arguments being supplied and were unused
2024-05-07 11:44:07 +02:00
Daniel Lockyer
29cc3003c7 Fixed misc JSDoc types
- nothing crazy, just fixing minor typing issues that I've come across
2024-05-07 11:44:07 +02:00
Ronald Langeveld
4f3bfebfea
Revert "Added latest_event_timestamp to email table (#20118)" (#20149)
This reverts commit 3246a8d2c9.
2024-05-07 08:58:42 +00:00
Daniel Lockyer
c298db912c Added JSDoc types for API controllers
- this adds a simple set of types to the @tryghost/api-framework
  package that should describe all of the keys available on a
  controller, and then rolls it out to all API controllers
- unfortunately, due to https://github.com/microsoft/TypeScript/issues/47107, we have
  to split apart `module.exports` into a variable assignment in order for type-checking
  to be done
- the main benefit of this is that `frame` is now typed, and editors understand what keys
  are available, so intellisense works properly
2024-05-07 10:49:44 +02:00
Ronald Langeveld
3246a8d2c9
Added latest_event_timestamp to email table (#20118)
ref ENG-832

- Added migrations for `latest_event_timestamp` column in emails table.
- updated schema
- updated emails model
2024-05-07 16:12:51 +08:00
Daniel Lockyer
de435238f4 Fixed wrong status code type
- `statusCode` should be a number, but we were passing a string
- this doesn't really affect anything, but tsserver was flagging it up
  as the wrong type
2024-05-07 08:29:48 +02:00
Daniel Lockyer
b29c897da3 Fixed passing error to IncorrectUsageError
- we should pass it as `err` and not `error`
- this probably slipped in because the catch parameter is called
  `error`, so I've updated that and fixed the references
2024-05-07 08:29:48 +02:00
Daniel Lockyer
efc59dd315 Fixed extra parameter to function
- `serializeTier` only takes 2 parameters
- flagged by tsserver
2024-05-07 08:29:48 +02:00
Daniel Lockyer
f72d7b77ac Fixed miscellaneous jsdoc comments
- this helps tsserver figure out what the type of things is around our
  codebase
- nothing crazy, mostly Express types for the middleware, application and router levels
2024-05-07 08:29:48 +02:00
Daniel Lockyer
265a8dd16f Added function names to more middleware
refs 319f251ad2

- this helps debugging because all middleware in the stack will have a
  function name, so it'll show up instead of `<anonymous>`
2024-05-06 17:51:39 +02:00
Daniel Lockyer
a50bb8d79e Added missing error messages for members uploader
fix https://linear.app/tryghost/issue/SLO-97/missing-messages-for-members-file-upload

- these were missing, so if the members importer wasn't given a file, it
  would crash with an HTTP 500 error
- also added a test to ensure we get a 422 back
2024-05-06 15:17:25 +02:00
Daniel Lockyer
5a8145139a Fixed handling cutoff boundary data in image + media upload
fix https://linear.app/tryghost/issue/SLO-95/unexpected-end-of-multipart-data-for-broken-image-upload-request

- in the event the client sends an invalid body to the image or media
  upload endpoints, Dicer will throw an error if the boundary data is
  malformed
- previously, we've just been bubbling that up as an InternalServerError
  and that results in an HTTP 500
- we can capture errors produced by dicer and return a handled
  BadRequestError, as it's the client's fault
- also includes breaking tests
2024-05-06 13:41:25 +02:00
Daniel Lockyer
3e79712466 Fixed handling malformed image + media upload requests
fix https://linear.app/tryghost/issue/SLO-94/unexpected-field-when-given-broken-image-upload-request

- in the event the body of an image or media upload request is malformed
  (broken metadata / blob or something), we get a MulterError and this
  bubbles up as an InternalServerError and spits out a HTTP 500
- we can capture this and return a BadRequestError, as it's the client's
  fault for not providing the correct body
- this implements that and adds breaking tests
2024-05-06 13:24:26 +02:00
Ghost CI
f34c33f330 v5.82.6 2024-05-03 16:04:14 +00:00
Chris Raible
d8b629c713
Added an optional timeout parameter to AdapterCacheRedis (#20131)
ref
https://linear.app/tryghost/issue/ENG-902/add-an-optional-timeout-in-the-redis-cache-adapter-in-case-redis

- Added an optional timeout parameter to AdapterCacheRedis, so that the
`get(key)` method will return `null` after the timeout if it hasn't
received a response from Redis
- When load testing the `LinkRedirectRepository` with the Redis cache
enabled on staging, we noticed that for some reason Redis stopped
responding to commands for ~30 seconds.
- The `LinkRedirectRepository` was waiting for the Redis cache to
respond and resulted in a drastic increase in response times for link
redirects
- This change will allow us to set a timeout on the `get(key)` method,
so that if Redis doesn't respond within the timeout, the method will
return `null` as if it were a cache miss.
- Then the `LinkRedirectRepository` will fall back to the database and
return the link redirect from the database instead of waiting
indefinitely for Redis to respond
2024-05-02 20:39:23 -07:00
Chris Raible
b9f7ea65e9
Revert "Added new member signup flow behind labs flag (#19986)" (#20130)
ref https://linear.app/tryghost/issue/KTLO-1/members-spam-signups

This reverts commit 01d0b2b304.

- Removed the new member signup flow because it didn't solve the
problems with spam signups
2024-05-02 13:02:32 -07:00
Daniel Lockyer
7950122ffe Protected against deleting non-existent image during upload
fix https://linear.app/tryghost/issue/SLO-93/undefined-path-error-with-bad-image-upload

- in the event we receive a request to upload an image, that doesn't
  contain an image, we still try and unlink the files
- this is a dangling promise, so it doesn't cause an explicit HTTP
  error, but it does show up as a console error
- fixed it by checking for the path, and early returning if it doesn't
  exist
- also added a test that would fail without this
2024-05-02 17:29:03 +02:00
Sag
e996213122
Moved POST /members/api/member behind alpha flag (#20124)
ref https://linear.app/tryghost/issue/SLO-78

- the `POST /members/api/member` endpoint is solely used by the alpha
feature `membersSpamPrevention` and should not be available otherwise
2024-05-02 16:34:32 +02:00
Daniel Lockyer
cc76fda3e8 Enabled includeLocalVariables option in Sentry
ref https://linear.app/tryghost/issue/SLO-92/enable-extra-sentry-integrations
refs https://docs.sentry.io/platforms/javascript/guides/node/configuration/options/#include-local-variables

- this will capture the local stack variables when producing a stack
  trace, which would be super useful when debugging
2024-05-02 13:52:40 +02:00
Daniel Lockyer
5b28dc9246 Enabled Sentry's ExtraErrorData integration
refs https://docs.sentry.io/platforms/javascript/guides/node/configuration/integrations/extraerrordata/
ref https://linear.app/tryghost/issue/SLO-92/enable-extra-sentry-integrations

- this enables the ExtraErrorData integration, which should help us
  capture more of the properties of the errors we're producing, which
  _may_ help with debugging
2024-05-02 13:52:40 +02:00
renovate[bot]
a33dccf8cd Update TryGhost packages 2024-05-01 17:01:41 +02:00
Daniel Lockyer
3f7a7fff44 Fixed HTTP 500 when adding unknown products to member
fix https://linear.app/tryghost/issue/SLO-89/cannot-read-properties-of-null-reading-get-an-unexpected-error

- if we pass an invalid ID when updating the products on a member, we
  throw a HTTP 500 error because `product` is `null`
- we can check for this and return a BadRequestError, because the user
  supplied an incorrect ID
2024-05-01 16:54:35 +02:00
Ghost CI
6b32548e5c Merged v5.82.5 into main 2024-05-01 13:22:28 +00:00
Ghost CI
a4c478958b v5.82.5 2024-05-01 13:22:26 +00:00
renovate[bot]
746df63307
Update dependency gscan to v4.43.1 (#20109)
ref https://linear.app/tryghost/issue/ENG-842/gluster-file-name-length-limit

- gscan v4.43.1 contains a bug fix to throw an large filenames
2024-05-01 11:26:21 +00:00
Daniel Lockyer
ddac3a9e8b Removed Sentry error capturing for failed URL decoding
fix https://linear.app/tryghost/issue/SLO-79/incorrectusageerror-the-url-httpsblogkongregatecompercentc0-couldnt-be

- we added this Sentry captureException whilst fixing a bug where
  decodeUrl could fail, and throw a 500 exception
- we added handling for that case and returned an empty string, but we
  also added Sentry error capturing
- at this point, I don't think we need to be capturing errors in Sentry,
  because the issue is already handled, and it only usually happens with
  malicious/incorrect URLs
- this is our #2 cause of Sentry alerts, so it's good to clean it up
2024-05-01 10:15:13 +02:00
renovate[bot]
60a3b5a913 Update TryGhost packages 2024-05-01 08:49:04 +02:00
Daniel Lockyer
7109743282 Fixed overly verbose DatabaseInfo require
- we don't need to deep require into the library as it exports what we
  need on the surface
- this should unblock https://github.com/TryGhost/Ghost/pull/19002, as
  it's randomly failing with this require
2024-04-30 21:39:56 +02:00
Michael Barrett
4cd85ab8b7
Added timeout when resizing an image (#20087)
refs
[ENG-827](https://linear.app/tryghost/issue/ENG-827/🐛-crash-on-resizing-animated-gif)

Added a timeout to the image resizing middleware to prevent crashes when
an image is taking too long to resize. When the timeout is reached and
the image has not been resized, the middleware will return the original
image
2024-04-30 08:39:30 +01:00
Ronald Langeveld
b2970cb4e0
Added integrity test for flags (#20094)
ref
https://ghost.slack.com/archives/C02G9E68C/p1714047709694639?thread_ts=1713956576.497899&cid=C02G9E68C
    
    - Ensures unique feature flags, avoiding configuration conflicts.
    - Enhances code reliability and simplifies feature tracking.
    - Prevents bad rebases was the reason for the initial duplication.
2024-04-29 02:39:15 +00:00
renovate[bot]
305029bc38 Update dependency ws to v8.17.0 2024-04-29 01:23:51 +00:00
renovate[bot]
f18f08928f Update dependency yjs to v13.6.15 2024-04-29 01:22:59 +00:00
Ghost CI
cf9ba60524 v5.82.4 2024-04-26 16:04:30 +00:00
Chris Raible
dcd65bfa4f
Added caching to the LinkRedirectRepository (#20036)
ref
https://linear.app/tryghost/issue/ENG-851/implement-a-minimal-but-complete-version-of-redirect-caching-to
ref https://app.incident.io/ghost/incidents/55

Often immediately after sending an email, sites receive a large volume
of requests to LinkRedirect endpoints from members clicking on the links in
the email.

We currently don't cache any of these requests in our CDN, because we
also record click events, update the member's `last_seen_at` timestamp,
and send webhooks in response to these clicks, so Ghost needs to handle
each of these requests itself. This means that each of these LinkRedirect requests
hits Ghost, and currently all these requests hit the database to lookup
where to redirect the member to.

Each one of these requests can make up to 11 database queries, which can
quickly exhaust Ghost's database connection pool. Even though the
LinkRedirect lookup query is fairly cheap and quick, these queries aren't
prioritized over the "record" queries Ghost needs to handle, so they can
get stuck behind other queries in the queue and eventually timeout.

The result is that members are unable to actually reach the destination
of the link they clicked on, instead receiving a 500 error in Ghost, or
it can take a long time (60s+) for the redirect to happen.

This PR uses our existing `adapterManager` to cache the redirect lookups
either in-memory or in Redis (if configured — by default there is no caching). This only removes 1 out of
11 queries per redirect request, so it won't reduce the load on the DB
drastically, but it at least decouples the serving of the LinkRedirect from
the DB so the member can be redirected even if the DB is under heavy
load.

Local load testing results have shown a decrease in response times from
60 seconds to ~50ms for the redirect requests when handling 500 requests
per second, and reduced the 500 error rate to 0.
2024-04-25 19:17:25 -07:00