closes https://linear.app/tryghost/issue/MOM-172
- staff users with no posts won't have a front-end URL so it can be confusing for them to appear in the internal link search results within the editor
- added filtering for `/404/` in staff URLs so we don't list unlinkable staff members
REF MOM-147
- Updated newsletter preview in settings to match the newsletter
template
- Updated spacing for a few edge-case newsletter template combinations
closes https://linear.app/tryghost/issue/MOM-83
- added additional labs flag to allow internal testing prior to private beta release
- bumped Koenig packages containing support for @-link feature
REF MOM-119
- Split subhead feature flag into two: editorSubtitle and
newsletterSubtitle
- Updated UI copy, feature flag names and class names from subhead to
subtitle
refs MOM-152 MOM-148 MOM-151
- Added Subheads behind a flag + toggle in settings.
- Removes Excerpt fields from post settings if flag is enabled.
- Added subhead toggle in newsletter settings.
- Loads of styling
---------
Co-authored-by: Sanne de Vries <sannedv@protonmail.com>
ref https://linear.app/tryghost/issue/SLO-128
- Sentry will record session replays when errors happen in Admin (Settings, Editor) to facilitate debugging
- The recorded sessions mask sensitive information (input fields, media items, content in the editor, metrics in the dashboard)
REF MOM-146
- These styling changes are a general newsletter template improvement,
and are also laying the groundwork for including a subhead in the
newsletter header. Both the newsletter template as well as the
newsletter preview in settings have been updated.
Got some code for us? Awesome 🎊!
Please include a description of your change & check your PR against this
list, thanks!
- [ ] There's a clear use-case for this code change, explained below
- [ ] Commit message has a short title & references relevant issues
- [ ] The build will pass (run `yarn test:all` and `yarn lint`)
We appreciate your contribution!
---------
Co-authored-by: Ryan Feigenbaum <48868107+royalfig@users.noreply.github.com>
We want to use a randomly generated 64 byte secret for the hmac, and
utf8 encoding isn't nice to work with for this, so we're going to use a
base64 string and decode it into a buffer for the secret.
- we don't need this in `ghost/core` as it's not used in there
- we need to declare this dependency for the apps, as they use it for
running tests
- this doesn't change the lockfile but it means we're declaring the
dependency in the right places now
ref
https://linear.app/tryghost/issue/KTLO-45/deploy-members-caching-solution-to-a-single-site-to-validate-and-test
Currently we only cache publicly available content. Any content that is
accessed by a logged in member is only cached for that specific member
based on their cookie. As a result, almost all requests from logged in
members bypass our caching layer and reach Ghost, which adds unnecessary
load to Ghost and its database.
This change adds experimental headers that allow our CDN to understand
which tier to cache the content against, and securely tell the CDN which
tier a logged in member has access to. With these changes, we can cache
the member content against the tier, rather than the individual member,
which should result in a higher cache HIT ratio and reduce the load on
Ghost.
For requests to the frontend of the site, Ghost will set a custom
`X-Member-Cache-Tier` header to the ID of the tier of the member who is
accessing the content. This tells the CDN which tier to cache the
content against.
For requests to either `/members/?token=...` endpoint (the magic link
endpoint) or `/members/api/member`, Ghost will set a `ghost-access` and
`ghost-access-hmac` cookie with the ID of the tier of the logged in
member. With these two pieces of information, our CDN can serve cached
content to logged in members.
These headers are experimental, and can only be enabled via Ghost's
config. To enable these headers, set `cacheMembersContent:enabled` to
`true` and provide an HMAC key in `cacheMembersContent:hmacSecret`.
DES-192
We often hear that people are not aware of the new features we ship.
Ways in which people can find out are social media/changelog/dashboard –
all of these are easy to miss. We'd like to introduce a template for a
simple notification in the sidebar that can be used any time a new and noteworthy feature has
shipped. The purpose of this is simply to notify and will
disappear forever after it's been dismissed.
ref https://linear.app/tryghost/issue/CFR-29
- Removed the mobiledoc and lexical columns from the posts input
serializer, meaning they will no longer be queried for.
Get helpers are essentially a gateway to the Content API. We already
strip out the mobiledoc and lexical fields in the output
serializer/returned response, but this means we're passing the mobiledoc
and lexical fields back from the db. This is pointless and these fields
are substantial in size - by far the largest fields in the whole ghost
db - leading to slowed performance.
I've updated the posts input serializer to strip out the lexical and mobiledoc
columns so we stop doing a `select *` with every query.
ref ENG-824
- the bug is causing resize prefixes being added to images served from
outside of Ghost.
- this now would only append the prefex to images served by Ghost and
other images urls' would get served as is.
- we can determine that by checking whether imageName doesn't exist,
meaning the source is a third party.
- this mostly affect edge case users, eg where a feature image url was
passed in via the API and doesn't get served by Ghost.
refs CFR-21
Reorganised middleware execution so that member data is not redundantly loaded for static assets or the sitemap.
---------
Co-authored-by: Michael Barrett <mike@ghost.org>
The use of Article and Actor in Activity meant that we got way more data in the
JSONLD representation, but it wasn't be picked up when reconstructing from data
over the wire. This makes sure that we can recreate the object from the JSONLD.
ref https://linear.app/tryghost/issue/MOM-126
Similar to using the Article object for object in Activity, this allows us to
more easily pull out all of the extra data for Actors. At the moment its the
full JSONLD representation, but we can slim that down in future.
ref https://linear.app/tryghost/issue/MOM-128
We want to render more than just the content, so we need to bulk out the
Article objects with metadata like feature images etc...
ref https://linear.app/tryghost/issue/MOM-127
This will save us on network fetches when trying to hydrate the inbox, instead
we can include all of the data we want/need. I had to improve the typing a bit
here to work properly which meant ensuring that we have a `type` property.
ref https://linear.app/tryghost/issue/MOM-126
Now that we're setting the recipient of our Create Activites to the Followers
Collection, we need to actually dereference it and pull out all the inboxes.
This is all done over the network at the moment, but we'll start storing this
information locally when we've got the DB wired up.
ref https://linear.app/tryghost/issue/MOM-126
This is the first step of handling delivery of Activities to our Followers as
we can dereference the Collection and get a list of all our Followers Inboxes
Now that we have the URI object, we don't need to convert the Actor to JSONLD
to get its resource ids. Instead we can have shared getters that expose the ids
as a URI, that can be realized as/when they're needed with the base URL.
This makes the code a little cleaner and more performant.
refs #20197
- adds a jackspeak resolution to Ghost core so we can try and ensure the compatible version of jackspeak/string-width is used when the lockfile is regenerated
ref https://linear.app/tryghost/issue/MOM-108
Apologies to my future self and maintainers if you come across this commit.
This is a bit of a mega commit because we need to cut corners somewhere and it
came down to commit atomicity or tests/code quality.
The main changes here are a bunch of tests, as well as some scaffolding for
Inbox handling of Activities and delivery of Activities. The structure is not
final at all - and we have logic split across services which isn't ideal - but
thsi will do for now as we play around and discover the structure through
building.
With TypeScript, when creating mock instances, it's preferable to maintain a
reference to the params, even if they're not used. This allows us to have
unused variables as long as they're prefixed with an underscore.
ref https://linear.app/tryghost/issue/MOM-73
We've made it easier to use by adding defaults for required header, as well as
adding support for signing POST requests.
ref https://linear.app/tryghost/issue/MOM-73
We need to add body parsing middleware here, so that NestJS has access to it.
We also attach the rawBody which is used to validate the HTTP Signatures
closes https://linear.app/tryghost/issue/MOM-80
- updated internal linking search results items
- removed visibility text from meta data
- added additional icon for paid/specific tier visibility
- added titles to icons
- bumped `@tryghost/koenig-lexical` to include support for meta icon titles
- bumped other Koenig packages due to sub-dependency updates
ref DES-228
This PR updates messaging and error handling in order to make Ghost calmer and friendlier. High level summary of the changes:
- Removed all onBlur validation in Settings -> now it’s possible to just click around without being warned to fill mandatory fields
- Removed lot of technical errors like `ValidationError: Validation (isEmpty) failed for locale`
- Completely removed the red background toast notifications, it was aggressive and raw esp. on the top
- Removed some unnecessary notifications (e.g. when removing a webhook, the removal already communicates the result)
- Now we show field errors on submitting forms, and in case of an error we show a “Retry” button in Settings too. This allowed to remove a lot of unnecessary error messages, like the big error message on the top, plus it’s consistent with the patterns outside Settings.
- Notification style is white now with filled color icons which makes everything much calmer and more refined.
- Removes redundant copy (e.g. "successful(ly)") from notifications
---------
Co-authored-by: Sodbileg Gansukh <sodbileg.gansukh@gmail.com>
ref https://linear.app/tryghost/issue/CFR-27
- updated packages to include performance improvement for NQL filter
strings including multiple neq filters for the same resource
- bumped `bookshelf-plugins`
- bumped NQL versions
We identified a performance fix that allows us to combine not equal
(neq) filters for the same resource in a logically-equivalent way that
also has far more performant resulting SQL.
We're effectively automatically combining strings like
'tag:-tag1+tag:-tag2` into 'tag:-[tag1,tag2]'.
- it appears as though we only accept `err` when it's in the constructor
of the IncorrectUsageError, so in its current form, it is ignored
- this commit performs a minor refactor to switch to constructing a new
IncorrectUsageError and then throwing it
- detected by tsserver complaining about the `err` property not existing
on the error
Data generator uses CSV imports for a massive speed increase, but
can't be used in some environments where SQL admin isn't
available. This allows us to set a flag to use the original
insert-based importer.
fix https://linear.app/tryghost/issue/SLO-104/cannot-read-properties-of-undefined-reading-0-an-unexpected-error
- if the request body didn't contain the correct keys, it'd just HTTP
500 out of there
- this adds some optional chaining so we end up with undefined if
anything isn't as expected, and the following if-statement does the
rest of the check for us
- this also adds a breaking test (the first E2E test for authentication, yay!)
closes https://linear.app/tryghost/issue/MOM-80
- bumps @tryghost/koenig-lexical to add support for search result metadata in internal links as well as some improvements to the internal linking UI/UX
- updates search service to fetch and expose additional `visibility` and `published_at` fields for post/page resources
- updates `searchLinks` method passed to editor to decorate the search results with appropriate meta text and icon based on publish date, post visibility and member settings
closes https://linear.app/tryghost/issue/MOM-106
- the search results can hide any matching authors/tags due to them appearing after matching posts which is typically a longer list that needs scrolling through
- changed the order to list matched authors and tags before posts, this matches the behaviour in our front-end search
refs https://docs.sentry.io/platforms/javascript/configuration/filtering/#using--1%20
- this simplifies our logic to determine whether we should send events
by moving the code to `beforeSend`
- `errorHandler` is going away in Sentry v8 so this results in a shorter
diff in the future
- the logic should be the same, always send non-Ghost errors, and only
send HTTP 500 Ghost errors
- due to the structure of our API controllers, the docName and methods
are under the same structure
- this code loops over the keys of the controller and forms the method
map
- however, it currently also loops over every character of the docName,
so the resulting map contains a weird structure of chars
- we don't need the docName for this, so we can just exclude it from the
keys
- this doesn't change any functionality
fix https://linear.app/tryghost/issue/SLO-101/http-500-with-invalid-multipart-data
- previously, busboy would error out if we supplied a body that was
invalid (such as an empty FormData)
- we would then return a HTTP 500 to the user, which causes all manner
of problems
- now we catch errors from busboy and return a nice BadRequestError
fix https://linear.app/tryghost/issue/SLO-85/fix-http-500-on-contentposts
- in the event we give the incorrect format in a filter, MySQL will
throw an error and we'll throw a HTTP 500 error
- we can capture this error and return a more useful error to the user
- ideally we'd do this in a validation step before attempting the query,
but parsing this out of NQL and detecting which columns are DATETIME
could be quite tricky
- this updates a bunch of places where we're just using Object to cheat
the system
- doing this means editor autocomplete and basic type checking is better
because we now have proper types in place
- functionality should not change, these are just comments
closes https://linear.app/tryghost/issue/MOM-101
- we were mapping over the grouped search results which meant we still got a group even if it's options/items list was empty after filtering for published
closes https://linear.app/tryghost/issue/MOM-103
- the `yield waitForProperty(...)` call that was supposed to return once the content refresh occurred never reached a valid state so the first search query (or any later query) where a content refresh occurred would never resolve causing search to look like it had stalled
- switched to waiting for the last running task to resolve instead which does the same as the previous code intended
- exported the `getPosts` request handler function so in mirage config so we can re-use it with different timing on a per-case basis
fix https://linear.app/tryghost/issue/SLO-87/cannot-read-properties-of-undefined-reading-createimpl-an-unexpected
refs https://github.com/jsdom/jsdom/issues/3709
- in the event we are given some HTML to parse, and that fails, we
currently return a HTTP 500 because it's unhandled
- the instance we saw was due to `<constructor>` crashing jsdom, we've
opened an issue for that
- in terms of handling the error gracefully, we can surround the code
in a try-catch and return a more suitable error. I've gone for a
ValidationError for now - you could debate whether a different one is
more appropriate
- also added Sentry error capturing so we're not blind to these,
ultimately we should make sure the parser can handle all
user-submitted data
closes https://linear.app/tryghost/issue/MOM-97
The 30s search content expiry didn't really make sense and caused unnecessary delays and server load now that search will be more widely used within the editor.
- replaced concept of time-based expiry with explicit expiry
- content still fetched on query if not already loaded or marked as stale
- added `.expireContent()` method on search service to allow explicit expiry
- updated editor to pre-fetch search content when not already loaded or marked as stale
- removes delay when first using internal linking search inside the editor
- updated post model to expire search content on save
- expires on published post save or delete
- expires on publish and unpublish
- updated tag model to expire content on create/save/delete
- only expires when name or url is changed
- updated user model to expire on save/delete
- only expires when name or url is changed
- does not handle creation because that's done server-side via invites
- this adds a simple set of types to the @tryghost/api-framework
package that should describe all of the keys available on a
controller, and then rolls it out to all API controllers
- unfortunately, due to https://github.com/microsoft/TypeScript/issues/47107, we have
to split apart `module.exports` into a variable assignment in order for type-checking
to be done
- the main benefit of this is that `frame` is now typed, and editors understand what keys
are available, so intellisense works properly
- `statusCode` should be a number, but we were passing a string
- this doesn't really affect anything, but tsserver was flagging it up
as the wrong type
- we should pass it as `err` and not `error`
- this probably slipped in because the catch parameter is called
`error`, so I've updated that and fixed the references
- this helps tsserver figure out what the type of things is around our
codebase
- nothing crazy, mostly Express types for the middleware, application and router levels
closes https://linear.app/tryghost/issue/MOM-103
- the `yield waitForProperty(...)` call that was supposed to return once the content refresh occurred never reached a valid state so the first search query (or any later query) where a content refresh occurred would never resolve causing search to look like it had stalled
- switched to waiting for the last running task to resolve instead which does the same as the previous code intended
- exported the `getPosts` request handler function so in mirage config so we can re-use it with different timing on a per-case basis
closes https://linear.app/tryghost/issue/MOM-101
- we were mapping over the grouped search results which meant we still got a group even if it's options/items list was empty after filtering for published
fix https://linear.app/tryghost/issue/SLO-95/unexpected-end-of-multipart-data-for-broken-image-upload-request
- in the event the client sends an invalid body to the image or media
upload endpoints, Dicer will throw an error if the boundary data is
malformed
- previously, we've just been bubbling that up as an InternalServerError
and that results in an HTTP 500
- we can capture errors produced by dicer and return a handled
BadRequestError, as it's the client's fault
- also includes breaking tests
fix https://linear.app/tryghost/issue/SLO-94/unexpected-field-when-given-broken-image-upload-request
- in the event the body of an image or media upload request is malformed
(broken metadata / blob or something), we get a MulterError and this
bubbles up as an InternalServerError and spits out a HTTP 500
- we can capture this and return a BadRequestError, as it's the client's
fault for not providing the correct body
- this implements that and adds breaking tests
fix https://linear.app/tryghost/issue/SLO-96/invalid-version-must-be-a-string-got-type-object-an-unexpected-error
- in the event that a non-semver Accept-Version header is given, the
current code will throw an error because the semver lib can't compare null
against a valid version
- the error in question is `Must be a string. Got type "object"`
- to fix this, we can just detect a null and early return with a
BadRequestError
- also adds a breaking test
ref https://linear.app/tryghost/issue/MOM-72
This module handles signing and validating HTTP signatures, which is a core
part of interfacing with ActivityPub enabled servers.
ref
https://linear.app/tryghost/issue/ENG-902/add-an-optional-timeout-in-the-redis-cache-adapter-in-case-redis
- Added an optional timeout parameter to AdapterCacheRedis, so that the
`get(key)` method will return `null` after the timeout if it hasn't
received a response from Redis
- When load testing the `LinkRedirectRepository` with the Redis cache
enabled on staging, we noticed that for some reason Redis stopped
responding to commands for ~30 seconds.
- The `LinkRedirectRepository` was waiting for the Redis cache to
respond and resulted in a drastic increase in response times for link
redirects
- This change will allow us to set a timeout on the `get(key)` method,
so that if Redis doesn't respond within the timeout, the method will
return `null` as if it were a cache miss.
- Then the `LinkRedirectRepository` will fall back to the database and
return the link redirect from the database instead of waiting
indefinitely for Redis to respond
