Commit Graph

32783 Commits

Author SHA1 Message Date
Simon Backx
4e361a65fa Added webmention delete test and cleaned up webmention tests
refs https://github.com/TryGhost/Team/issues/2553

- Added webmention delete test
- Fixed dangling promise in test
- Fixed missing .persist in tests
2023-02-13 17:15:34 +01:00
Daniel Lockyer
4bca725215
Added basic unit test for cache adapter module
- this helps improve our testing capabilities of the new cache adapters
2023-02-13 17:10:42 +01:00
Kevin Ansfield
247d0f1f92
Removed editor.url from config.development.json
no issue

- local development Ghost installs should not default to a locally served `koenig-lexical.umd.js` file but should use the default CDN version
- a locally served file should be set in `config.local.json` when you want to develop against an unreleased version
2023-02-13 15:10:49 +00:00
Kevin Ansfield
440d4c41f9
Fixed media thumbnail uploads in koenig-lexical editor
no issue

- added `requestMethod` to `fileTypes` map
- added pass-through of `formData` options to `upload(file, options)` so `url` property can be passed to map uploaded image file to media file
- fixed `resourceName` for media thumbnail uploads
2023-02-13 15:00:29 +00:00
Simon Backx
0e8d13bdde 🐛 Fixed email replacements without fallback
fixes https://github.com/TryGhost/Team/issues/2557

When a member doen't have a name, and the first_name replacement doesn't have a fallback, we did show %recipient.first_name% instead of an empty string.
2023-02-13 15:58:40 +01:00
Aileen Booker
6f0d1b0ff9
Added milestone email service behind alpha flag (#16241)
refs
https://www.notion.so/ghost/Marketing-Milestone-email-campaigns-1d2c9dee3cfa4029863edb16092ad5c4

Added milestone email service behind a flag. The service will currently
run on boot and is meant to be scheduled soon, which should happen in
the next step. For now it's protected behind the alpha flag.
2023-02-13 16:29:01 +02:00
Simon Backx
48f9485f46
🐛 Fixed storing email failures with an empty message (#16260)
no issue

- When we receive an email failure with an empty message, the saving of
the model would fail because of schema validation that requires strings
to be non-empty.
- This adds more logging to the email analytics service to help debug
future issues
- Performance improvement to storing delivered, opened and failed emails
by replacing COALESCE with WHERE X IS NULL (tested and should give a
decent performance boost locally).
2023-02-13 15:25:36 +01:00
Kevin Ansfield
fb882e2e2a
Updated koenig-lexical upload function to handle multiple file types
refs https://github.com/TryGhost/Koenig/pull/491

- `<KoenigComposer>` now takes a `fileUpoader` object in place of `imageUploadFunction`
- updated the upload functions in `koenig-lexical-editor.js` to match expected patterns, handle multiple files and file types, and return expected upload progress, result, and error details
2023-02-13 13:38:35 +00:00
Sam Lord
9e42d1356a Updated avatar styles for webmentions
no issue

This makes the style for generated webmentions much better.
2023-02-13 13:16:24 +00:00
Fabien "egg" O'Carroll
90b7a3f4d0 Added verified member and verify method to Mention entity
closes https://github.com/TryGhost/Team/issues/2548

Rather than use a setter here we've used a verify method which takes the HTML
string and naively validates that the target URL is present. This is so that the
logic of verification is encapsulated in the Mention, and should mean that
erroneous verification doesn't happen.

We could consider down the line that the verify method fetches the content
itself, but if we're to do that we should pass in `got` as a param, so that it's
possible to stub in tests.

One thing to think about when it comes time to making this as performant as
possible is doing a single fetch of the source document and using that for
verification and metadata extraction. At that point we should probably
consolidate both of those operations, either moving the metadata extraction into
the Mention entity (passing in any necessary deps) OR we move the verification
out to the same layer as metadata extraction.
2023-02-13 19:23:34 +07:00
Fabien "egg" O'Carroll
bf6bcdbff3 Added e2e tests for updating Mentions
closes https://github.com/TryGhost/Team/issues/2552

We send a Webmention for the same URL twice, but change the contents
of the source document, and we check that the source metadata is
updated appropriately.

We should consider extending all of these tests to include featured
images and logos etc...
2023-02-13 19:09:31 +07:00
Simon Backx
77032262c4
🐛 Fixed subscriptions visible as "Active" within Ghost Admin (#16255)
fixes https://github.com/TryGhost/Team/issues/2542 
fixes https://github.com/TryGhost/Team/issues/2543 
fixes https://github.com/TryGhost/Team/issues/2544

- Hides incomplete subscriptions
- Shows Past Due subscriptions
- Fixed UI issues with 3+ subscriptions
- Fixed missing complimentary subscription when one subscription was
incomplete/inactive
- Fixed sending a paid subscription started email for incomplete
subscriptions. This change also required us to actually send the email
when the incomplete subscription eventually becomes active. So the
introduction of a new `SubscriptionActivatedEvent` made sense/was
required (because sending a SubscriptionCreatedEvent again would cause
other issues).
2023-02-13 13:07:53 +01:00
Daniel Lockyer
87f207a3ea
Updated lockfile 2023-02-13 12:28:47 +01:00
Naz
7c18263227 Extracted adapter-cache-redis from core codebase
refs https://github.com/TryGhost/Toolbox/issues/515

- This implementation allows to use Redis cluster as a caching adapter. The cache adapter can be configured through same adapter configuration interface as others. It accepts following config values:
  - "ttl" - time in SECONDS for stored entity to live in Redis cache
  - "keyPrefix" - special cache key prefix to use with stored entities
  - "host" / "port" / "password" / "clusterConfig" - Redis instance specific configs
- Set test coverage to non-standard 75% because the code mostly consists of the glue code that would require unnecessary Redis server mocking and would be a bad ROI. This module has been used in production for a long time already, so can be considered pretty stable.
2023-02-13 19:01:43 +08:00
Naz
b3252f956f Added adapter-cache-redis package
refs https://github.com/TryGhost/Toolbox/issues/515

- Redis-based caches can be used on hosted-environments to store information with high memory impact - when in-memory caches would be too impractical to use
- This is a placeholder package for a cluster-aware Redis cache implementation compatible with Ghost's cache adapter interface (a41d351f16/packages/adapter-base-cache)
2023-02-13 19:01:43 +08:00
Naz
1c12d789f9 Added async cache support to Tags repository
refs https://github.com/TryGhost/Toolbox/issues/515

- The caches relying on external storage like e.g.: Redis, the get/set operations are usually async. The tags repository should be working with these as caching is expected to be non-in-memory for these data.
2023-02-13 19:01:43 +08:00
Naz
9d4ea7b8c9 Moved cache initialization behind the hostSetting
refs https://github.com/TryGhost/Toolbox/issues/515

- By moving the cache initialization behind the hostSettings configuration we can limit the experimental feature to only hosted environments with special capacities
- An example configuration to enable Tags caching looks like this:
```
    "hostSettings": {
        "tagsPublicCache": {
            "enabled": true
        },
```
In addition to have the caching backed by a Redis backend or even InMemoryTTL cache the site configuration should include a cache adapter configuration like this:
```
    "adapters": {
        "cache": {
            "active": "Memory",
            "tagsPublic": {
                "adapter": "TTL",
                "ttl": 60000 // 60 * 1000 minute
            },
```
2023-02-13 19:01:43 +08:00
Naz
5216220541 Public tags repository caching
refs https://github.com/TryGhost/Toolbox/issues/515

- There are a lot of repeated cacheable tag-related queries coming from
{{get}} helpers in themes that can be cached.
- Having a repository layer deal with very specific type of query allows
to add extra functionality, like caching, on top of the database query
- This commit is wiring code that addds a default in-memory cache to
all db queries. Note, it lasts forever and has no "reset" listeners. The
production cache is mean to have a short time-to-live (TTL) - removes a need
to keep the cache always fresh.
- Kept the cache key shortened. Without a "context" and any other non-model options the cache-key can store more variations of queries. For example, there is no member-specific or integration-specific query results, so having those in the cache key would only partition the cache and use up more memory.
2023-02-13 19:01:43 +08:00
renovate[bot]
b48262b5d8 Update sentry-javascript monorepo to v7.37.1 2023-02-13 09:22:57 +01:00
renovate[bot]
ccfe6dd638 Update dependency minimist to v1.2.8 2023-02-13 09:21:59 +01:00
Fabien 'egg' O'Carroll
57695d03df
Fixed rate limit test (#16258)
closes https://github.com/TryGhost/Team/issues/2547

Changed the configuration for testing to be a bit more strict, by slowing down the amount of requests it can handle to give CI enough time to kick in the rate limiter. Before this, CI simply wasn't hitting the API fast enough to trigger the rate limiter.

Co-authored-by: Ronald Langeveld <hi@ronaldlangeveld.com>
2023-02-13 15:16:56 +08:00
renovate[bot]
2c9fb2ad5e
Update dependency ember-template-lint to v5.5.0 2023-02-10 11:43:30 +00:00
Kevin Ansfield
d6b8ff124b
Updated <KoenigComposer> props to use generic cardConfig object
refs aed55d415d

- `unsplashConfig` has changed to a generic `cardConfig` object
2023-02-10 11:13:57 +00:00
Ronald Langeveld
b37aff52ca Mention alert toggle only visible to admins
refs https://ghost.slack.com/archives/C02G9E68C/p1675916876727609

- Added the mentions alert toggle to only be visible to owners and
  admins, since it will not have any effect for non-admins staff users.
2023-02-10 17:24:02 +08:00
Fabien "egg" O'Carroll
3df19235b0 Added support for updating Mentions when Webmentions are resent
refs https://github.com/TryGhost/Team/issues/2535

At the moment we only update the metadata of the Webmention source, so
that we can capture update titles, excerpts, etc... when a post is updated.
2023-02-10 11:43:16 +07:00
Fabien "egg" O'Carroll
78bebc6279 Refactored the assignment of Mention metadata
refs https://github.com/TryGhost/Team/issues/2535

Moving this into a separate method allows us to set the metadata externally from
the Mention entity and keep all of the validation, without having duplicate code
2023-02-10 11:43:16 +07:00
renovate[bot]
b271d5a53a Update metascraper to v5.33.8 2023-02-10 02:02:53 +00:00
renovate[bot]
3a11525ad7
Update dependency mysql2 to v3.1.2 2023-02-09 20:10:27 +00:00
Daniel Lockyer
49e9be4368 Reconfigured clean-expired-comped job to execute during morning UTC
refs https://ghost.slack.com/archives/C04MSE4MKJT/p1675948815531779

- running at a fixed hh:mm every day means a platform with a large number
  of Ghost sites will get hammered with DB requests when they all start
  up
- this reconfigures the cron to run at a random minute and second
  between 0am and 5am, which gives a 6 hour window
2023-02-09 15:58:43 +01:00
John O'Nolan
5b4abea56d
Fixed broken link 2023-02-09 10:53:53 -04:00
Ronald Langeveld
4096571b8d Commented out flaky webmention test
refs https://ghost.slack.com/archives/C02G9E68C/p1675934676146349

- Commented out the rate limiter test as it proves to be flaky on CI,
  till we find a better way of implementing it.
2023-02-09 21:35:15 +08:00
Fabien 'egg' O'Carroll
4e8af72305
Added support for deleting mentions (#16238)
refs https://github.com/TryGhost/Team/issues/2534

As we're using soft deletes for mentions we need to store the `deleted` column
as well as enforce a `'deleted:false'` filter on the bookshelf model. 

We've also implemented the handling for deleting mentions. Where we remove a
mention anytime we receive and update from or to a page which no longer exists.

Co-authored-by: Steve Larson <9larsons@gmail.com>
2023-02-09 17:29:13 +07:00
Steve Larson
d20696805f
Added mentions deleted column (#16251)
refs https://github.com/TryGhost/Team/issues/2534

This is so that we can support soft deletes for Mentions.
We need to add the defaults to the model so that write continue to work.

Co-authored-by: Fabien "egg" O'Carroll <fabien@allou.is>
2023-02-09 16:19:07 +07:00
Daniel Lockyer
d750612508
Merged v5.33.8 into main
v5.33.8
2023-02-09 10:08:43 +01:00
Ghost CI
83bf4ccb53 v5.33.8 2023-02-09 09:06:34 +00:00
Ronald Langeveld
ce567b9816
Added Mentions email notification settings (#16245)
closes https://github.com/TryGhost/Team/issues/2526

- Mention emails can now be toggled inside staff user' profiles, if they
have the webmention flag enabled on their Ghost site.
- Removed the flag dedicated to webmention email notifications and is
now handled by the `webmention` flag.
- Does not send email notification if `webmention` flag is not enabled.
- Updated tests.

---------

Co-authored-by: Fabien "egg" O'Carroll <fabien@allou.is>
2023-02-09 17:03:03 +08:00
Simon Backx
da97519082 Moved email event fetching to main thread (#16231)
refs https://github.com/TryGhost/Team/issues/2482

This moves the processing of Mailgun events to the main thread. By using a simple approach where we emit a start event on the worker thread (via the job manager) and listen for it on the main thread. This is needed because for now the job manager doesn't support scheduling periodic jobs on the main thread (not offloaded).

Apart from that, the email processor now uses the email event storage directy instead of emitting events (it is still emitting event for now). This makes sure we await for the event to be processed before continuing with the next event.
2023-02-09 09:44:44 +01:00
Simon Backx
ea2c69565f
Moved email event fetching to main thread (#16231) 2023-02-09 09:36:39 +01:00
renovate[bot]
fd79ca3f5a Update dependency html-validate to v7.13.2 2023-02-09 09:30:14 +01:00
Simon Backx
b7be02f057
🐛 Fixed throwing errors during link redirects
no issue

Errors weren't correctly passed on to Express in the middleware.
2023-02-09 09:26:45 +01:00
Kevin Ansfield
f7aefa2feb
🐛 Fixed default email recipients always being "everyone" rather than matching post visibility (#16247)
refs https://github.com/TryGhost/Ghost/issues/16125

The previous fix for incorrect recipient details being shown when
re-sending a failed email introduced another bug that prevented the
"Match post visibility" default recipients setting from working.

- the server always sets `post.emailSegment` to `'all'` for new posts so
the publish flow recipient filter logic that checked for
`post.emailSegment` being present always defaulted to `'all'` rather
than falling back to the selected default recipients setting
- when a post has been published but the email failed it will have its
`newsletter` value set so we can use that as a check for using the
`post.emailSegment` value in place of the default recipients setting
2023-02-09 09:24:50 +01:00
Ronald Langeveld
30dc2a8228
Added mention_notifications column (#16242)
refs https://github.com/TryGhost/Team/issues/2526

- created a migration for a new boolean column in users that would
determine if the staff user gets an email when the publication receive a
new mention.
2023-02-09 16:15:54 +08:00
Ronald Langeveld
9fc13cfe65
Implemented mentions rate limiter (#16223)
closes https://github.com/TryGhost/Team/issues/2419

- adds a rate limiter implementation to the mentions receiving
endpoint.
- Current configuration is `{"minWait": 10,
             "maxWait": 100,
             "lifetime": 1000,
             "freeRetries": 100}` which is still very open and almost unrestricted. 
- currently makes use of database storage to track the limits, but can be relatively easily swapped out to something eg Redis should we find this endpoint getting hit too often and maliciously.
2023-02-09 14:57:48 +08:00
Naz
93a10d8f4f Optimized queries for tiers (aka products)
refs https://github.com/TryGhost/Toolbox/issues/515
refs dd4d6aeae5

- The `productRepository.list` call produced 5 db queries and a transaction wrapping this call.
- Transaction is not needed in this situation as there are no possible writes in the meantime (transaction wrapping code was put in there through refed commit to guard against failing Stripe API calls, which are no longer involved when calling the list method)
-  The `limit: 'all'` makes sure all product entries are fetched AND removes an extra aggregation query called over stripe_prices join
- The 'monthlyPrice' and 'yearlyPrice' relations are not needed because this data is not used in downstream code - only slug and type are used for visiblity/content gating  (ref. 1 3b6759ca6d/ghost/core/core/server/services/members/content-gating.js (L44-L55), ref. 2 3b6759ca6d/ghost/core/core/server/api/endpoints/utils/serializers/output/mappers/posts.js (L39-L54))
2023-02-09 13:07:57 +08:00
Steve Larson
6c97edec25
moved sending mentions to job (#16234)
refs TryGhost/Team#2523
-implemented jobsService within mention sending service
-updated tests so we don't need to sleep
2023-02-08 16:29:12 -06:00
Sam Lord
0d7944861c Data generator: Swap order of free and premium newsletters
no issue
Free and premium newsletters were the other way around in the demo-data. This was a good opportunity to stop the email table importer from relying on the newsletter name, and use the order alone.
2023-02-08 22:12:19 +00:00
Kevin Ansfield
923a9a1cd6
🐛 Fixed default email recipients always being "everyone" rather than matching post visibility (#16247)
refs https://github.com/TryGhost/Ghost/issues/16125

The previous fix for incorrect recipient details being shown when
re-sending a failed email introduced another bug that prevented the
"Match post visibility" default recipients setting from working.

- the server always sets `post.emailSegment` to `'all'` for new posts so
the publish flow recipient filter logic that checked for
`post.emailSegment` being present always defaulted to `'all'` rather
than falling back to the selected default recipients setting
- when a post has been published but the email failed it will have its
`newsletter` value set so we can use that as a check for using the
`post.emailSegment` value in place of the default recipients setting
2023-02-08 11:41:48 +00:00
Simon Backx
4436a1cd18 🐛 Fixed throwing errors during link redirects
no issue

Errors weren't correctly passed on to Express in the middleware.
2023-02-08 11:50:22 +01:00
Aileen Booker
3b6759ca6d
Added initial basic milestone emails in-memory repository (#16216)
refs
https://www.notion.so/ghost/Marketing-Milestone-email-campaigns-1d2c9dee3cfa4029863edb16092ad5c4

This adds a milestone entity and in-memory repository in a new
`milestone-emails` package. This also adds a first initial definition of
milestones and their types which is held in the default config to avoid
DB changes when, e. g. values change.

This should get everything in place to begin with the service
implementation.
2023-02-07 12:47:35 +02:00
Ghost CI
c518446c17 Merged v5.33.7 into main 2023-02-07 10:32:47 +00:00