Commit Graph

215 Commits

Author SHA1 Message Date
Fabien "egg" O'Carroll
e5d31e2900 Added resource data to Mentions Admin API
refs https://github.com/TryGhost/Team/issues/2503

This is in the MentionController atm as it's considered a presentation
concern. We might want to consider moving this into the MentionsAPI in
future so that we can simplify the controller and even remove it
completely in favour of putting the data-mapping in the endpoint definition.
2023-02-02 16:13:49 +07:00
Ronald Langeveld
bef3516f72 Added e2e test for filtering members by tier id
no issue

- Added a test for filtering members by tier id since we now have a new
  way of filtering members in the admin.
2023-02-02 15:10:19 +08:00
Simon Backx
8e66edee2b
🐛 Fixed storing original files for images (#16117)
fixes https://github.com/TryGhost/Team/issues/481

This change fixes an issue when multiple images with the same name are
uploaded in parallel. The current system does not guarantee that the
original filename is stored under NAME+`_o`, because the upload for the
original file and the resized file are happening in parallel.

Solution:
- Wait for the storage of the resized image (= the image without the _o
suffix) before storing the original file.
- When that is stored, use the generated file name of the stored image
to generate the filename with the _o suffix. This way, it will always
match and we don't risk both files to have a different number suffix.
We'll also set the `targetDir` argument when saving the file, to avoid
storing the original file in a different directory (when uploading a
file around midnight both files could be stored in 2023/01 and 2023/02).

Some extra optimisations needed with this fix:
- Previously when uploading image.jpg, while it already exists, it would
store two filenames on e.g., `image-3.jpg` and `image_o-3.jpg`. Note the
weird positioning of `_o`. This probably caused bugs when uploading
files named `image-3.jpg`, which would store the original in
`image-3_o.jpg`, but this original would never be used by the
handle-image-sizes middleware (it would look for `image_o-3.jpg`). This
fix would solve this weird naming issue, and make it more consistent.
But we need to make sure our middlewares (including handle-image-sizes)
will be able to handle both file locations to remain compatible with the
old format. This isn't additional work, because it would fix the old bug
too.
- Prevent uploading files that end with `_o`, e.g. by automatically
stripping that suffix from uploaded files. To prevent collisions.

Advantage(s):
- We keep the original file name, which is better for SEO.
- No changes required to the storage adapters.

Downside(s):
- The storage of both files will nog happen parallel any longer. But I
expect the performance implications to be minimal.
- Changes to the routing: normalize middleware is removed
2023-01-30 16:40:50 +01:00
Simon Backx
762238c7e7 Cleaned DomainEvents import 2023-01-30 15:46:31 +01:00
Simon Backx
60670c21ce Fixed webmentions test awaiting events
no issue

The test was not waiting for dispatched events.
2023-01-30 15:45:16 +01:00
Steve Larson
477295a262
added mentions admin browse api e2e tests (#16177)
refs TryGhost/Team#2468
-added simple browse api tests for webmentions
-need to add user access tests and possibly site access (private sites)
2023-01-30 07:59:00 -06:00
Simon Backx
8f8ca481a6
Fixed configUtils and adapter cache issues in E2E tests (#16167)
no issue

There are a couple of issues with resetting the Ghost instance between
E2E test files:

These issues came to the surface because of new tests written in
https://github.com/TryGhost/Ghost/pull/16117

**1. configUtils.restore does not work correctly**
`config.reset()` is a callback based method. On top of that, it doesn't
really work reliably (https://github.com/indexzero/nconf/issues/93)

What kinda happens, is that you first call `config.reset` but
immediately after you correcty reset the config using the `config.set`
calls afterwards. But since `config.reset` is async, that reset will
happen after all those sets, and the end result is that it isn't reset
correctly.

This mainly caused issues in the new updated images tests, which were
updating the config `imageOptimization.contentImageSizes`, which is a
deeply nested config value. Maybe some references to objects are reused
in nconf that cause this issue?

Wrapping `config.reset()` in a promise does fix the issue.

**2. Adapters cache not reset between tests**
At the start of each test, we set `paths:contentPath` to a nice new
temporary directory. But if a previous test already requests a
localStorage adapter, that adapter would have been created and in the
constructor `paths:contentPath` would have been passed. That same
instance will be reused in the next test run. So it won't read the new
config again. To fix this, we need to reset the adapter instances
between E2E tests.

How was this visible? Test uploads were stored in the actual git
repository, and not in a temporary directory. When writing the new image
upload tests, this also resulted in unreliable test runs because some
image names were already taken (from previous test runs).

**3. Old 2E2 test Ghost server not stopped**
Sometimes we still need access to the frontend test server using
`getAgentsWithFrontend`. But that does start a new Ghost server which is
actually listening for HTTP traffic. This could result in a fatal error
in tests because the port is already in use. The issue is that old E2E
tests also start a HTTP server, but they don't stop the server. When you
used the old `startGhost` util, it would check if a server was already
running and stop it first. The new `getAgentsWithFrontend` now also has
the same functionality to fix that issue.
2023-01-30 14:06:20 +01:00
Fabien 'egg' O'Carroll
73bddef7c5
Used job queue for processing incoming Webmentions
refs https://github.com/TryGhost/Team/issues/2419

We use a job queue to ensure that webmentions can be processed outside of
the request/response cycle, but still finish executing if the processed is closed.

With this we're able to update the e2e tests to await the processing of the mention
rather than sleepign for arbitrary lengths of time, and we've reintroduced the tests
removed previously

- aa14207b69
- 48e9393159
2023-01-30 18:11:30 +07:00
Naz
d3ff4348c8
Fixed test name as content version header is always present
refs 9ba251238a

- The test name should have been updated along with referenced change.
2023-01-26 19:20:15 +08:00
Ronald Langeveld
dd74f42376
Added mentions email notifications (#16170)
closes https://github.com/TryGhost/Team/issues/2429

- sends email notifications to staff users when their site receives a Webmention.
- currently behind a flag, that can be toggled in the labs settings.
2023-01-25 21:10:29 +08:00
Daniel Lockyer
977c95157a
Removed flaky test
refs https://ghost.slack.com/archives/C02G9E68C/p1674558376712339

- this test is also flaky and causing random failures
2023-01-25 11:41:59 +01:00
Fabien "egg" O'Carroll
48e9393159 Removed flaky test
This test is failing because the `sleep` isn't long enough. Removing this test
until we've refactored to use the jobs service, at which point we can remove the
sleep and wait for the job to be complete.
2023-01-24 18:50:04 +07:00
Daniel Lockyer
56b0e65583
Revert "Added email snapshots for API versioning tests (#16139)"
- this reverts commit 85051199e3
- the tests here rely on dynamic content (the Ghost version number)
2023-01-20 18:05:36 +01:00
Fabien 'egg' O'Carroll
33ebe971f8
Added BookshelfMentionRepository implementation (#16156)
fixes https://github.com/TryGhost/Team/issues/2418

This stores the received webmentions in the database.

Co-authored-by: Simon Backx <simon@ghost.org>
2023-01-20 14:32:50 +01:00
Rishabh Garg
c7230f1858
Improved email stability (#16159)
- bumps new email flows to GA improving overall email stability
2023-01-20 18:55:20 +05:30
Simon Backx
e879406659
Added outbound link tagging setting (#16146)
fixes https://github.com/TryGhost/Team/issues/2432
    
Adds outbound_link_tagging setting (enabled by default and behind
feature flag). If the feature flag is enabled, and the setting is
disabled, we won't add ?ref to links in emails.
    
This includes new E2E tests for email click tracking, which were also
extended to check outbound link tagging (for both MEGA and the new email
stability flow).

Also fixes a test fixture for the comments_enabled setting.
2023-01-20 13:41:36 +01:00
Simon Backx
8a95c62ff1
🐛 Fixed post links being marked as edited when they were not (#16153)
fixes https://github.com/TryGhost/Team/issues/2461

- Ignores 'edited' links when there is only one second differences.
- Make sure we don't set updatedAt when linking a post to a redirect
2023-01-20 13:31:54 +01:00
Simon Backx
6c2af0793c Fixed failing Stripe webhook test
refs acf0baa8c7

Due to the bump in express-test, we now handle string bodies 'properly'. So they now pass all the Express middlewares. In the past this failing test did not really pass by the bodyParser.raw middleware,
so the content-type check on the `bodyParser.raw({type: 'application/json'})` middleware was not executed. Now it is, and the test fails because the content-type header was not set to application/json.
2023-01-18 12:05:55 +01:00
Daniel Lockyer
9ba251238a Added Content-Version header to all API requests
refs https://github.com/TryGhost/Team/issues/2400

- we've deemed it useful to start to return `Content-Version` for all
  API requests, because it becomes useful to know which version of Ghost
  a response has come from in logs
- this should also help us detect Admin<->Ghost API mismatches, which
  was the cause of a bug recently (ref'd issue)
2023-01-18 08:38:07 +01:00
Daniel Lockyer
85051199e3
Added email snapshots for API versioning tests (#16139) 2023-01-18 08:09:28 +01:00
Naz
1f6b2b4d5e
Added email content testing capabilities
refs https://github.com/TryGhost/Toolbox/issues/499

- Outgoing emails have been a weak point of Ghost's stability recently. The concept of "emailMockReceiver" similarly to "webhookMockReceiver", allows to test side-effects like outgoing emails.
- This is a first iteration which should lay groundwork for testing all outgoing emails in the future
- The change adds a new concept of "email mock receiver" which is very similar to how the "webhook mock receiver" works. The email mock receiver exposes two methods to record and verify snapshots:
- matchHTMLSnapshot - records and verifies only the HTML content of the outgoint email
- matchMetadataSnapshot - records and verifies all the non-HTML properties sent along an email content, e.g.: to address, plaintext, subject, etc.
- What's missing is matching content based on dynamic content like dates, links with JWT tokens, etc.
2023-01-17 12:50:49 +08:00
Fabien 'egg' O'Carroll
14f282d640
Improved newsletter delivery rates and email suppression handling
This introduces the new suppressions feature which will automatically
unsubscribe members from newsletters when their email is added to the
suppression list in Mailgun, this is usually due to emails either
permanently bouncing to the address, or the member making a spam
complaint.

Both Members and Admins are able to see that the email has been added to
the list, and Members are be able to request their email be removed from
the list via Portal.

Overall this feature should improve delivery rates of newsletters and
improve the rating of the domain you're sending from.
2023-01-13 15:46:57 +07:00
Ronald Langeveld
0e2d1b3afd
Added redeemed offers filtering for members (#16071)
closes https://github.com/TryGhost/Team/issues/2011

- Gives publishers the ability to filter members based on which offer they used (redeemed) when they subscribed for a paid membership.
- On the offers page, the redemption count number links to a the members page with the filter already applied making it easy to have insight on which members used the offer / coupon.
2023-01-11 20:13:09 +08:00
Simon Backx
26635f192f Added visible theme errors in admin
fixes https://github.com/TryGhost/Team/issues/2393
2023-01-06 15:51:34 +01:00
Simon Backx
7b3712a15b
Added visible theme errors in admin (#16081)
refs https://github.com/TryGhost/Team/issues/2393

- During boot and loading the active theme, we now cache the result of
the gscan validation. Cache configuration can happen in
`adapters.cache.gscan`
- We now also return non-fatal errors when activating or adding a theme.
- When the `themeErrorsNotification` feature flag is on, we fetch the
active theme (which includes the validation information) when loading
admin
- If the currently active theme has errors, we show an error
notification that can open the error modal
- Added a new endpoint: `/ghost/api/admin/themes/active/` that returns
the result of the last gscan validation of the active theme. If no cache
is available, it will run a new gscan validation.
- Added new permissions for the active action/endpoint (author, editor,
administrator)
2023-01-06 13:44:27 +01:00
Fabien "egg" O'Carroll
2d9114450c Fixed email_recipients fixtures for tests
The email_recipient fixtures were using duplicate and mismatched email addresses
rather than having them correctly map to the Members, which is required for testing
email suppressions.
2023-01-05 17:11:06 +07:00
Simon Backx
913ad18b71
Added DomainEvents.allSettled utility method (#16075)
no issue

With the increased usage of DomainEvents, it gets harder to build
reliable tests without having to resort to timeouts. This utility method
allows us to wait for all events to be processed before continuing with
the test.

This change should speed up tests and make them more reliable.

It only adds extra code when running tests and shouldn't impact
production.
2023-01-04 14:30:35 +01:00
Simon Backx
819d0d884c
Improved email verification required checks (#16060)
fixes https://github.com/TryGhost/Team/issues/2366
refs https://ghost.slack.com/archives/C02G9E68C/p1670232405014209

Probem described in issue.

In the old MEGA flow:
- The `email_verification_required` check is now repeated inside the job

In the new email service flow:
- The `email_verification_required` is now checked (didn't happen
before)
- When generating the email batch recipients, we only include members
that were created before the email was created. That way it is
impossible to avoid limit checks by inserting new members between
creating an email and sending an email.
- We don't need to repeat the check inside the job because of the above
changes

Improved handling of large imports:
- When checking `email_verification_required`, we now also check if the
import threshold is reached (a new method is introduced in
vertificationTrigger specifically for this usage). If it is, we start
the verification progress. This is required for long running imports
that only check the verification threshold at the very end.
- This change increases the concurrency of fastq to 3 (refs
https://ghost.slack.com/archives/C02G9E68C/p1670232405014209). So when
running a long import, it is now possible to send emails without having
to wait for the import. Above change makes sure it is not possible to
get around the verification limits.

Refactoring:
- Removed the need to use `updateVerificationTrigger` by making
thresholds getters instead of fixed variables.
- Improved awaiting of members import job in regression test
2023-01-04 11:22:12 +01:00
Simon Backx
789e2c96c0
🐛 Fixed SingleUseTokens being cleared on boot (#15999)
fixes https://github.com/TryGhost/Team/issues/1996

**Issue**
Our Magic links are valid for 24 hours. After first usage, the token
lives for a further 10 minutes, so that in the case of email servers or
clients that "visit" links, the token can still be used.

The implementation of the 10 minute window uses setTimeout, meaning if
the process is interrupted, the 10 minute window is ignored completely,
and the token will continue to live for the remainder of it's 24 hour
validity period. To prevent that, the tokens are cleared on boot at the
moment.

**Solution**

To remove the boot clearing logic, we need to make sure the tokens are
only valid for 10 minutes after first use even during restarts.

This commit adds 3 new fields to the SingleUseToken model:
- updated_at: for storing the last time the token was changed/used). Not
really used atm.
- first_used_at: for storing the first time the token was used
- used_count: for storing the number of times the token has been used

Using these fields:
- A token can only be used 3 times
- A token is only valid for 10 minutes after first use, even if the
server restarts in between
- A token is only valid for 24 hours after creation (not changed)

We now also delete expired tokens in a separate job instead of on boot /
in a timeout.
2023-01-04 09:49:39 +01:00
Elena Baidakova
235446b034
🐛 Fixed tiers order by monthly_price (#16013)
refs TryGhost/Team#2273
- According to documentation (https://ghost.org/docs/content-api/#order)
tiers should be ordered from lowest to highest monthly price.
2023-01-03 12:59:18 +04:00
Simon Backx
f568b35f26 Fixed 2023 snapshot tests
no issue

Updated the snapshot files for the email preview test snapshots that contain 2023 in the email footer.
2023-01-02 11:18:07 +01:00
Simon Backx
0825a2d7f4
🐛 Fixed importing existing member resetting newsletters (#16017)
fixes https://github.com/TryGhost/Team/issues/2386

**Issue:**
- When trying to import a member that already exists, and has
'subscribed' set to 'true' in the CSV, the newsletters the member is
subscribed to are reset to the default newsletters.
- When ediging a member with the API and setting `subscribed` to true,
the same happens.

**Cause:**
A faulty check for the `status` property of a newsletter.

Fixed and added a new E2E test.
2022-12-15 17:45:11 +01:00
Elena Baidakova
9579791185
Resubscribe member to default newsletters after removing from suppression list (#15974)
refs TryGhost/Team#2372
2022-12-13 13:31:50 +04:00
Rishabh
ce45571dc0 Updated email faqs in portal to show sender email address
refs https://github.com/TryGhost/Team/issues/2348

- updates email faqs to show the sender email from newsletter data
2022-12-12 15:09:32 +05:30
Elena Baidakova
62681037f0
Added email_suppression property to response for PUT api/member method (#15946)
closes TryGhost/Team#2359
- Keep the same response structure for `api/member` GET and PUT methods
2022-12-06 13:51:26 +04:00
Naz
879aad263d
Added email verification trigger test
refs https://github.com/TryGhost/Toolbox/issues/476

- The email verification trigger and host settings related bugs have been a cause of bugs in past releases.  The admin client verification source did not have any test coverage in the past.
- The members test suite size is getting out of hand. This test is quite verbose, because of the state it's trying to check.
- In the future we should consider splitting up Member API (and probably other) test suites into smaller pieces.
2022-12-05 18:24:28 +07:00
Naz
e170f293e3
Extracted sleep method to e2e framework module
no issue

- The sleep method has been used in 8 modules reimplementing the same thing over and over again. It's usually a sign of async event processing outside of the request/response loop. It's good to have a single point of implementation for a "hack" like this, so we could track it easier and address the even processing delay in a more optimal way centrally if it ever becomes a bottleneck
2022-12-05 17:26:29 +07:00
Simon Backx
2a3a72e4d5
Added email debug API (#15915)
fixes https://github.com/TryGhost/Team/issues/2346

- Adds email batch browse endpoint
- Adds email recipient failures browse endpoint
- Adds new fixtures and E2E tests for the new API
- Added support for snapshot tests to have 'nullable' types.
2022-12-02 09:44:32 +01:00
Simon Backx
4c166e11df
Added E2E tests for batch sending (#15910)
refs https://github.com/TryGhost/Team/issues/2339

- Includes a new pattern in the job manager that allows us to properly
await jobs.
- Added new convenience mocking methods to stub settings
- Tests the main flows for bulk sending:
    - Sending in multiple batches
    - Sending to multiple segments
    - Handling a failed batch and retrying that batch
- Fixes bug in batch generation (ordering not working)

In a different PR I'll add more detailed tests.
2022-12-01 13:43:49 +01:00
Naz
80decf8303
Fixed invalid query strings in tests
closes https://github.com/TryGhost/Team/issues/2324

- It seemed like the "limit" query parameter did not work properly returning multiple entries from the endpoint. In reality the whole query string was ignored because of an error in the "filter" part of the query ^_^
2022-11-30 13:31:06 +07:00
Elena Baidakova
1b784b5ec5
Added events to Post Analytics page (#15886)
closes TryGhost/Team#2313
- Added Sent event to Post analytics and Members feed. Now post can be
Sent or Received or Bounced.
- Excluded Delivered event from Sent filter on backend.
2022-11-28 17:43:35 +04:00
Naz
22738b1b50 🔒 Disabled editable relations by default
refs https://github.com/TryGhost/Ghost/security/advisories/GHSA-9gh8-wp53-ccc6
refs https://github.com/TryGhost/Toolbox/issues/465

- Bookshelf relations allows us to edit relational records by default, which was used liberally in the codebase.
- Not having a clear track record of editable relations left the model layer prone to triggering unwanted nested saves and created a vulnerability where members were able to edit newsletter settings.
- With explicit editable relations it's easier to keep track of relations having editable access to related records. Makes the relational data modification pattern safer to use too.
- Anyone running 5.x should update to 5.24.1

Credits: Dave McDaniel and other members of [Cisco Talos](https://talosintelligence.com/vulnerability_reports)
2022-11-28 18:39:39 +07:00
Elena Baidakova
8d9d22e5a7
Added member API for removing email from suppression list (#15867)
closes TryGhost/Team#2306
2022-11-23 14:41:00 +04:00
Simon Backx
4b4592630f
Added new email batch sending service (#15865)
fixes https://github.com/TryGhost/Team/issues/2284

New batch sending flow (still WIP). Logs the sent emails instead of actually sending them. Unit tests are coming in later commits.
2022-11-23 11:33:44 +01:00
Rishabh
5780fc2a93 Added new source and source type columns to emails table
refs https://github.com/TryGhost/Team/issues/2280

We are moving away from storing html and plaintext on email and instead will store the email data in source and source_type columns which allows us to store the email in other formats like mobiledoc and lexical. Storing in those formats allows greater flexibility for later html generation

- adds new `source` column that stores `mobiledoc`/`lexical`/`html` data for a newsletter
- adds new `source_type` column that stores one of `mobiledoc`/`lexical`/`html` to identify type of source
2022-11-23 15:04:11 +05:30
Simon Backx
44f189b56a
Added email service package (#15849)
fixes https://github.com/TryGhost/Team/issues/2282

Added a new email service package that is used when the email stability
flag is enabled. Currently not yet implemented so will throw an error
for all entry points (if flag enabled).

Removed usage of `labs.isSet.bind` across the code, because that breaks
the stubbing of labs by `mockManager.mockLabsEnabled` and
`mockManager.mockLabsDisabled`. `flag => labs.isSet(flag)` should be
used instead.

All email depending tests now disable the `emailStability` feature flag
to keep the tests passing + make sure we still run all the tests for the
old flow while the email stability package is being built.
2022-11-21 10:29:53 +01:00
Simon Backx
2220686113 Sped up comments and feedback tests by reusing Ghost app instance
refs https://ghost.slack.com/archives/C02G9E68C/p1668777066462859

Added a duplicate method to the mmebers agent, so we can reuse the same app instance and options, whenever we need multiple agents (each with their own cookies).
2022-11-18 14:58:35 +01:00
Fabien 'egg' O'Carroll
8283de99c8
Wired up EmailSuppressionList to Members Admin API (#15848)
refs https://github.com/TryGhost/Team/issues/2268

The approach of using the service to lead email suppression data as
opposed to bookshelf relations allows us to wire things up without
having implemented the database. The getBulkSuppressionData allows us to
do this without much of a DB performance hit.
2022-11-18 16:28:13 +07:00
Ronald Langeveld
08232bd538
Cleaned up GA Flags (#15762)
closes https://github.com/TryGhost/Team/issues/2126

- Cleaned up the following GA flags: `newsletterPaywall`, `freeTrial`, `compExpiring`, `searchHelper`, `emailAlerts`, `fixNewsletterLinks`.
2022-11-18 16:09:25 +07:00
Ronald Langeveld
e0787b4e83
Added specific newsletter support for bulk unsubscribes (#15742)
closes https://github.com/TryGhost/Team/issues/2013

Added support to bulk unsubscribe a selected (filtered) list on members from specific, selected newsletters.
2022-11-16 14:29:00 +07:00