Commit Graph

924 Commits

Author SHA1 Message Date
Naz
3f78e959b3
Bumped email-mock-receiver
refs https://github.com/TryGhost/Team/issues/2691

- This bump changes the "sentEmailCount" method to a more descriptive "assertSentEmailCount" and adds chaining to this method.
2023-04-06 22:13:17 +02:00
naz
70ae3efd5c
Update email mock receive (#16578)
refs TryGhost/Team#2691

- The bump adds possibility to make email's html/text snapshots with dynamic content. The breaking change here is with separate "matchPlaintextSnapshot" method extracted out of "matchMetadataSnapshot" to handle dynamic content in "text" part of the sent email.
2023-04-06 17:24:23 +02:00
Simon Backx
ba8f082d41
Added awaiting jobs and events by default to all tests (#16505)
no issue

This change waits for domain events and jobs before continuing with the
next test. This prevents issues where background tasks in tests are
executed when the next test is running and the configurations have
changed, causing random error logs and test failures.

It also includes a change in Stripe mocking in one E2E test to make use
of the new StripeMocker instead of custom mocking in each test (also to
reduce error logs).
2023-04-06 09:05:16 +02:00
Chris Raible
16c625a630
Added retries to flaky email test (#16556)
refs TryGhost/Team#2891

- test was flaking frequently enough that we had to remove it — not a perfect fix but figure it's better to enable retries than to completely remove the test
- ran CI 5 times (x 4 environments) and it passed 5 times in a row
2023-04-05 22:01:59 -07:00
Sanne de Vries
92663ea731
Fixed email width and dark mode images in email template (#16566)
Refs https://github.com/TryGhost/Team/issues/2845

---------

Co-authored-by: Fabien "egg" O'Carroll <fabien@allou.is>
Co-authored-by: Simon Backx <simon@ghost.org>
2023-04-05 15:53:51 +02:00
Rishabh Garg
bb21dc47f0
Updated member snapshot for comments tests
refs TryGhost/Team#2888
2023-04-05 00:25:31 +05:30
Rishabh
76ed31464b Fixed snapshots for frontend frontend member data APIs
refs https://github.com/TryGhost/Team/issues/2888
2023-04-05 00:01:55 +05:30
Rishabh
ace4d17b3e Updated unit tests for frontend member data 2023-04-04 23:51:45 +05:30
Chris Raible
e95c531e8b
🐛 Fixed member signup emails being sent with escaped subject line (#16544)
closes TryGhost/Team#2895

- this was caused by the subject line being passed through the i18n
translator, which was escaping the content
- passing in `interpolation: {escapeValue: false}` when retrieving the
value prevents the content from being escaped
- modified a test to ensure the subject line is not escaped
2023-04-04 10:12:28 -07:00
Simon Backx
d0042b550a
Fixed redirecting back to account home after sign in in Portal (#16487)
refs https://github.com/TryGhost/Team/issues/2674

When going to /#/portal/account when not signed in, you are redirected
to the login page. But once signed in, you aren't redirected back to the
account page. This fixes this issue by adding an extra and optional
redirect parameter when requesting a magic token via email.

This new parameter allows to override the default behaviour of using the
Referer HTTP header, which doesn't include the hash/fragment part of the
URL.

The referrer is already restricted to only allow redirects to the site,
not external URLs.
2023-04-04 18:07:37 +02:00
Fabien "egg" O'Carroll
41e8c20d88 Deleted flakey test
refs https://github.com/TryGhost/Team/issues/2891

This test is failing more and more frequently and is being removed with the
intention of reinstating it once it has been fixed
2023-04-04 20:16:44 +07:00
Simon Backx
e13f052af8
Added Portal signup terms settings and migration (#16545)
fixes https://github.com/TryGhost/Team/issues/2885 
fixes https://github.com/TryGhost/Team/issues/2896 
fixes https://github.com/TryGhost/Team/issues/2877

This change adds the new portal_signup_terms_html setting and
portal_signup_checkbox_required setting and the corresponding migration.
2023-04-04 10:04:12 +02:00
Sanne de Vries
57fcfe76d8
Updated images in email template to be work with light/dark mode
refs https://github.com/TryGhost/Team/issues/2845

We needed to update the html out of the cards to include images for light
and dark mode, and then we've used CSS to show/hide them 

Co-authored-by: Fabien "egg" O'Carroll <fabien@allou.is>
2023-04-04 00:32:08 +07:00
Simon Backx
d06e3dca60 Added id column to post analytics export
refs https://github.com/TryGhost/Team/issues/2678

Also fixes snapshots
2023-04-03 14:50:43 +02:00
Simon Backx
bef310cb2a Readded failing test
no issue

Test was failing and removed temporarily.
2023-04-03 11:35:40 +02:00
Simon Backx
33237c4df7 Added newsletter auto border color and dynamic color picker
fixes https://github.com/TryGhost/Team/issues/2879
fixes https://github.com/TryGhost/Team/issues/2880

- Replaced black border color with 'auto' based on background color.
- When a color is 'auto', the color that are visible in the UI (color pciker) will be dynamic based on the background color.
2023-04-03 11:27:57 +02:00
Daniel Lockyer
5cd67f83ea Added i18n labs flag
- this adds the labs flag and scaffolding to wire it into Ghost + Portal
2023-03-30 18:15:57 +02:00
Fabien 'egg' O'Carroll
0f72816951
Newsletter email template customisation (#16523)
Refs https://github.com/TryGhost/Team/issues/2845

Co-authored-by: Sanne de Vries <sannedv@protonmail.com>
2023-03-30 15:33:46 +02:00
Rishabh Garg
dc811af070
Merged mentions UI and email lab flags (#16518)
closes https://github.com/TryGhost/Team/issues/2851

We had two separate flags to manage the Mentions beta, one for showing
UI and other for sending emails. This change combines them both under
the single `webmentions` flag that was previously only used to show the
UI.
2023-03-29 01:12:56 +05:30
Simon Backx
109cdeb492 Fixed exporting post metrics without email
no issue

Bookshelf by default returns an empty model when requesting .related('email') for a post without an email. So we need to be a bit smarter to know if a post has an email or not. This fixed an issue where we always showed 'published and emailed' instead of 'published only'.

Since this change also included some changes to test helpers, it also made some changes to the email service because coverage dropped below 100% as a result of fixing the .related method mocking. Ideally we want to move test test helpers to a seperate package in the future.
2023-03-28 12:41:32 +02:00
Simon Backx
7366f726e5 Removed post access copy in post exporter
no issue
2023-03-28 12:08:00 +02:00
Naz
2dd94d4300
Fixed self-serve api key type
refs https://github.com/TryGhost/Ghost/pull/16461

- The referenced migration had an incorrect 'type' assigned to the self-serve integration's api key. Should have been "admin" instead of "core"
2023-03-28 11:59:17 +02:00
Simon Backx
37397e88b6 Renamed paid_signups to paid_conversions in post metrics export
no issue
2023-03-28 11:43:37 +02:00
Simon Backx
59f94e552a
Added newsletter color columns (#16506)
fixes https://github.com/TryGhost/Team/issues/2825

Adds 3 new columns to the newsletters table to allow more customisation:
- background_color, default to 'light'
- border_color, nullable, null = no border
- title_color, nullable, null = auto
2023-03-28 11:38:43 +02:00
Fabien "egg" O'Carroll
349f3b01eb Fixed snapshots for post stats export e2e-api test
The Content-Disposition header was not matching because we use the date in the
filename for the exported CSV file. I haven't created a new matcher here because
it's quite unique, and we don't want an `anyContentDisposition` matcher because
the filename is a part of the API.
2023-03-28 09:48:17 +07:00
Simon Backx
465c2ee8d5 Updated post export E2E test ordering
no issue

Ordering on title differs between MySQL and SQLite, so replaced it with published_at.
2023-03-27 11:22:33 +02:00
Simon Backx
2fedce8a37 Added E2E tests for post export endpoint
fixes https://github.com/TryGhost/Team/issues/2788
2023-03-27 11:11:49 +02:00
Simon Backx
868adf30fd Fixed batch sending snapshot date matching
no issue

The replacement didn't match dates with only one number.
2023-03-27 10:24:34 +02:00
Rishabh Garg
0c743d67af
🐛 Fixed member newsletter subscription not saving in Admin (#16490)
fixes https://github.com/TryGhost/Team/issues/2783 
refs cb05fae5a3

The root cause of the issue was the fact we no longer checked for lack of `newsletters` property on member data before checking its `subscribed` property which is now deprecated. This caused a cascading effect where `subscribed:false` property on a member overrides the value for `newsletters` data. The check was accidentally removed in a previous bug fix.

So for members that were not subscribed to any newsletters, saving a newsletter subscription failed as they had their `subscribed` set to `false`, and it was resetting the newsletter subscription to empty always.
2023-03-24 23:29:49 +05:30
Sanne de Vries
387dfa59c9
Fixed issue with email template in Outlook (#16486)
Refs https://github.com/TryGhost/Team/issues/2801

- It was not possible to click latest post links in Outlook due to <a>
tag wrapping around a table
- The post meta data wouldn't display properly when centered in Outlook

---------

Co-authored-by: Simon Backx <simon@ghost.org>
2023-03-24 14:54:16 +01:00
Simon Backx
d9c92816e0 Added dynamic text truncation in email latest posts
refs https://github.com/TryGhost/Team/issues/2675

Truncate text depending on mobile/desktop and feature image.
2023-03-24 12:15:16 +01:00
Naz
2231981880
Added e2e test checking integration access
refs https://github.com/TryGhost/Team/issues/2790

- The Self-Serve Integration should only be accessible to the Owner and Admin user roles otherwise we risk accidental indirect increase in role permissions - Self-Serve Integration has permissions which editors/contributors don't have.
2023-03-24 11:31:25 +01:00
Naz
fe4e9897fc
Added migration for Self-Serve Integration permissions
refs https://github.com/TryGhost/Team/issues/2790

- This migration adds permissions for Self-Serve Migration Integration to have access to Admin APIs:

POST /ghost/api/admin/db
POST /ghost/api/admin/db/media/inline
POST /ghost/api/admin/members/upload
GET  /ghost/api/admin/tags/:id
GET  /ghost/api/admin/tags/slug/:slug
2023-03-24 11:31:25 +01:00
Naz
0b107f5af5
Added migrations for Self-Serve Migration Integration and API key
refs https://github.com/TryGhost/Team/issues/2790
refs 3747df1bc8 (diff-396038cecd7a381616d00954ae18a655ae2a8af71ea65866bf09d2c7cc1b5235)

- This integration will be used to perform self-serve migrations.
- The integration will be limited to these Admin API endpoints:
POST /ghost/api/admin/db
POST /ghost/api/admin/db/media/inline
POST /ghost/api/admin/members/upload
GET   /ghost/api/admin/tags/:id
GET   /ghost/api/admin/tags/slug/:slug
2023-03-24 11:28:29 +01:00
Simon Backx
21ded0b4b8 Corrected email link in subscription box
refs https://github.com/TryGhost/Team/issues/2674

The replacement didn't work inside an attribute, so I removed it to a link that doesn't do anything.
2023-03-24 10:23:51 +01:00
Simon Backx
a9791e95cc Fixed including current email in last 3 posts in newsletters
refs https://github.com/TryGhost/Team/issues/2675

The current post could be inclnuded in the last 3 posts if it was also published. This is now fixed and covered by a test.
2023-03-24 09:20:32 +01:00
Simon Backx
0107d2bb77 Fixed subscription status not showing correctly in emails
refs https://github.com/TryGhost/Team/issues/2674

- The segment detection doesn't work outside the main post content. So the data-gh-segment attribute didn't work. It is now replaced with just a simple email replacement that is empty for a free member.
- Fixed that a trialing member was shown as 'paid'. This is now replaced with 'trialing'.

This commit also includes E2E tests for a couple of member statusses.
2023-03-24 08:55:36 +01:00
Sanne de Vries
3f0ec1af3d Updated subscription box in email preview and email template
Refs https://github.com/TryGhost/Team/issues/2738

- With all footer additions and changes, the subscription box did not match the rest of the design and is therefore updated.
2023-03-23 19:39:58 +01:00
Aileen Nowak
37383fde90 Moved Milestone emails to GA
- Moves Milestone emails from public beta to GA Moved Milestone emails to GA

- Moves Milestone emails from public beta to GA Moved Milestone emails to GA

- Moves Milestone emails from public beta to GA Moved Milestone emails to GA

- Moves Milestone emails from public beta to GA Moved Milestone emails to GA

- Moves Milestone emails from public beta to GA Moved Milestone emails to GA

- Moves Milestone emails from public beta to GA Moved Milestone emails to GA

- Moves Milestone emails from public beta to GA Moved Milestone emails to GA

- Moves Milestone emails from public beta to GA Moved Milestone emails to GA

- Moves Milestone emails from public beta to GA
2023-03-23 17:35:23 +00:00
Sanne de Vries
0455672832 Updatet latest post section in email template
Refs https://github.com/TryGhost/Team/issues/2675
2023-03-23 13:16:48 +01:00
Daniel Lockyer
2941154353
Added dynamic date matching to email sending snapshot
- we don't want to include the actual date here because it'll change
- this adds a regex to match the date and replace it with a standard
  "date" string, which won't change
2023-03-23 09:20:23 +01:00
Sam Lord
5065950401 Mocked mail to prevent errors in e2e test suite
refs: https://github.com/TryGhost/Toolbox/issues/389
2023-03-22 16:25:23 +00:00
Sam Lord
47d9b4a51d Fixed errors being logged by unit tests
refs: https://github.com/TryGhost/Toolbox/issues/389
2023-03-22 16:25:23 +00:00
Simon Backx
860048a89d Fixed snapshots 2023-03-22 17:03:11 +01:00
Simon Backx
46f305efaf Fixed hiding name if empty in email subscription box
no issue

Moved some CSS around because I moved the wrong code in the previous commit.
2023-03-22 16:51:56 +01:00
Simon Backx
55c281debf Reverted removeStyleTags from email-renderer
no issue

Some things break in some email clients with this new setting. Disabled it for now and moved the required css style to hide the member name row to @media all.
2023-03-22 16:40:35 +01:00
Simon Backx
4eebf6612a Added image dimensions to the latest posts mobile version in email
fixes https://github.com/TryGhost/Team/issues/2799

Separate image dimensions for mobile and normal images.
2023-03-22 15:54:27 +01:00
Simon Backx
480c1a7004 Removed name from subscription details if missing
refs https://github.com/TryGhost/Team/issues/2736

If the name is not known for a member, we'll hide the name row in the subscription details in an email. This method is supported in most email clients, and requires the support of `<style>` in `<head>`.
2023-03-22 15:32:07 +01:00
Sanne de Vries
47e343ec18 Updated latest posts UI in email template
Refs https://github.com/TryGhost/Team/issues/2801
2023-03-22 15:16:55 +01:00
Sam Lord
3718d4acda Enable labs in playwright tests
no issue

Allows us to write tests for features which are still in development.
2023-03-22 11:16:49 +00:00
Simon Backx
bc0126c54e
Added subscription status text in newsletters (#16442)
fixes https://github.com/TryGhost/Team/issues/2736

Shows the actual subscription status (expires on DD MMM YYYY) in every
email when show subscription details is enabled.
2023-03-22 11:52:41 +01:00
Daniel Lockyer
247163c710
Fixed tests for verification during batch sending
refs 59df99388c

- it seems the `getSignupEvents` method is now called twice during these
  tests
2023-03-22 09:44:01 +01:00
Aileen Booker
8d290c4560
Milestone emails templates and sending implementation (#16318)
refs
https://www.notion.so/ghost/Marketing-Milestone-email-campaigns-1d2c9dee3cfa4029863edb16092ad5c4?pvs=4

- Added email template for milestones with using a configuration file
for different member milestone values, as we're sending different
content for each one
- Implement sending the email to users who have
`milestone-notifications` enabled, currently still behind a flag

Co-authored-by: Peter Zimon <peter.zimon@gmail.com>
2023-03-21 13:39:40 +00:00
Sanne de Vries
792bfdb498 Added UI for subscription box in newsletters
Refs https://github.com/TryGhost/Team/issues/2738

- Added preview to email newsletter settings
- Added subscription box UI to email template
2023-03-21 11:40:04 +01:00
Rishabh
0b7e09c986 Updated fallback for storing source site title
refs https://github.com/TryGhost/Team/issues/2754

Previously, we didn't have any backup for storing source site title and it could have stored as empty if missing. This change ensures the source site title is stored as site host instead as fallback if not present.
2023-03-21 16:09:21 +05:30
Simon Backx
a016f53b7e Added resolveMx stub to all E2E tests
refs https://ghost.slack.com/archives/C04TMVA1D7A/p1679391135060629
refs 1b0adcf4ab

The same issue when sending emails from E2E tests, where the tests timeout when resolving mx records for invalid domains.
2023-03-21 11:25:06 +01:00
Simon Backx
a5dff4207e Moved posts service to its own package
fixes https://github.com/TryGhost/Team/issues/2778

It is easier to add extra classes using the latest patterns if it has its own package.
2023-03-20 16:06:42 +01:00
Elena Baidakova
aca8c5805c Improved members-actions e2e browser test
refs TryGhost/Team#2667
- Added notification check before checking that user successfully unsubscribed from all newsletters. It helps to make the test more stable
2023-03-20 18:48:35 +04:00
Simon Backx
07ec33fb3a
Added latest posts to email template (#16448)
refs https://github.com/TryGhost/Team/issues/2769

Needs some extra styling and design, this is only a minimal version
behind the feature flag.
2023-03-20 14:30:42 +01:00
Simon Backx
a513542b56
Added show_latest_posts column to newsletter table (#16402) 2023-03-20 11:10:49 +01:00
Sanne de Vries
1b0adcf4ab Fixed missing stub for DNS in GhostMailer Direct tests
- without this, Node will try and resolve the domain name but local DNS
  resolvers can take a while to timeout, which causes the tests to timeout
- `nodemailer-direct-transport` calls `dns.resolveMx`, so if we stub that
  function and return an empty array, we can avoid any real DNS lookups
2023-03-17 18:16:30 +01:00
Simon Backx
a6654b5e6c Updated GhostMailer Direct unit tests to async await style
no issue

Tests were timing out for some users, probably because not all errors are caught.
2023-03-17 17:34:01 +01:00
Steve Larson
0a4d568f19 updated snapshot
no refs
-updated snapshot to pass tests
2023-03-17 10:57:50 -05:00
Peter Zimon
9d25c2a058 Fixed mobile size footer buttons in newsletters
refs. https://github.com/TryGhost/Team/issues/2740

- alignment of buttons in the footer of newsletter email template was off
2023-03-17 16:34:23 +01:00
Steve Larson
067b35de6b fixed member newsletter test
no refs
-fixed flaky test that seemed to have sync issues with fixture data and db
2023-03-17 10:27:09 -05:00
Simon Backx
b3c410a923 Reverted stats snapshot change 2023-03-17 16:06:00 +01:00
Simon Backx
c591f1f86a Updated email batch sending snapshots 2023-03-17 16:00:14 +01:00
Sanne de Vries
48a3159d3d Added mobile feedback buttons to email template
Refs https://github.com/TryGhost/Team/issues/2740
2023-03-17 15:58:14 +01:00
Daniel Lockyer
b7a03ca765 🐛 Fixed mainEntityOfPage property within page metadata
fixes https://github.com/TryGhost/Ghost/issues/16367
refs https://github.com/schemaorg/schemaorg/discussions/3274

- after a discussion with the schemaorg maintainers, it looks like
  `mainEntityOfPage` should refer to the entity itself vs the parent
  entity
- this commit updates the code to use the entity URL and fixes the
  associated tests
2023-03-17 15:13:28 +01:00
Simon Backx
005b5f20fb Added E2E tests for comment CTA
fixes https://github.com/TryGhost/Team/issues/2727

Adds some E2E tests and snapshots for comment CTA in all possible situations.
2023-03-17 11:12:47 +01:00
Simon Backx
298e3da745 Fixed comment CTA button link
refs https://github.com/TryGhost/Team/issues/2672

- Use #ghost-comments instead of #ghost-comments-root
- Fixed snapshots
2023-03-17 10:27:23 +01:00
Simon Backx
423337a048 Fixed snapshots 2023-03-17 10:24:21 +01:00
Sanne de Vries
717ed114f4 Fixed center aligned post title in emails
Refs https://github.com/TryGhost/Team/issues/2740
2023-03-17 10:07:17 +01:00
Chris Raible
0640b9c4aa
Modified outbound link tagger to use ?ref=domain instead of ?ref=siteTitle (#16431)
refs TryGhost/Team#2755

- Outbound link tagger will now append ?ref=example.com or ?ref=example.ghost.io instead of ?ref=siteTitle
2023-03-17 14:18:26 +05:30
Naz
c487b12518 Reverted zip's root file copying fix
refs https://github.com/TryGhost/Toolbox/issues/523

- The reverted fix did not take into account the "original path" of the
files would be truncated. This path has to be full relative to the root
of the zip to later be used during importer url substitution logic.
- This reverts commit 831a76505c.
2023-03-16 21:52:24 +01:00
Steve Larson
7f223b56dd
fixed member newsletter test (#16425)
no ref
-fixed flaky test that seemed to have sync issues with fixture data and
db
2023-03-16 10:43:17 -05:00
Sanne de Vries
cdcb3dcd6f
Updated email template to include comment link in header and footer (#16423)
Refs https://github.com/TryGhost/Team/issues/2740

---------

Co-authored-by: Steve Larson <9larsons@gmail.com>
2023-03-16 16:10:53 +01:00
Elena Baidakova
60b3731199 Separated admin and portal tests to different projects
refs TryGhost/Team#2667
- Portal tests are much more stable when running in the separate DB from admin tests.
2023-03-16 18:35:09 +04:00
Simon Backx
450e01d1c0 Added placeholder basic subscription details to email template
refs https://github.com/TryGhost/Team/issues/2736

The renewal date is still missing, and style and desin is only a placeholder.
2023-03-15 17:12:56 +01:00
Simon Backx
80043f2699 Fixed snapshot tests in batch sending 2023-03-15 14:11:05 +01:00
Simon Backx
6e5ddc6a9a Fixed email preview snapshots 2023-03-15 14:03:24 +01:00
Sanne de Vries
e0c3573079 Updated email template snapshot resulting in test failure
No ref
2023-03-15 12:45:54 +01:00
Simon Backx
da2e56916e
Added show_subscription_details column to newsletter table (#16403)
fixes https://github.com/TryGhost/Team/issues/2708

Adds `show_subscription_details` column to newsletter table. Disabled by
default.
2023-03-15 11:53:45 +01:00
Fabien 'egg' O'Carroll
bdaddd1a64
Added setting to store last mention report email date (#16399)
Because there is no guarantee about a daily job running exactly once a
day, we need to store the last time that the email was sent, so that we
can refrain from sending one if it's been less than a day since the
last.

A setting has been used for this as we don't currently have a pattern
for it, we might want to consider moving this to some kind of cache
based solution in future. This has been added as a core setting so that
we don't expose it via the API.

The setting is stored as a number to allow us to store value as unix timestamp.

---------

Co-authored-by: Rishabh <zrishabhgarg@gmail.com>
2023-03-15 15:34:41 +05:30
Simon Backx
b27c7bc707 Fixed batch sending snapshots dates
no issue

The snapshots contained the current date, so they broke every day. This commit fixes the issue by setting a fixed date.
2023-03-15 09:45:36 +01:00
Simon Backx
bb019a0ab5 Added E2E test for sending email without post title
fixes https://github.com/TryGhost/Team/issues/2724

This change also includes new snapshots for email sending (similar for email previews, but this time for the real emails to make sure we catch changes).
2023-03-14 16:05:18 +01:00
Simon Backx
350b291c34
Added show_comment_cta column to newsletter table (#16401)
fixes https://github.com/TryGhost/Team/issues/2707

Adds the new column for storing whether we need to show a comment CTA in
emails. Enabled by default.
2023-03-14 15:03:02 +01:00
Simon Backx
1e435fb328
Implemented showPostTitleSection (#16404)
fixes https://github.com/TryGhost/Team/issues/2705

- Added showPostTitleSection to newsletter model in admin
- Wired up UI to admin model so it saves to the database
- Implemented showPostTitleSection in newsletter preview and added some
minor temporary css styling
- Implemented showPostTitleSection in newsletter template in backend,
and added some extra CSS styling to fix spacing
2023-03-14 11:29:43 +01:00
Simon Backx
69d07c892c
Fixed getLazyRelation returning new model for optional relations (#16411)
no issue

When using `getLazyRelation` on an optional relation that is not set, it
will return a newly created model instead of a model from the database.
- Adds a new require option to `getLazyRelation`, that throws an error
if the relation is not set (off by default to match existing use cases)
- This caused a bug (not visible because we always pass a newsletter id)
in email previews, where when the newsletter id was not explicitly set,
it would use `newsletter = (await post.getLazyRelation('newsletter')) ??
(await this.models.Newsletter.getDefaultNewsletter());`, which always
returned the first one, and could return a newly initiated newsletter
with all properties set to undefined.
- Some page snapshots are altered by this, because the usage of
`getLazyRelation` on a post no longer sets the email relation to some
new model.
2023-03-14 10:58:58 +01:00
Simon Backx
ff5e7202c6
Added show_post_title_section column to newsletters table (#16397)
fixes https://github.com/TryGhost/Team/issues/2706

Only includes the migration. The setting is not used at the moment, but
will be enabled by default.
2023-03-14 09:48:13 +01:00
Aileen Booker
eeb7546abb
Added handling for initial and skipped Milestones (#16405)
refs
https://www.notion.so/ghost/Marketing-Milestone-email-campaigns-1d2c9dee3cfa4029863edb16092ad5c4?pvs=4

- When milestones will be activated we would send out emails to users
that are way above the achieved milestone, as we didn't record
milestones before
- The plan is to implement a 0 milestone and don't send an email for
achieving those and also add all achieved milestones in the first run
until a first milestone is stored in the DB, then increment from there.
- This change takes care of two cases:
1. Milestones gets enabled and runs initially. We don't want to send
emails unless there's already at least one milestone achieved. For that
we add a 0 milestone helper and add a `initial` reason to the meta
object for the milestone event, so we can choose not to ping Slack and
also disable email sending for all milestones achieved in this initial
run.
2. All achieved milestones will be stored in the DB, even when that
means we skip some. This introduces the `skipped` reason which also
doesn't send emails for the skipped milestones, but will do for
correctly achieved milestones (always the highest one).
- Added handling for slack notifications to not attempt sending when
reason is `skipped` or `initial`
2023-03-13 19:01:11 +02:00
Fabien 'egg' O'Carroll
4525c7293c
Removed email notifications for new mentions
These are no longer desired and will be replaced with a digest style
email.
2023-03-13 18:33:57 +07:00
Steve Larson
8d76832b07 added e2e tests to cover page events
no refs
-added e2e tests for sending mentions on various page events
2023-03-10 15:25:10 -06:00
Simon Backx
80cec29144
Added Stripe Mocker to make testing easier (#16396)
no issue

The Stripe Mocker mocks the Stripe API in memory, to make it much easier
to test subscription flows. Currently it is more a POC to see if it
works well. It probably needs a bit more work to support more scenarios.
- Added new tests for the subscription stats endpoint for 3D secure +
free trial flows using the new Stripe Mocker
- Updated members admin api tests to use Stripe Mocker (+ added new test
for deleting members with Stripe cancellation)
- Some tests called mockStripe at the beginning, but that method did
nothing apart from disabling network (which is the default now), then
they mocked Stripe inside the tests file... so I've removed those
because those conflict with the new mocker that is enabled when calling
mockStripe. We'll need to port those over later.
2023-03-10 18:43:35 +01:00
Daniel Lockyer
89493893d1 Removed all unused variables from test files
- this cleans up all imports or variables that aren't currently being used
- this really helps keep the tests clean by only allowing what is needed
- I've left `should` as an exemption for now because we need to clean up
  how it is used
2023-03-10 14:29:55 +01:00
Daniel Lockyer
42333a3349 Removed custom assertions
- these are no longer used in favor of snapshots in tests
2023-03-10 14:29:55 +01:00
Daniel Lockyer
08f84475e7 Removed sleep lib from test utils
- this is generally an anti-pattern in tests and leads to flaky
  behaviour when tests are ran on different machines/loads
- this is currently unused so it is an easy removal
2023-03-10 14:29:55 +01:00
Elena Baidakova
6f9f83bee8 Revert "Removed explicit timeout for offer creation helper"
This reverts commit ac427e4253.
2023-03-10 13:44:45 +04:00
Elena Baidakova
ac427e4253 Removed explicit timeout for offer creation helper
refs TryGhost/Team#2667
- Removed timeout to make it the same as a global one
2023-03-10 13:24:38 +04:00
Ronald Langeveld
ab01e4fcbf Added new Tier e2e test
refs https://github.com/TryGhost/Team/issues/1446

- Added a test for successfully creating a new tier.
2023-03-10 13:29:25 +08:00
Simon Backx
832610fd2a
🐛 Fixed retrying failed emails when rescheduling them (#16383)
fixes https://github.com/TryGhost/Team/issues/2560

When an email fails, and you reschedule the post, the error dialog was
shown (from the previous try). The retry button on that page allowed you
to retry sending the email immediately, which could be very confusing.

- The email error dialog is no longer shown for scheduled emails
- The email status is no longer polled for scheduled emails
- Retrying an email is not possible via the API if the post status is
not published or sent
- Added some extra snapshot tests
- When retrying an email, we immediately update the email status to
'pending' to have a better API response (instead of still returning
failed).
- Disabled email sending retrying in development (otherwise very hard to
test failed emails if it takes 10 mins before it gives up automatic
retrying)
2023-03-09 12:32:22 +01:00
Elena Baidakova
8b191a8d01 Removed all tiers archive from portal tests to reduce relation between tests
refs TryGhost/Team#2667
2023-03-09 12:55:27 +04:00
Ronald Langeveld
7539a681fe
Excludes sites with canonical_url meta from sitemap. (#16376)
closes https://github.com/TryGhost/Team/issues/2531

This commit fixes the issue where non-canonical URLs are included in the
XML sitemap, leading to poor SEO for our user's sites. The solution
implemented is to exclude any page or post that specifies a canonical
URL in its metadata from the sitemap.

To achieve this, a condition has been added to the 'addUrl' method,
which checks for the existence of a canonical URL in the metadata of the
resource being added to the sitemap. If a canonical URL is present, the
resource is excluded from the sitemap.

With this fix, our user's sites will have better SEO and improved search
engine visibility.
2023-03-09 16:38:43 +08:00
Steve Larson
cb05fae5a3
🐛 Fixed maintaining archived newsletter subscriptions for members (#16375)
refs #16355
-archived newsletters will no longer be unsubscribed when saving member
-fixed bug with initialMember load
-fixed errors in unit tests
2023-03-08 16:39:28 -06:00
Simon Backx
4184b279d2
🐛 Fixed HTML escaping when using outbound link tagging (#16380)
fixes https://github.com/TryGhost/Team/issues/2666

- Somehow occurrences of `&map_` got replaced with `&#x21A6;`
- Disables escaping &, ', " and other HTML characters when not needed
(escaping is already handled by mobiledoc/lexical)
- Bumps unit test coverage of link replacer to 100%
2023-03-08 16:30:54 +01:00
Elena Baidakova
00cda68125
Disabled fraud detection for e2e browser tests (#16379)
refs TryGhost/Team#2667
2023-03-08 16:01:08 +04:00
Simon Backx
3d89dbb775 Fixed whistelisted domains in Playwright tests and network connectivity
no issue

- Nock doesn't support multiple calls to enableNetConnect -> only the last one counts. This fixes that issue.
- Some tests interacted directly with nock instead of using the mockManager to restore everything.
2023-03-08 12:40:21 +01:00
Simon Backx
df9df4be80 Allowed Stripe network access in Playwright tests
no issue

Previous test change that blocks network access broke the Playwright tests, that need to access the Stripe checkout page.
2023-03-08 11:49:36 +01:00
Naz
831a76505c
Fixed root zip image/media/files copying during import
https://github.com/TryGhost/Toolbox/issues/523

- During import process of content files the files from the root directory were also copied over. This is causing chaos in the root of content folder with files that only needed for data import. For example, the csv files needed for Revue import were also copied over by "file importer" even though those do not belong to any content.
- Any content import files - images, media, files, should be in according folders in the imported zip file. The root files in the base zip directory are for data-related imports
2023-03-08 11:03:47 +08:00
Simon Backx
38de815d98
Removed old email flow (#16349)
fixes https://github.com/TryGhost/Team/issues/2611

The old email flow is no longer used since we introduced the email stability flow. This commit removes the related code and tests. The general test coverage decreased a bit as a result, because the old email flow probably had a high test coverage. The new flow is in separate packages, so it couldn't contribute to a higher test coverage (but it does have 100% unit test coverage).
2023-03-07 16:08:40 +01:00
Simon Backx
3db434736b
🐛 Fixed replacements with fallback in plaintext newsletters (#16372)
fixes https://github.com/TryGhost/Team/issues/2683

When sending a newsletter with a replacement that has a fallback, the
replacement only happens in the HTML version of the newsletter. The
plaintext version isn't replaced.

This commit fixes the issue and adds some tests to make sure it doesn't
happen again.

The cause of the issue was that we used the original matched Regex text
to replace. But that was calculated on the HTML version, so double
quotes were encoded. This change updates the generated 'token' regex to
also match on both a double quote as the escaped double quote.
2023-03-07 15:34:43 +01:00
Simon Backx
f50224f445
Improved network access and mocking in tests (#16371)
refs https://github.com/TryGhost/Team/issues/2667

Some tests still accessed the internet. Now network access is disabled
by default. This change also introduces two helper methods related to
networking (mocking Slack and Mailgun).

This fixes two unreliable tests:
- Staff service was accessing a Slack test API -> timeout possible
- MentionSendingService was trying to send webmentions for every post
publish/change -> possible timeouts and job issues
2023-03-07 13:20:28 +01:00
Tim Gates
5394fa81fc
Fixed typos (#16359) 2023-03-06 13:11:45 +01:00
Naz
33316a6440
Added e2e test covering media inliner job start
refs https://github.com/TryGhost/Toolbox/issues/523

- The test is useful for future iterations of the response format and as a quick reference on which parameters the media inlining endpoint accepts.
2023-03-06 17:43:33 +08:00
Sam Lord
472e52a536 Made URL class warn on duplicate URLs in the test suite
refs: https://github.com/TryGhost/Toolbox/issues/389

Instead of logging errors, this will warn when adding a duplicate URL in the test environment.

At the moment, this is happening a lot in the test suite. While we also need to fix the root cause of this so we're not erroring in the product, it's a massive amount of spam in the logs when running the test suite which could prevent us from finding other errors which are causing issues.
2023-03-03 18:25:53 +00:00
Sam Lord
13b0f51b13 Prevent simple error logs by stubbing log library
refs: https://github.com/TryGhost/Toolbox/issues/389

This removes many error logs when the end-to-end test suite is run with the log-level set to error. Many errors are intentional, so the resolution is typically to stub the error log function and assert that it would have been called.
2023-03-03 18:25:53 +00:00
Sam Lord
cd5aca7227 Prevented .well-known file tests from attempting to send mail
refs: https://github.com/TryGhost/Toolbox/issues/389

These tests would log errors while attempting to send email.
2023-03-03 18:25:53 +00:00
Sam Lord
5b2adbe4c5 Prevents errors from being logged during the offers tests
refs: https://github.com/TryGhost/Toolbox/issues/389

Intentional errors now call a logging library stub
2023-03-03 18:25:53 +00:00
Sam Lord
64d94c8c8c Prevent posts-legacy tests from logging errors
refs: https://github.com/TryGhost/Toolbox/issues/389

The newsletter fixtures no longer errors when accessing the test image, and tests which intentionally error now stub the logging call
2023-03-03 18:25:53 +00:00
Sam Lord
a0acccc088 Prevent errors in E2E tests where front-end is disabled and Ghost Admin accesses themes
refs: https://github.com/TryGhost/Toolbox/issues/389
2023-03-03 18:25:53 +00:00
Steve Larson
3e18715caa added twitter embed tests
refs TryGhost/Team#2663
-added basic coverage for twitter embeds
2023-03-03 10:54:03 -06:00
Naz
a701724903
Renamed content file importer test
refs https://github.com/TryGhost/Toolbox/issues/523

- Test was not renamed after the "image" module changed it's name to "ContentFileImporter"
2023-03-02 17:59:36 +08:00
Naz
ab3959f3f2
Added document file support to the zip importer
refs https://github.com/TryGhost/Toolbox/issues/523

- When migrating or importing ZIP files into Ghost there's often a need to include document files.
- When document files are present in the imported zip file they are now copied across and processed along with the rest of import files: json, images, csvs, etc.
- The importer also searches for use of the document files in the imported "posts" substituting the links with local ones
- The document files importer recognizes media files inside of "files" or "content/files" folders present in the zip.
- The supported media file extensions are same as for file upload widget:
".pdf",".json",".jsonld",".odp",".ods",".odt",".ppt",".pptx",".rtf",".txt",".xls",".xlsx",".xml"

with following content-types:
"application/pdf", "application/json", "application/ld+json", "application/vnd.oasis.opendocument.presentation", "application/vnd.oasis.opendocument.spreadsheet", "application/vnd.oasis.opendocument.text", "application/vnd.ms-powerpoint", "application/vnd.openxmlformats-officedocument.presentationml.presentation", "application/rtf", "text/plain", "application/vnd.ms-excel", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", "application/xml", "application/atom+xml"
2023-03-02 17:54:19 +08:00
Naz
cd49992cd9
Added generic file import handler
refs https://github.com/TryGhost/Toolbox/issues/523
refs c2534e3c86/packages/mg-assetscraper/lib/AssetScraper.js (L14-L16)
refs https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types

- Importer needs to process and recognize document files like pdfs, presentations etc to be able to import them into sites file storage.
- The handler allows a new root directory "files" to place imported documents
- The handler adds validation and processing for following file extensions:
".pdf",
".json",
".jsonld",
".odp",
".ods",
".odt",
".ppt",
".pptx",
".rtf",
".txt",
".xls",
".xlsx",
".xml"
- With following content types:
"application/pdf",
"application/json",
"application/ld+json",
"application/vnd.oasis.opendocument.presentation",
"application/vnd.oasis.opendocument.spreadsheet",
"application/vnd.oasis.opendocument.text",
"application/vnd.ms-powerpoint",
"application/vnd.openxmlformats-officedocument.presentationml.presentation",
"application/rtf",
"text/plain",
"application/vnd.ms-excel",
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"application/xml",
"application/atom+xml"
2023-03-02 17:54:18 +08:00
Ronald Langeveld
7c805b7176 Revert "Added new created tier e2e test"
This reverts commit dc98868961.
2023-03-02 16:39:17 +08:00
Ronald Langeveld
dc98868961 Added new created tier e2e test
ref https://github.com/TryGhost/Team/issues/1446

- Added e2e test to check if a new tier can be added successfully.
2023-03-02 16:11:12 +08:00
Naz
2361669bb6
Added media file support to the zip importer
refs https://github.com/TryGhost/Toolbox/issues/523

- When migrating or importing ZIP files into Ghost there's often a need to include more than just post content and images.
- When media files are present in the imported zip file they are now copied across and processed along with the rest of import files: json, images, csvs, etc.
- The importer also searches for use of the media files in the imported "posts" substituting the links with local ones
- The media importer recognizes media files inside of "media" or "content/media" folders present in the zip. The supported media file extensions are same as for media upload widget:
".mp4", ".webm", ".ogv", ".mp3", ".wav", ".ogg", ".m4a"

with following content-types:
 "video/mp4", "video/webm", "video/ogg", "audio/mpeg", "audio/vnd.wav", "audio/wave", "audio/wav", "audio/x-wav", "audio/ogg", "audio/mp4", "audio/x-m4a"
2023-03-02 14:35:37 +08:00
Naz
49553e5942
Renamed "image" importer to generic ContentFileImporter
refs https://github.com/TryGhost/Toolbox/issues/523

- This is ground work before introducing a "media" content type importer
- Previous "image" file name was not describing well what the importer was capable of doing
2023-03-02 14:35:37 +08:00
Sam Lord
90ef8f008e Prevent error logs from occuring in Ghost unit test suite
refs: https://github.com/TryGhost/Toolbox/issues/389
2023-03-01 16:37:46 +00:00
Naz
eb8d9df923
Made image importer type dynamic
refs https://github.com/TryGhost/Toolbox/issues/523

- Having a dynamic importer type allows to reuse the image importer implementation for other file types e.g.: 'media'
2023-03-01 18:28:17 +08:00
Naz
edebe1571c
Extracted image store to constructor parameter
refs https://github.com/TryGhost/Toolbox/issues/523

- We need to be able to use different storage mechanisms when importing different types of content
- Having the storage passed in using constructor DI allows to have more flexible storage mechanism in the Images importer (soon to become a generic file importer)
2023-03-01 18:21:46 +08:00
Naz
4573d8b4b8
Refactored image importer to use class
refs https://github.com/TryGhost/Toolbox/issues/523

- The class syntax would allow swapping out the storage mechanism in the importer making it universal to use with other file types like media or generic files.
2023-03-01 18:10:39 +08:00
Naz
2bf98e288f
Removed unused module from test suite
refs https://github.com/TryGhost/Toolbox/issues/523
2023-03-01 18:10:39 +08:00
Naz
e14520115a
Extracted Image Importer test suite
refs https://github.com/TryGhost/Toolbox/issues/523
refs 3119a5cc4c

- The "index.test.js" is doing too much, has to be broken down to allow things to move into new packages
2023-03-01 16:48:38 +08:00
Daniel Lockyer
4c2ff0539d
Fixed tests for ImportManager audio types
refs c5a4ee89c9

- as of the referenced commit, the supported list of mime types has been
  updated
2023-03-01 09:14:09 +01:00
Naz
3119a5cc4c
Extracted ImageHandler test suite
refs https://github.com/TryGhost/Toolbox/issues/523

- The "importer/index.test.js" test suite is testing more than it should. ImageHandler test suite section is one of the examples of test cases that should live in a separate file.
- Having these tests in different files makes it easier to reason about coverage and extract to it's own packages.
2023-03-01 15:55:10 +08:00
Naz
ce5eea5707
Added media file handling to Ghost importer
refs https://github.com/TryGhost/Toolbox/issues/523

- When a zip file is imported into Ghost we need to recognize and process media files with following extensions:
".mp4",".webm", ".ogv", ".mp3", ".wav", ".ogg", ".m4a"
- The media files can come from a "media" or "content/media" folder inside of zip file
2023-03-01 15:55:10 +08:00
Naz
23e2fdde95
Added image importer test coverage
refs https://github.com/TryGhost/Toolbox/issues/523

- The test cases for "content" and "content/images" zip import file structure were missing.
2023-03-01 15:55:10 +08:00
monkey sees
838516efb0 🐛 Fixed Handlebars’ asset helper for SafeString input
closes https://github.com/TryGhost/Ghost/issues/16332

Passing `SafeString` input to `asset` helper was resulting in the
exception being thrown. This meant that we couldn’t combine `asset`
helper with other helpers which produce `SafeString` e.g. `concat`
helper for string concatenation.
2023-03-01 08:39:14 +01:00
Reupen Shah
c5a4ee89c9 🐛 Fixed uploads of m4a files with audio/mp4 content type
fixes https://github.com/TryGhost/Ghost/issues/16301

Previously, audio/x-m4a was allowed but not audio/mp4. This meant
uploads of m4a files failed in some cases e.g. Firefox on Windows.
2023-03-01 08:32:21 +01:00
Sam Lord
579ecd5b0b Prevent error logging in Urls tests from being output
refs: https://github.com/TryGhost/Toolbox/issues/389
2023-02-28 11:14:11 +00:00
Sam Lord
6512fc770d Prevent xmlprc tests from outputting errors for failure cases
refs: https://github.com/TryGhost/Toolbox/issues/389
2023-02-28 11:14:11 +00:00
Sam Lord
007adb6a33 Prevent tests from erroring when calling validate.init()
refs: https://github.com/TryGhost/Toolbox/issues/389

Calling validate always uses the cache system, so this commit makes sure that the cache system is always initialised correctly by the tests.
2023-02-28 11:14:11 +00:00
Daniel Lockyer
93e0898c12
Extracted version-match middleware to separate package
- this middlware is standalone and has 100% test coverage, therefore we
  can move it out to a separate package and remap the requires
  accordingly
2023-02-27 18:34:03 +01:00
Aileen Nowak
d441fe2e2a Fixed tests 2023-02-23 17:21:20 +02:00
Aileen Nowak
086da740a2 Changed maxPercentageFromMilestone to 10% 2023-02-23 17:21:20 +02:00
Aileen Nowak
74a65f3b1c Simplyfied milestone event meta data
no issue

- Reduced the amount of diffeerent properties by not populating a `currentARR` and `currentMembers` fields, but use a `currentValue` instead.
- The type of milestone can still be determined by its `type` property, so we actually don't need two different props here
2023-02-23 17:21:20 +02:00
Aileen Nowak
d820b961b0 Avoid sending milestone email when actual value is too far from milestone
no issue

- Switches to used newly added config values throughout the services
- Updated the `shouldSendEmail` fn to check if actual value is too far from achieved milestone as determined by the percentage setting (e. g. 998 members should not accidentally receive an email for achieving 100 members)
2023-02-23 17:21:20 +02:00
Simon Backx
89ababb71f
Updated email event fetching to stop when begin and end are the same (#16326)
no issue

Optimization that makes sure we stop fetching when it is no longer
needed.
2023-02-23 15:44:01 +01:00
Simon Backx
3ded0bbee8
Added outbound link tagging setting (#16324)
no issue

Enable or disable outbound link tagging in both web posts and
newsletters.
2023-02-23 14:37:23 +01:00
Aileen Nowak
ff3568293c Updated milestones e2e test to better check for last email sent safeguard
refs https://www.notion.so/ghost/Marketing-Milestone-email-campaigns-1d2c9dee3cfa4029863edb16092ad5c4?pvs=4

- Didn't check for the specific case before and fixed a bug that was allowiing the test to pass even though it should fail
2023-02-23 14:10:12 +02:00
Simon Backx
f6bc3479f9
Added webmentions emails flag (#16321)
fixes https://github.com/TryGhost/Team/issues/2599

Disables sending webmentions notification emails, and hides it behind a new flag instead.
2023-02-23 11:42:14 +01:00
Aileen Booker
e0331bbfcf
Added MilestoneCreatedEvent to staff service notifications (#16307)
refs
https://www.notion.so/ghost/Marketing-Milestone-email-campaigns-1d2c9dee3cfa4029863edb16092ad5c4?pvs=4

- Added MilestoneCreatedEvent subscription to staff-service incl. first
handling
- Logs information for now instead of actually sending an email
2023-02-23 11:20:13 +02:00
Aileen Nowak
80ebd1c5be Fixed random timeouts in milestone service e2e test
no issue

- Passed specific config to `useFakeTimers` so the test only fakes the `setTimeout` call
2023-02-22 20:26:53 +02:00
Simon Backx
7220be92e2 Webmentions sending tests stability improvements
no issue
2023-02-22 18:25:06 +01:00
Steve Larson
18eeb9e523 moved spam prevention test
no refs
-spam prevention test was causing subsequent tests to fail randomly
-moving to the end ensures (for now) we don't interrupt other tests
-seems to be an issue with awaiting the jobservice which do concurrent
2023-02-22 09:52:26 -06:00
Simon Backx
a7090dddcd Email snapshot updates 2023-02-22 16:49:28 +01:00
Simon Backx
f45d1810a6 Improved webmention receiving E2E test reliability
refs https://github.com/TryGhost/Team/issues/2596
2023-02-22 16:19:56 +01:00
Aileen Booker
3aa75ee5dc
Improved milestone e2e test (#16314)
refs
https://www.notion.so/ghost/Marketing-Milestone-email-campaigns-1d2c9dee3cfa4029863edb16092ad5c4?pvs=4

- Check with more realistic dates and validate changes on milestone
model
2023-02-22 16:15:21 +02:00
Aileen Booker
cf7d34d862
Added BookshelfMilestoneRepository implementation (#16305)
refs
https://www.notion.so/ghost/Marketing-Milestone-email-campaigns-1d2c9dee3cfa4029863edb16092ad5c4?pvs=4

This stores the received milestones in the database.
2023-02-22 15:53:29 +02:00
Ronald Langeveld
76640d9b7e
Added page resources to emails (#16315)
refs https://github.com/TryGhost/Team/issues/2572

- Passes post resource data to emails.
- Updated email templates accordingly.
- Added snapshot e2e tests.
2023-02-22 21:02:15 +08:00
Aileen Booker
4556e9be88
Added milestone_notifications column (#16306)
refs
https://www.notion.so/ghost/Marketing-Milestone-email-campaigns-1d2c9dee3cfa4029863edb16092ad5c4?pvs=4

- created a migration for a new boolean column in users that would
determine if the staff user gets an email when a new milestone is
achieved.
2023-02-22 14:13:50 +02:00
Aileen Booker
32630c27cc
Added milestones database table (#16300)
refs
https://www.notion.so/ghost/Marketing-Milestone-email-campaigns-1d2c9dee3cfa4029863edb16092ad5c4?pvs=4

- added new `milestones` database table to be able to store created
milestones.
- updated schema and tests to match.
2023-02-22 10:46:43 +02:00
Naz
ad6097b695
Refactored post's extra-attributes to use options
refs https://github.com/TryGhost/Toolbox/issues/522

- Having simpler method signature makes it easier to use it in different context - needed for changes in public resource repository
- TLDR of the changes - reduced parameter 'frame.options' -> 'options'
2023-02-22 15:46:56 +08:00
Ghost CI
d3563d9c06 Merged v5.35.1 into main 2023-02-21 15:36:02 +00:00
Simon Backx
461634c57c Fixed snapshot tests 2023-02-21 16:15:00 +01:00
Simon Backx
b665b1a3cc Implemented email analytics retrying (#16273)
fixes https://github.com/TryGhost/Team/issues/2562

New event fetching loops:
- Reworked the analytics fetching algorithm. Instead of starting again
where we stopped during the last fetching minus 30 minutes, we now just
continue where we stopped. But with ms precision (because no longer
database dependent after first fetch), and we stop at NOW - 1 minute to
reduce chance of missing events.
- Apart from that, a missing fetching loop is introduced. This fetches
events that are older than 30 minutes, and just processes all events a
second time to make sure we didn't skip any because of storage delays in
the Mailgun API.
- A new scheduled fetching loop, that allows us to schedule between a
given start/end date (currently only persisted in memory, so stops after
a reboot)

UI and endpoint changes:
- New UI to show the state of the analytics 'loops'
- New endpoint to request the analytics loop status
- New endpoint to schedule analytics
- New endpoint to cancel scheduled analytics
- Some number formatting improvements, and introduction of 'opened'
count in debug screen
- Live reload of data in the debug screen

Other changes:
- This also improves the support for maxEvents. We can now stop a
fetching loop after x events without worrying about lost events. This is
used to reduce the fetched events in the missing and scheduled event
loop (e.g. when the main one is fetching lots of events, we skip the
other loops).
- Prevents fetching the same events over and over again if no new events
come in (because we always started at the same begin timestamp). The
code increases the begin timestamp with 1 second if it is safe to do so,
to prevent the API from returning the same events over and over again.
- Some optimisations in handing the processing results (less merges to
reduce CPU usage in cases we have lots of events).

Testing:
- You can test with lots of events using the new mailgun mocking server
(Toolbox repo `scripts/mailgun-mock-server`). This can also simulate
events that are only returned after x minutes because of storage delays.
2023-02-21 16:00:32 +01:00
Simon Backx
923c522778
Implemented email analytics retrying (#16273)
fixes https://github.com/TryGhost/Team/issues/2562

New event fetching loops:
- Reworked the analytics fetching algorithm. Instead of starting again
where we stopped during the last fetching minus 30 minutes, we now just
continue where we stopped. But with ms precision (because no longer
database dependent after first fetch), and we stop at NOW - 1 minute to
reduce chance of missing events.
- Apart from that, a missing fetching loop is introduced. This fetches
events that are older than 30 minutes, and just processes all events a
second time to make sure we didn't skip any because of storage delays in
the Mailgun API.
- A new scheduled fetching loop, that allows us to schedule between a
given start/end date (currently only persisted in memory, so stops after
a reboot)

UI and endpoint changes:
- New UI to show the state of the analytics 'loops'
- New endpoint to request the analytics loop status
- New endpoint to schedule analytics
- New endpoint to cancel scheduled analytics
- Some number formatting improvements, and introduction of 'opened'
count in debug screen
- Live reload of data in the debug screen

Other changes:
- This also improves the support for maxEvents. We can now stop a
fetching loop after x events without worrying about lost events. This is
used to reduce the fetched events in the missing and scheduled event
loop (e.g. when the main one is fetching lots of events, we skip the
other loops).
- Prevents fetching the same events over and over again if no new events
come in (because we always started at the same begin timestamp). The
code increases the begin timestamp with 1 second if it is safe to do so,
to prevent the API from returning the same events over and over again.
- Some optimisations in handing the processing results (less merges to
reduce CPU usage in cases we have lots of events).

Testing:
- You can test with lots of events using the new mailgun mocking server
(Toolbox repo `scripts/mailgun-mock-server`). This can also simulate
events that are only returned after x minutes because of storage delays.
2023-02-20 16:44:13 +01:00
Steve Larson
2d84b7d990
Upgraded got package from v9.6.0 to v11.8.6 (#16261)
Refs TryGhost/Team#2459
-upgraded got from v9.6.0 to v11.8.6 to support following redirects (and
other fixes)
-got v12+ requires ESM, so we do not want to upgrade further at this
time
-required changes to a few libraries that use externalRequests
-mention discovery service tests updated to test for follow redirects
2023-02-20 09:33:11 -06:00
Simon Backx
52a26a7f80 Removed unreliable upload test
no issue

This test failed randomly. It is hard to improve the test without defeating the purpose of the test.
2023-02-20 13:58:01 +01:00
Aileen Booker
386298310b
Added timeout for running milestone service (#16290)
no issue

- Instead of running milestone service directly on boot, set a random
timeout of 0-4 days to run after boot
- Updated tests
- Service is still behind a beta flag
2023-02-20 14:14:06 +02:00
Simon Backx
ec3f262ea4 Increased scheduling default test timeout 2023-02-17 13:29:07 +01:00
Fabien "egg" O'Carroll
8908a51547 Improved verification logic for Mentions
refs https://github.com/TryGhost/Team/issues/2550

By using cheerio to parse the HTML we can correctly look for elements
which use the target URL as the href attribute, rather than doing a
plaintext search. This closer to what the spec says.
2023-02-17 19:07:11 +07:00
Fabien 'egg' O'Carroll
101cabb188
Added separate job service for webmentions (#16285)
[Added initial mentions-jobs
service](3656190114)

This is the result of running `cp -r jobs mentions-jobs` in the services
directory.

[Waited for mentions-jobs queue before
shutdown](2bb1a12a89)

This matches the functionality of the existing jobs service where we
will wait
for jobs to complete before closing the process.

[Used mentions-jobs service in the mentions
service](4e4f9fdd00)

This ensures that any delays in the mentions jobs queue does not effect
other
parts of the application.
2023-02-17 13:05:36 +01:00
Daniel Lockyer
2edfd8b8e0
Increased timeout to allow for delays in browser e2e test
- this test was randomly timing out, and the tests above allow for a
  greater timeout before failing
2023-02-17 12:13:09 +01:00
Aileen Booker
2f57e95a5d
Slack notifications service for Milestones behind flag (#16281)
refs
https://www.notion.so/ghost/Marketing-Milestone-email-campaigns-1d2c9dee3cfa4029863edb16092ad5c4?pvs=4

- Added a `slack-notifications` repository which handles sending Slack
messages to a URL as defined in our Ghost(Pro) config (also includes a
global switch to disable the feature if needed) and listens to
`MilestoneCreatedEvents`.
- Added a `slack-notification` service which listens to the events on
boot.
- In order to have access to further information such as the reason why
a Milestone email hasn't been sent, or the current ARR or Member value
as comparison to the achieved milestone, I added a `meta` object to the
`MilestoneCreatedEvent` which then gets accessible by the event
subscriber. This avoid doing further requests to the DB as we need to
have this information in relation to the event occurred.

---------

Co-authored-by: Fabien "egg" O'Carroll <fabien@allou.is>
2023-02-17 12:59:18 +02:00
Steve Larson
1e00bd2af9 Handled empty content when sending webmentions
refs https://ghost.slack.com/archives/C02G9E68C/p1676564978732119

- `cheerio` errors when trying to parse `null`
2023-02-17 10:46:48 +01:00
Naz
f27c7ba044
Fixed variable name typo 2023-02-17 17:27:51 +08:00
Simon Backx
8dfa490638
Added outbound link tagging for web posts (#16201)
fixes https://github.com/TryGhost/Team/issues/2433

- Moved all outbound link tagging code to separate OutboundLinkTagger
- Because a site can easily enable/disable this feature, we don't store
the ?refs in the HTML but add them on the fly for now in the Content
API.
2023-02-16 11:26:35 +01:00
Fabien "egg" O'Carroll
c6b90b6ac7 Added verified property to Mentions Admin API
closes https://github.com/TryGhost/Team/issues/2551

Rather than blindly passing all data through the API we explicitly include each
new property. This allows us to make changes to the core entities without
affecting the API. The verified property is being added now to give design the
ability to display these mentions differently.

We also needed to include the verified property in the return value of toJSON,
this was missed as part of the original entity changes
2023-02-16 14:31:05 +07:00
Aileen Nowak
a400353881 Renamed milestone-emails to milestones
no issue

- The way we're going to implement milestones diverged from the original idea of handling email sending within the milestone-emails package, as we'll be sending events instead and will utilise the StaffService to listen to them and send the emails
- This renames the package as well as the service in core itself and all relevant tests
2023-02-15 10:43:05 +02:00
Aileen Booker
41e4132a17
Refactored milestones service to preserve API instance (#16272)
no issue

- For better testability with in-memory repository, refactor the
milestones service to preserve the API instance
- Fetching the information about Stripe live mode from Stripe service
was causing difficulties when testing. As a workaround we switched to
reading the live mode keys and determine it that way.

---------

Co-authored-by: Fabien "egg" O'Carroll <fabien@allou.is>
2023-02-15 08:43:40 +02:00
Ronald Langeveld
f9eeec10db Updated email-preview test snapshot
no issue

- fixed up tests failing from the outdated email snapshot.
2023-02-15 11:21:37 +08:00
Ronald Langeveld
c651f5b10f
Added verified mention column (#16264)
refs https://github.com/TryGhost/Team/issues/2549

- Added `verified` column to mentions table
- Added a new migration
- Updated schema
- Updated integrity test
2023-02-14 15:46:07 +08:00
Kevin Ansfield
c489343831
Fixed saving and rendering of Lexical posts containing new cards
closes https://github.com/TryGhost/Team/issues/2558

- bumped `kg-lexical` packages so we're working with latest suite of default nodes and renderer
- added a `render()` method directly to our `lexicalLib` object
  - allows us to pass through all of Ghost's config for image transforms etc in one place rather than every time we want to render something
2023-02-13 16:35:34 +00:00
Simon Backx
4e361a65fa Added webmention delete test and cleaned up webmention tests
refs https://github.com/TryGhost/Team/issues/2553

- Added webmention delete test
- Fixed dangling promise in test
- Fixed missing .persist in tests
2023-02-13 17:15:34 +01:00
Daniel Lockyer
4bca725215
Added basic unit test for cache adapter module
- this helps improve our testing capabilities of the new cache adapters
2023-02-13 17:10:42 +01:00
Aileen Booker
6f0d1b0ff9
Added milestone email service behind alpha flag (#16241)
refs
https://www.notion.so/ghost/Marketing-Milestone-email-campaigns-1d2c9dee3cfa4029863edb16092ad5c4

Added milestone email service behind a flag. The service will currently
run on boot and is meant to be scheduled soon, which should happen in
the next step. For now it's protected behind the alpha flag.
2023-02-13 16:29:01 +02:00
Simon Backx
48f9485f46
🐛 Fixed storing email failures with an empty message (#16260)
no issue

- When we receive an email failure with an empty message, the saving of
the model would fail because of schema validation that requires strings
to be non-empty.
- This adds more logging to the email analytics service to help debug
future issues
- Performance improvement to storing delivered, opened and failed emails
by replacing COALESCE with WHERE X IS NULL (tested and should give a
decent performance boost locally).
2023-02-13 15:25:36 +01:00
Fabien "egg" O'Carroll
bf6bcdbff3 Added e2e tests for updating Mentions
closes https://github.com/TryGhost/Team/issues/2552

We send a Webmention for the same URL twice, but change the contents
of the source document, and we check that the source metadata is
updated appropriately.

We should consider extending all of these tests to include featured
images and logos etc...
2023-02-13 19:09:31 +07:00
Simon Backx
77032262c4
🐛 Fixed subscriptions visible as "Active" within Ghost Admin (#16255)
fixes https://github.com/TryGhost/Team/issues/2542 
fixes https://github.com/TryGhost/Team/issues/2543 
fixes https://github.com/TryGhost/Team/issues/2544

- Hides incomplete subscriptions
- Shows Past Due subscriptions
- Fixed UI issues with 3+ subscriptions
- Fixed missing complimentary subscription when one subscription was
incomplete/inactive
- Fixed sending a paid subscription started email for incomplete
subscriptions. This change also required us to actually send the email
when the incomplete subscription eventually becomes active. So the
introduction of a new `SubscriptionActivatedEvent` made sense/was
required (because sending a SubscriptionCreatedEvent again would cause
other issues).
2023-02-13 13:07:53 +01:00
Fabien 'egg' O'Carroll
57695d03df
Fixed rate limit test (#16258)
closes https://github.com/TryGhost/Team/issues/2547

Changed the configuration for testing to be a bit more strict, by slowing down the amount of requests it can handle to give CI enough time to kick in the rate limiter. Before this, CI simply wasn't hitting the API fast enough to trigger the rate limiter.

Co-authored-by: Ronald Langeveld <hi@ronaldlangeveld.com>
2023-02-13 15:16:56 +08:00
Ronald Langeveld
4096571b8d Commented out flaky webmention test
refs https://ghost.slack.com/archives/C02G9E68C/p1675934676146349

- Commented out the rate limiter test as it proves to be flaky on CI,
  till we find a better way of implementing it.
2023-02-09 21:35:15 +08:00
Steve Larson
d20696805f
Added mentions deleted column (#16251)
refs https://github.com/TryGhost/Team/issues/2534

This is so that we can support soft deletes for Mentions.
We need to add the defaults to the model so that write continue to work.

Co-authored-by: Fabien "egg" O'Carroll <fabien@allou.is>
2023-02-09 16:19:07 +07:00
Ronald Langeveld
ce567b9816
Added Mentions email notification settings (#16245)
closes https://github.com/TryGhost/Team/issues/2526

- Mention emails can now be toggled inside staff user' profiles, if they
have the webmention flag enabled on their Ghost site.
- Removed the flag dedicated to webmention email notifications and is
now handled by the `webmention` flag.
- Does not send email notification if `webmention` flag is not enabled.
- Updated tests.

---------

Co-authored-by: Fabien "egg" O'Carroll <fabien@allou.is>
2023-02-09 17:03:03 +08:00
Simon Backx
ea2c69565f
Moved email event fetching to main thread (#16231) 2023-02-09 09:36:39 +01:00
Ronald Langeveld
30dc2a8228
Added mention_notifications column (#16242)
refs https://github.com/TryGhost/Team/issues/2526

- created a migration for a new boolean column in users that would
determine if the staff user gets an email when the publication receive a
new mention.
2023-02-09 16:15:54 +08:00
Ronald Langeveld
9fc13cfe65
Implemented mentions rate limiter (#16223)
closes https://github.com/TryGhost/Team/issues/2419

- adds a rate limiter implementation to the mentions receiving
endpoint.
- Current configuration is `{"minWait": 10,
             "maxWait": 100,
             "lifetime": 1000,
             "freeRetries": 100}` which is still very open and almost unrestricted. 
- currently makes use of database storage to track the limits, but can be relatively easily swapped out to something eg Redis should we find this endpoint getting hit too often and maliciously.
2023-02-09 14:57:48 +08:00
Steve Larson
6c97edec25
moved sending mentions to job (#16234)
refs TryGhost/Team#2523
-implemented jobsService within mention sending service
-updated tests so we don't need to sleep
2023-02-08 16:29:12 -06:00
Steve Larson
5ccb778ac8
enabled always sending mentions (#16227)
refs TryGhost/Team#2507
-removed labs check from sending service
-labs flag controls seeing mentions in admin and receiving mentions
2023-02-03 14:52:20 -06:00