closes https://github.com/TryGhost/Team/issues/2483
- the button is toggled depending on `stripeConnectIntegrationToken` which is a calculated value
- however, this calculated value isn't reset when Stripe is disconnected right away without closing the modal
- the reset action was already available and it's now passed to `StripeSettingsForm`, so that it can be called when Stripe is disconnected
refs https://github.com/TryGhost/Team/issues/2713
- currently, it's not clear enough that if the theme activation was successful
- highlighting "successful" makes it easier to know the main action was successful even though there are errors to fix
refs https://github.com/TryGhost/Team/issues/2795
- this helps reduce conflicts when multiple PRs are coming in to add new
languages
- I'd also like to enforce alphabetically sorted keys but I can't figure
out the magic eslint incantation yet
refs TryGhost/Team#2401
- Previously, if you opened a new post using the plus button on the sidebar while on a different post's analytics page, the breadcrumbs would point back to Analytics instead of the Posts list view. Going back to Analytics brought you back to the analytics page for the post you just created, which wasn't populating since it likely hadn't been published yet.
- This commit fixes the issue by setting fromAnalytics = false if transitioning to a new Post, even if the previous route was the analytics page.
refs https://github.com/TryGhost/Team/issues/2617
- some sites have SVG favicons and it made bookmark card favicons look broken in some email clients as the SVG support is poor across email clients
- this fix simply hides the SVG favicons in emails
no issue
This change makes the verification trigger behave closer to the original idea. As well as having a threshold to trigger email verification, the trigger should also rely on the number of "organic" / verified members. This means that large publications will not trigger the verification flow when they import members, but spammers / suspicious sites worth verifying will not have the organic member count to allow for larger imports.
refs
https://www.notion.so/ghost/Marketing-Milestone-email-campaigns-1d2c9dee3cfa4029863edb16092ad5c4?pvs=4
- Added email template for milestones with using a configuration file
for different member milestone values, as we're sending different
content for each one
- Implement sending the email to users who have
`milestone-notifications` enabled, currently still behind a flag
Co-authored-by: Peter Zimon <peter.zimon@gmail.com>
refs https://github.com/TryGhost/Team/issues/2754
Previously, we didn't have any backup for storing source site title and it could have stored as empty if missing. This change ensures the source site title is stored as site host instead as fallback if not present.
refs https://github.com/TryGhost/Team/issues/2754
The name of the setting used for storing last timestamp of mentions report is `last_mentions_report_email_timestamp`, but in the code we incorrectly camel-cased it which meant we were not reading from and writing to the correct setting, and ended up sending a mentions report every few hours every time the job ran.
refs TryGhost/Team#2667
- Added notification check before checking that user successfully unsubscribed from all newsletters. It helps to make the test more stable
no issue
The migration was added to 5.39, but this was not correct. It should have been added to 5.40. This commit fixes moves the migration to the right folder. Running the migration twice locally will skip it because it detects the table is already correct.
- without this, Node will try and resolve the domain name but local DNS
resolvers can take a while to timeout, which causes the tests to timeout
- `nodemailer-direct-transport` calls `dns.resolveMx`, so if we stub that
function and return an empty array, we can avoid any real DNS lookups
refs TryGhost/Team#2755
Previous change for outbound tagging had incorrectly changed the internal links ref for newsletter to include domain. This change fixes it back to use newsletter name for internal links.
refs https://ghost.slack.com/archives/C02G9E68C/p1679055933852679
- The posts api is used in the update-check job. This also uses the email wrapper, which needs the email-service
- The email service is not initalized in the worker threads, this causes an error
- Adds a check, because this part is optional in worker threads
no issue
When opening the 'body style' dropdown in the newsletter settings, the dropdown was not completely visible. The next 'liquid' tab covered the dropdown. This makes some adjustments to the overflow of the liquid containers and z-index of the buttons and containers to fix that.
It also removes overflow-y: auto from one of the containers because it cropped the shadow if the dropdown, while that container could never scroll (parent is scrollable).
fixes https://github.com/TryGhost/Team/issues/2672
- Moves the feature from behind the flag
- Also hides the comment CTA for email only posts
Co-authored-by: Sanne de Vries <sannedv@protonmail.com>
refs https://github.com/TryGhost/Toolbox/issues/523
- During import process of content files the files from the root directory were also copied over. This is causing chaos in the root of content folder with files that only needed for data import. For example, the csv files needed for Revue import were also copied over by "file importer" even though those do not belong to any content.
- The approach of filtering on a "handler" level was taken over filtering in the import manager due to how our globing patterns work. See a previous commit with reverted previous fix for more context.
refs https://github.com/TryGhost/Toolbox/issues/523
- The reverted fix did not take into account the "original path" of the
files would be truncated. This path has to be full relative to the root
of the zip to later be used during importer url substitution logic.
- This reverts commit 831a76505c.
fixes https://github.com/TryGhost/Team/issues/2734
With WebMentions flag on, navigating to `/ghost/#/mentions` on Admin when there are no mentions gives a 400 error page instead of the intended empty state.
refs https://github.com/TryGhost/Ghost/issues/15502
- this commit migrates Portal from CRA to Vite, as it brings the
package more inline with the direction we're going in terms of tooling
for builds
- the bulk of the changes here are just config related to get things
working with Vite, and then cleaning up all the CRA boilerplate
refs https://github.com/TryGhost/Ghost/issues/15502
- this adds the scaffolding for enabling i18n translations within Ghost
core
- also adds `yarn translate:ghost` as an option to the `i18n` package
- the locale is currently hardcoded to `en` so we don't utilize the
translations until we're ready
- `compilerOptions.baseUrl` is nice but it means we end up with two
different styles of import paths for project files - relative and from
`src/`
- to fix this, we're going to standardize on relative imports and remove
the ability to import relative to the top-level project folder
- this removes the jsconfig line and fixes all existing imports
This moves the shared data between all templates into a method so that
it's calculated in a single place, as well as exposing separate render
methods for HTML and Text. We've also added some new custom helpers
for use in the handlebars templates
This package contains the business logic for the sending of mention
report emails, it could eventually be included in the webmentions
package I think, but has been kept separate for now in favour of
smaller packages.
refs https://ghost.slack.com/archives/C025584CA/p1678886661494989?thread_ts=1678886465.260379&cid=C025584CA
- if we're running a prerelease, our usual format is `<major>.<minor>.<patch>-pre.<commit hash>.<build>`
- if you wanna see what commit it was built on, you have to copy and
paste the hash into a GitHub URL
- this provides an easier route to the commit by detecting if the
version is our special format and links direct to GitHub if so
Because there is no guarantee about a daily job running exactly once a
day, we need to store the last time that the email was sent, so that we
can refrain from sending one if it's been less than a day since the
last.
A setting has been used for this as we don't currently have a pattern
for it, we might want to consider moving this to some kind of cache
based solution in future. This has been added as a core setting so that
we don't expose it via the API.
The setting is stored as a number to allow us to store value as unix timestamp.
---------
Co-authored-by: Rishabh <zrishabhgarg@gmail.com>
- we use ES6 `import` for all other imports so this standardizes the
React package to also use import vs require
- merged with other imports from the react package if present
Refs https://github.com/TryGhost/Team/issues/2612
- the recent posts were used to link to the editor, now they're linked to their analytics page
- now that we have an analytics screen for each post it makes sense to link to that screen where possible
refs https://github.com/TryGhost/Team/issues/2612
- member event feeds previously had links to posts that opened the
front-end post URL in a new tab
- now that we have an analytics screen for each post it makes sense to
link to that screen where possible because it allows drill-down into
site performance
refs https://github.com/TryGhost/Toolbox/issues/524
- The 'mediaInliner' beta flag is not used for anything anymore, so there's no need to keep it around. The only surface of the feature is the `POST /db/media/inline` endpoint that will be used through internal tooling mostly and won't be accessible through the Admin UI.
refs https://github.com/TryGhost/Toolbox/issues/524
- We no longer need the UI of any kind for the media inliner feature as
the inliner endpoint will be used mostly for internal purposes through
automated tooling
- This reverts commit 901485c47b.
refs https://github.com/TryGhost/Toolbox/issues/524
- The endpoints is for internal (non-documented) usage at the moment and is fine without the labs flag.
- This should allows us to test the inlining on production sites sooner
fixes https://github.com/TryGhost/Team/issues/2725
- Added to emails if labs flag enabled, comments enabled and comment CTA button enabled
- Links to comment section
- Design and styling not added yet
fixes https://github.com/TryGhost/Team/issues/2724
This change also includes new snapshots for email sending (similar for email previews, but this time for the real emails to make sure we catch changes).
fixes https://github.com/TryGhost/Team/issues/2705
- Added showPostTitleSection to newsletter model in admin
- Wired up UI to admin model so it saves to the database
- Implemented showPostTitleSection in newsletter preview and added some
minor temporary css styling
- Implemented showPostTitleSection in newsletter template in backend,
and added some extra CSS styling to fix spacing
no issue
When using `getLazyRelation` on an optional relation that is not set, it
will return a newly created model instead of a model from the database.
- Adds a new require option to `getLazyRelation`, that throws an error
if the relation is not set (off by default to match existing use cases)
- This caused a bug (not visible because we always pass a newsletter id)
in email previews, where when the newsletter id was not explicitly set,
it would use `newsletter = (await post.getLazyRelation('newsletter')) ??
(await this.models.Newsletter.getDefaultNewsletter());`, which always
returned the first one, and could return a newly initiated newsletter
with all properties set to undefined.
- Some page snapshots are altered by this, because the usage of
`getLazyRelation` on a post no longer sets the email relation to some
new model.
refs
https://www.notion.so/ghost/Marketing-Milestone-email-campaigns-1d2c9dee3cfa4029863edb16092ad5c4?pvs=4
- When milestones will be activated we would send out emails to users
that are way above the achieved milestone, as we didn't record
milestones before
- The plan is to implement a 0 milestone and don't send an email for
achieving those and also add all achieved milestones in the first run
until a first milestone is stored in the DB, then increment from there.
- This change takes care of two cases:
1. Milestones gets enabled and runs initially. We don't want to send
emails unless there's already at least one milestone achieved. For that
we add a 0 milestone helper and add a `initial` reason to the meta
object for the milestone event, so we can choose not to ping Slack and
also disable email sending for all milestones achieved in this initial
run.
2. All achieved milestones will be stored in the DB, even when that
means we skip some. This introduces the `skipped` reason which also
doesn't send emails for the skipped milestones, but will do for
correctly achieved milestones (always the highest one).
- Added handling for slack notifications to not attempt sending when
reason is `skipped` or `initial`
The service.js files are rarely unit tested because they are
essentially glue code which acts as a singleton and doesn't use
dependency injection, this lil cocktail makes it difficult to stub
dependencies, as well as to call the init method more than once in a
test suite. This change should give us improved unit test coverage
no issue
The Stripe Mocker mocks the Stripe API in memory, to make it much easier
to test subscription flows. Currently it is more a POC to see if it
works well. It probably needs a bit more work to support more scenarios.
- Added new tests for the subscription stats endpoint for 3D secure +
free trial flows using the new Stripe Mocker
- Updated members admin api tests to use Stripe Mocker (+ added new test
for deleting members with Stripe cancellation)
- Some tests called mockStripe at the beginning, but that method did
nothing apart from disabling network (which is the default now), then
they mocked Stripe inside the tests file... so I've removed those
because those conflict with the new mocker that is enabled when calling
mockStripe. We'll need to port those over later.
fixes https://github.com/TryGhost/Team/issues/2607
When a free trial converts to a paid subscription, and increases the MRR, it just creates a 'updated' paid subscription event.
To fix this, we need to count updated events that didn't change plan but do have a positive MRR. As an extension we could also check if the MRR change matches the expected MRR for the corresponsing Stripe plan, but that requires a more complex condition check (because for yearly subscriptions we need to convert to monthly), I don't think that is required here.
fixes https://github.com/TryGhost/Team/issues/2644
A 3D secure payment first has a status of incomplete, then active.
With the current logic, this creates 2 MemberPaidSubscriptionEvents:
- `created` with mrr_delta of 0
- `active` with mrr_delta of 5
We need to also count 'active' events. And to complement that, also 'inactive' events to make sure we balance out in rare cases.
refs https://github.com/TryGhost/Team/issues/2489
- adds attribution title and url for new free members and paid subscriptions to email alert
---------
Co-authored-by: Peter Zimon <peter.zimon@gmail.com>
- this cleans up all imports or variables that aren't currently being used
- this really helps keep the tests clean by only allowing what is needed
- I've left `should` as an exemption for now because we need to clean up
how it is used
- this is generally an anti-pattern in tests and leads to flaky
behaviour when tests are ran on different machines/loads
- this is currently unused so it is an easy removal
- we have our own class in order to add the `hasRegisteredListener`
function
- this commit refactors the implementation to use the class syntactic
sugar, which means we get better editor autocomplete
- this shouldn't change the functionality
fixes https://github.com/TryGhost/Team/issues/2560
When an email fails, and you reschedule the post, the error dialog was
shown (from the previous try). The retry button on that page allowed you
to retry sending the email immediately, which could be very confusing.
- The email error dialog is no longer shown for scheduled emails
- The email status is no longer polled for scheduled emails
- Retrying an email is not possible via the API if the post status is
not published or sent
- Added some extra snapshot tests
- When retrying an email, we immediately update the email status to
'pending' to have a better API response (instead of still returning
failed).
- Disabled email sending retrying in development (otherwise very hard to
test failed emails if it takes 10 mins before it gives up automatic
retrying)
closes https://github.com/TryGhost/Team/issues/2531
This commit fixes the issue where non-canonical URLs are included in the
XML sitemap, leading to poor SEO for our user's sites. The solution
implemented is to exclude any page or post that specifies a canonical
URL in its metadata from the sitemap.
To achieve this, a condition has been added to the 'addUrl' method,
which checks for the existence of a canonical URL in the metadata of the
resource being added to the sitemap. If a canonical URL is present, the
resource is excluded from the sitemap.
With this fix, our user's sites will have better SEO and improved search
engine visibility.
fixes https://github.com/TryGhost/Team/issues/2666
- Somehow occurrences of `&map_` got replaced with `↦`
- Disables escaping &, ', " and other HTML characters when not needed
(escaping is already handled by mobiledoc/lexical)
- Bumps unit test coverage of link replacer to 100%
no issue
- Nock doesn't support multiple calls to enableNetConnect -> only the last one counts. This fixes that issue.
- Some tests interacted directly with nock instead of using the mockManager to restore everything.
no issue
- vscode started adding warnings to all uses of decorators in Admin which was rather annoying
- added `jsconfig.json` with the necessary compiler options and an exclude list to keep performance happy
refs https://github.com/TryGhost/Toolbox/issues/524
- The "tableName" property is not present on the bookshelf model instance - it was a mistake assumption using it for logs. In case the logs are too ambiguous we could figure pass around model names separately into the "inlineSimpleFields" method
refs https://github.com/TryGhost/Toolbox/issues/524
- Mobiledoc from a migrated content could contain html/markdown cards that could have a variety of different resource <> url pairs. To avoid complex logic and parsing html/md going with a simplest approach - matching external content URLs purely based on provided domains. This gives useful enough of a tool to migrate external content for a specific service (e.g Revue)
- In cases where the content is not supported the fetching will fail with a message and move on to the next match - which is a reasonable behavior for a migration tool
refs https://github.com/TryGhost/Toolbox/issues/524
- Fetching media from a remote server is an expensive network operation. Given there's probability for the content to reuse the same image in different posts or in multiple places, we could save on extra fetches by adding caching to the remote media fetch method
https://github.com/TryGhost/Toolbox/issues/523
- During import process of content files the files from the root directory were also copied over. This is causing chaos in the root of content folder with files that only needed for data import. For example, the csv files needed for Revue import were also copied over by "file importer" even though those do not belong to any content.
- Any content import files - images, media, files, should be in according folders in the imported zip file. The root files in the base zip directory are for data-related imports
fixes https://github.com/TryGhost/Team/issues/2611
The old email flow is no longer used since we introduced the email stability flow. This commit removes the related code and tests. The general test coverage decreased a bit as a result, because the old email flow probably had a high test coverage. The new flow is in separate packages, so it couldn't contribute to a higher test coverage (but it does have 100% unit test coverage).
fixes https://github.com/TryGhost/Team/issues/2683
When sending a newsletter with a replacement that has a fallback, the
replacement only happens in the HTML version of the newsletter. The
plaintext version isn't replaced.
This commit fixes the issue and adds some tests to make sure it doesn't
happen again.
The cause of the issue was that we used the original matched Regex text
to replace. But that was calculated on the HTML version, so double
quotes were encoded. This change updates the generated 'token' regex to
also match on both a double quote as the escaped double quote.
refs. https://github.com/TryGhost/Team/issues/2489
- Alignment of avatar in free and paid signup email template was off
- Paragraphs was mixed in free signup email template which resulted in an unwanted space
refs https://github.com/TryGhost/Team/issues/2667
Some tests still accessed the internet. Now network access is disabled
by default. This change also introduces two helper methods related to
networking (mocking Slack and Mailgun).
This fixes two unreliable tests:
- Staff service was accessing a Slack test API -> timeout possible
- MentionSendingService was trying to send webmentions for every post
publish/change -> possible timeouts and job issues
refs 6460522352
- these were both bumped around the time that Playwright browser tests
started going awry, so they may be the cause of some random failures
no issue
Disables email sending retrying in test mode by default. This is to prevent test timeouts and to make testing more reliable in case where we manually let a batch fail.
refs https://github.com/TryGhost/Toolbox/issues/524
- We need to be able to inline external media in all internal resources: tags, users, post's meta fields.
- This change adds media inlining logic to all these resources
- these models should be as lightweight as possible to require, so we
need to measure how long they take
- this has already found a couple of models which need optimizing
refs https://github.com/TryGhost/Toolbox/issues/523
- The test is useful for future iterations of the response format and as a quick reference on which parameters the media inlining endpoint accepts.
refs https://github.com/TryGhost/Toolbox/issues/523
- Media inlining is a long running piece of logic that suits perfectly for a job. It is not as critical to be offloaded into a separate thread at this point, so leaving it as "inline" while the feature is in experimental stage.
refs https://github.com/TryGhost/Toolbox/issues/523
- This is a first pass media inliner going through all posts and checking to inline media from specified domains
- As a working copy the inliner looks for image content from Revue and Substack
refs: https://github.com/TryGhost/Toolbox/issues/389
Instead of logging errors, this will warn when adding a duplicate URL in the test environment.
At the moment, this is happening a lot in the test suite. While we also need to fix the root cause of this so we're not erroring in the product, it's a massive amount of spam in the logs when running the test suite which could prevent us from finding other errors which are causing issues.
refs: https://github.com/TryGhost/Toolbox/issues/389
This removes many error logs when the end-to-end test suite is run with the log-level set to error. Many errors are intentional, so the resolution is typically to stub the error log function and assert that it would have been called.
refs: https://github.com/TryGhost/Toolbox/issues/389
The newsletter fixtures no longer errors when accessing the test image, and tests which intentionally error now stub the logging call
refs https://github.com/TryGhost/Toolbox/issues/523
- When importing a zip with Revue data and no "files" or "media" the importer blew up. This is due to a bug where the files from the root directory are still being copied over (the should not be!)
- Added protective extra ".data" check to make sure code doest not blow up when there's no data property to be processed
refs https://github.com/TryGhost/Toolbox/issues/524
- Having a button in Admin UI allows to manually trigger media inlining job once import has been processed.
- Added a "start" button hidden behind labs calling Admin API endpoint that starts external media inlining job.
refs https://github.com/TryGhost/Toolbox/issues/524
- This is groundwork to build up on when adding media inlining functionality. This service will be used by the API to trigger post's content processing and media inlining.
refs https://github.com/TryGhost/Toolbox/issues/524
- This flag will be used to test the trigger to external media inliner job
- The Admin UI is still unclear so sticking it behind the flag for some experimentation before shipping the feature
closes https://github.com/TryGhost/Team/issues/2400
- used semver comparison to detect when the app version is less than the content-version header in any API response to toggle `upgradeStatus.requiresRefresh` that is used to conditionally show the upgrade banner
- only works on minors as we don't store the full Ghost patch version in `config.APP.version`
refs https://github.com/TryGhost/Toolbox/issues/523
- When migrating or importing ZIP files into Ghost there's often a need to include document files.
- When document files are present in the imported zip file they are now copied across and processed along with the rest of import files: json, images, csvs, etc.
- The importer also searches for use of the document files in the imported "posts" substituting the links with local ones
- The document files importer recognizes media files inside of "files" or "content/files" folders present in the zip.
- The supported media file extensions are same as for file upload widget:
".pdf",".json",".jsonld",".odp",".ods",".odt",".ppt",".pptx",".rtf",".txt",".xls",".xlsx",".xml"
with following content-types:
"application/pdf", "application/json", "application/ld+json", "application/vnd.oasis.opendocument.presentation", "application/vnd.oasis.opendocument.spreadsheet", "application/vnd.oasis.opendocument.text", "application/vnd.ms-powerpoint", "application/vnd.openxmlformats-officedocument.presentationml.presentation", "application/rtf", "text/plain", "application/vnd.ms-excel", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", "application/xml", "application/atom+xml"
refs https://github.com/TryGhost/Toolbox/issues/523
refs c2534e3c86/packages/mg-assetscraper/lib/AssetScraper.js (L14-L16)
refs https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types/Common_types
- Importer needs to process and recognize document files like pdfs, presentations etc to be able to import them into sites file storage.
- The handler allows a new root directory "files" to place imported documents
- The handler adds validation and processing for following file extensions:
".pdf",
".json",
".jsonld",
".odp",
".ods",
".odt",
".ppt",
".pptx",
".rtf",
".txt",
".xls",
".xlsx",
".xml"
- With following content types:
"application/pdf",
"application/json",
"application/ld+json",
"application/vnd.oasis.opendocument.presentation",
"application/vnd.oasis.opendocument.spreadsheet",
"application/vnd.oasis.opendocument.text",
"application/vnd.ms-powerpoint",
"application/vnd.openxmlformats-officedocument.presentationml.presentation",
"application/rtf",
"text/plain",
"application/vnd.ms-excel",
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"application/xml",
"application/atom+xml"
refs https://github.com/TryGhost/Toolbox/issues/523
- To support generic "file" import handling the content file handler needs more configuration options for properties like type, extensions, content path etc. This refactor makes the handler configurable and reusable for any type of file import
refs https://github.com/TryGhost/Toolbox/issues/523
- We need to process generic files like .pdf, .md, etc. on the importer "handler" stage.
- The media handler can process more than just media files after few refactorings. Renaming it to a generic content file handler indicates it can process any type of content file.
- In future we can substitute the built-in "images" import handler with this generic content file handler.
refs https://github.com/TryGhost/Toolbox/issues/523
- Making "content/media" a strict requirement for the import folder structure breaks the importer (glob library does not expand a subdirectory pattern). Unless it become a strict requirement we can use general content directory matching in combination with file extensions.
refs https://github.com/TryGhost/Toolbox/issues/523
- The naming was referring to "image", which is a leftover from a copy-paste.
- It's much nicer to read a skimmable, columnar urlJoin method parameters instead of one long line
refs https://github.com/TryGhost/Toolbox/issues/523
- When migrating or importing ZIP files into Ghost there's often a need to include more than just post content and images.
- When media files are present in the imported zip file they are now copied across and processed along with the rest of import files: json, images, csvs, etc.
- The importer also searches for use of the media files in the imported "posts" substituting the links with local ones
- The media importer recognizes media files inside of "media" or "content/media" folders present in the zip. The supported media file extensions are same as for media upload widget:
".mp4", ".webm", ".ogv", ".mp3", ".wav", ".ogg", ".m4a"
with following content-types:
"video/mp4", "video/webm", "video/ogg", "audio/mpeg", "audio/vnd.wav", "audio/wave", "audio/wav", "audio/x-wav", "audio/ogg", "audio/mp4", "audio/x-m4a"
refs https://github.com/TryGhost/Toolbox/issues/523
- This is ground work before introducing a "media" content type importer
- Previous "image" file name was not describing well what the importer was capable of doing
closes https://github.com/TryGhost/Team/issues/2403
- Fixes the links in `list-item.hbs` rendereing the url filter params with brackets
eg `['postid']` where the filter UI requires the filter params not to
have brackets to allow it to render with the filter pre-selected.
- The API handles both with and without brackets.
fixes https://github.com/TryGhost/Toolbox/issues/480
- this is a bit tricky because the files are a mess but I think this
covers everything and deduplicates having multiple .gitignore files
across the repo so we can focus on keeping the definitions in one
fixes https://github.com/TryGhost/Team/issues/2625
- Adds an unique option to the mentions API. Enabling this will only
return the latest mention from each source.
- The frontend can fetch the related sources for each page by doing an
extra request to the mentions API.
refs https://github.com/TryGhost/Toolbox/issues/523
- We need to be able to use different storage mechanisms when importing different types of content
- Having the storage passed in using constructor DI allows to have more flexible storage mechanism in the Images importer (soon to become a generic file importer)
refs https://github.com/TryGhost/Toolbox/issues/523
- The class syntax would allow swapping out the storage mechanism in the importer making it universal to use with other file types like media or generic files.
Refs https://github.com/TryGhost/Team/issues/2400
- Adds a banner to the admin to indicate that a new version is available
- This is just the UI that hasn't been wired up to the actual version check yet
refs https://github.com/TryGhost/Toolbox/issues/523
- url-utils module is actively being stripped down from usages across codebase. The "urlUtils.STATIC_IMAGE_URL_PREFIX" and "store.staticFileURLPrefix" are the same values - the main difference is with the value coming from store we can make this module more generic for any store type (images, media, files, etc)
refs https://github.com/TryGhost/Toolbox/issues/523
- The "importer/index.test.js" test suite is testing more than it should. ImageHandler test suite section is one of the examples of test cases that should live in a separate file.
- Having these tests in different files makes it easier to reason about coverage and extract to it's own packages.
refs https://github.com/TryGhost/Toolbox/issues/523
- When a zip file is imported into Ghost we need to recognize and process media files with following extensions:
".mp4",".webm", ".ogv", ".mp3", ".wav", ".ogg", ".m4a"
- The media files can come from a "media" or "content/media" folder inside of zip file
refs https://github.com/TryGhost/Toolbox/issues/523
- We need media file imports. Media handler is one of the pre-processing parts that need to be done during the import process
- Media import handler is handling import files with following extensions: ".mp4",".webm", ".ogv", ".mp3", ".wav", ".ogg", ".m4a"
- The implementation is largely a copy-paste with class syntax adjustments from the "/core/server/data/importer/handles/image.js" module
- There are a lot of code similarities between media and image import handlers. The new "ImporterMedia" class could serve as a generic base class for file-storage-related imports
closes https://github.com/TryGhost/Ghost/issues/16332
Passing `SafeString` input to `asset` helper was resulting in the
exception being thrown. This meant that we couldn’t combine `asset`
helper with other helpers which produce `SafeString` e.g. `concat`
helper for string concatenation.
fixes https://github.com/TryGhost/Ghost/issues/16301
Previously, audio/x-m4a was allowed but not audio/mp4. This meant
uploads of m4a files failed in some cases e.g. Firefox on Windows.
no issue
- the Ember Data `Model` class has an `errors` property by default that
is set to a `DS.Errors` instance but the Theme model was overriding that
with an `errors` attr
- it hasn't been an issue so far but causes problems in Ember/Ember Data
3.28.x because that tries to use the `DS.Errors` interface on the
overridden attr property which then throws errors because the `errors`
attr doesn't have the right methods
refs: https://github.com/TryGhost/Toolbox/issues/389
Calling validate always uses the cache system, so this commit makes sure that the cache system is always initialised correctly by the tests.
refs https://github.com/TryGhost/Ghost/issues/15502
- in order to use the translations, strings must be wrapped in the `t`
function, which is passed through AppContext
- whilst I haven't instrumented all public strings, the vast majority
are done here and the strings have been brought into the JSON locale files using `yarn translate:generate`
The automatic_tax option is required to enable tax collection for
Stripe Checkout sessions. We've used getters here rather than an
explicit function, might wanna change that in future.
This will be used to test the use of the Stripe Tax feature during
development. We want it behind a private beta flag so that we can
stick it in production without causing problems with payment systems
for existing sites.
refs TryGhost/Team#2605
-updated unparse to look at both subscribed and subscribed_to_emails
-subscribed is for backwards compatibility
-may want to retire subscribed since we can't set from front-end
- I'm shifting some code around but the code coverage keeps flagging
- in order to not get sidetracked, I'm reducing the threshold by 1% and I'll
increase it again once we're in a better state
- these have been lingering around for a while but don't really get a
lot of love
- I'm planning on moving the migration utils/schema commands into a
separate package at some point, and these files are just cluttering
things up
- to make life easier, I'm going to inline them for now
refs https://github.com/TryGhost/Ghost/pull/9097
- this was originally added to measure the AMP parsing time but AMP is
becoming less and less important (it's disabled by default in Ghost
now)
- this also causes a lot of noise when viewing logs, and I've never
experienced an issue with AMP parsing time that we've needed to review
these logs
- therefore, this commit deletes it
closes https://github.com/TryGhost/Toolbox/issues/525
- When the update check is run as an offloaded job (in a worker thread) the errors that are handled over "logger.error" do not get bubbled up to the job manager to be handled fully. With an optional "rethrowErrors" flag it's not possible to throw errors when the update check is run as a "job" and handle as usual when run in the main thread (through the admin route trigger)
refs https://github.com/TryGhost/Ghost/issues/15502
- this adds the package to Portal and integrates it into the state we
pass around
- note: I've currently left the detection of the language
(`site.locale`) commented for now until we flesh more of the
implementation out
refs https://github.com/TryGhost/Toolbox/issues/525
refs https://github.com/TryGhost/Ghost/pull/16224
- The update check job has been failing due to uninitialized Members Service. Loading Tiers in the context of update check is not important because all we are looking for is the posts count.
- The products repository list service is also about to be replace by Tiers repository, so getting correctly initialized members service is not crucial.
no issue
- Reduced the amount of diffeerent properties by not populating a `currentARR` and `currentMembers` fields, but use a `currentValue` instead.
- The type of milestone can still be determined by its `type` property, so we actually don't need two different props here
no issue
- Switches to used newly added config values throughout the services
- Updated the `shouldSendEmail` fn to check if actual value is too far from achieved milestone as determined by the percentage setting (e. g. 998 members should not accidentally receive an email for achieving 100 members)
no issue
- Instead of having those values hard coded, make them quickly configurable
- Those values are deciding if we send an email or not, the milestones will be created either way:
- minDaysSinceImported: the amount of that days should've passed since the last member was imported
- minDaysSinceLastEmail: minimum amount of days that need to pass
- maxPercentageFromMilestone: the max percentage that the current value can be away from the achieved milestone (e. g. 999 members should not send and email for 100 members)
no issue
Before, when base data included labels for members, the random generated labels would also be generated. This prevents that, and ensures that the base-data labels are applied correctly to members.
fixes https://github.com/TryGhost/Team/issues/2601
Sets the value of outbound_link_tagging to the same value of
members_track_sources, so that is disabled by default for privacy
sensitive sites.
Also makes sure the `outbound_link_tagging` setting is available in
admin (currently excluded because it is in the analytics group)
refs https://github.com/TryGhost/Toolbox/issues/522
- Using a generic resource repository for caching purposes didn't prove to be effective as code was moved to api-level caching. There's no need to introduce extra abstraction level for simple calls like findPage when there's no extra logic to it.
refs https://github.com/TryGhost/Toolbox/issues/522
- Caching on a repository level was pretty hard to achieve with more complex models like "posts", so that approach was abandoned in favor of API-response level caching.
- Also removed use of "public-resource-repository" as it was not serving any specific purpose anymore.
no issue
- The caching has been moved down the layer - the the api-framework's "pipeline", so there's no need to add complexity to post fetching logic with repository pattern.
refs https://github.com/TryGhost/Toolbox/issues/522
- When caching responses the posts cache can create a situation where it becomes stale within the TTL period and would give stale responses to shared caches.
- Having full cache reset on 'site.changed' event makes cached content evergreen reducing the risk of caching stale content in shared caches
refs https://github.com/TryGhost/Toolbox/issues/522
- The main feature of this cache wrapper is being able to "reset" the the cache without calling the "reset" on the wrapped cache. Being able to invalidate caches without accessing the data is a feature needed to run on caches with shared environment.
- Cache invalidation happens through a special "reset time" key being added to each key when setting or getting a value, when the cache is reset the reset time is set to a new value - essentially invalidating all previously accessible values.
refs https://github.com/TryGhost/Toolbox/issues/522
- The public posts "browse" endpoint is causing the most strain on the instance performance. Caching responses with small TTL would allow to reduce the amount of request processing.
refs https://github.com/TryGhost/Toolbox/issues/522
- API-level response caching allows to cache responses bypassing the "pipeline" processing
- The main usecase for these caches is caching GET requests for expensive Content API requests
- To enable response caching add a "cache" key with a cache instance as a value, for example for posts public cache configuration can look like:
```
module.exports = {
docName: 'posts',
browse: {
cache: postsPublicService.api.cache,
options: [ ...
```
no refs
-spam prevention test was causing subsequent tests to fail randomly
-moving to the end ensures (for now) we don't interrupt other tests
-seems to be an issue with awaiting the jobservice which do concurrent
no issue
- we added support for and a demo of clicking below the editor moving focus to it in the Koenig repo but the implementation in Admin was missing
- replaced the commented-out mobiledoc handling in `<GhKoenigEditorLexical>` with the new Koenig-lexical equivalent
No issue
- These specific styles were clashing with the Tailwind classes used in the Lexical editor. As they are not used anywhere in Admin, the simplest solution is to remove them.
- there's a weird situation when we have mixed versions of the
dependency because different libraries try to compare instances
- this brings the usage up to 1.2.21 so we can fix the build for now
refs https://github.com/TryGhost/Toolbox/issues/522
- Having simpler method signature makes it easier to use it in different context - needed for changes in public resource repository
- TLDR of the changes - reduced parameter 'frame.options' -> 'options'
no issue
Previously the number of opened emails was being generated incorrectly as the number of delivered emails was being reported too high.
Also, the faker date function occasionally fails for dates which are
too close together so this switches to manually generating a date
between the two.
fixes https://github.com/TryGhost/Team/issues/2562
New event fetching loops:
- Reworked the analytics fetching algorithm. Instead of starting again
where we stopped during the last fetching minus 30 minutes, we now just
continue where we stopped. But with ms precision (because no longer
database dependent after first fetch), and we stop at NOW - 1 minute to
reduce chance of missing events.
- Apart from that, a missing fetching loop is introduced. This fetches
events that are older than 30 minutes, and just processes all events a
second time to make sure we didn't skip any because of storage delays in
the Mailgun API.
- A new scheduled fetching loop, that allows us to schedule between a
given start/end date (currently only persisted in memory, so stops after
a reboot)
UI and endpoint changes:
- New UI to show the state of the analytics 'loops'
- New endpoint to request the analytics loop status
- New endpoint to schedule analytics
- New endpoint to cancel scheduled analytics
- Some number formatting improvements, and introduction of 'opened'
count in debug screen
- Live reload of data in the debug screen
Other changes:
- This also improves the support for maxEvents. We can now stop a
fetching loop after x events without worrying about lost events. This is
used to reduce the fetched events in the missing and scheduled event
loop (e.g. when the main one is fetching lots of events, we skip the
other loops).
- Prevents fetching the same events over and over again if no new events
come in (because we always started at the same begin timestamp). The
code increases the begin timestamp with 1 second if it is safe to do so,
to prevent the API from returning the same events over and over again.
- Some optimisations in handing the processing results (less merges to
reduce CPU usage in cases we have lots of events).
Testing:
- You can test with lots of events using the new mailgun mocking server
(Toolbox repo `scripts/mailgun-mock-server`). This can also simulate
events that are only returned after x minutes because of storage delays.
no refs.
- Spark email client injected styles for the new email signup email which broke the layout. This commit forces those styles for the email to be consistent in all clients.
closes https://github.com/TryGhost/Team/issues/2572
The mentions browse output previously only showed resource info if the
mentioned resource type was a `post`.
Additionally, the `resource_type` column basically defaulted to `post`
regardless whether it was a page or in fact a post.
With this change we now have `resource_type` wired in to correctly
determine if the mentioned url was a page or a post.
fixes https://github.com/TryGhost/Team/issues/2562
New event fetching loops:
- Reworked the analytics fetching algorithm. Instead of starting again
where we stopped during the last fetching minus 30 minutes, we now just
continue where we stopped. But with ms precision (because no longer
database dependent after first fetch), and we stop at NOW - 1 minute to
reduce chance of missing events.
- Apart from that, a missing fetching loop is introduced. This fetches
events that are older than 30 minutes, and just processes all events a
second time to make sure we didn't skip any because of storage delays in
the Mailgun API.
- A new scheduled fetching loop, that allows us to schedule between a
given start/end date (currently only persisted in memory, so stops after
a reboot)
UI and endpoint changes:
- New UI to show the state of the analytics 'loops'
- New endpoint to request the analytics loop status
- New endpoint to schedule analytics
- New endpoint to cancel scheduled analytics
- Some number formatting improvements, and introduction of 'opened'
count in debug screen
- Live reload of data in the debug screen
Other changes:
- This also improves the support for maxEvents. We can now stop a
fetching loop after x events without worrying about lost events. This is
used to reduce the fetched events in the missing and scheduled event
loop (e.g. when the main one is fetching lots of events, we skip the
other loops).
- Prevents fetching the same events over and over again if no new events
come in (because we always started at the same begin timestamp). The
code increases the begin timestamp with 1 second if it is safe to do so,
to prevent the API from returning the same events over and over again.
- Some optimisations in handing the processing results (less merges to
reduce CPU usage in cases we have lots of events).
Testing:
- You can test with lots of events using the new mailgun mocking server
(Toolbox repo `scripts/mailgun-mock-server`). This can also simulate
events that are only returned after x minutes because of storage delays.
Refs TryGhost/Team#2459
-upgraded got from v9.6.0 to v11.8.6 to support following redirects (and
other fixes)
-got v12+ requires ESM, so we do not want to upgrade further at this
time
-required changes to a few libraries that use externalRequests
-mention discovery service tests updated to test for follow redirects
no issue
- Instead of running milestone service directly on boot, set a random
timeout of 0-4 days to run after boot
- Updated tests
- Service is still behind a beta flag
refs 0306c397d0
refs https://github.com/TryGhost/Toolbox/issues/522
- 'extraProperties' should have been cleaned up along with the referenced commit. This property does not perform any logic in current codebase (see ref) and makes it problematic to make "post" resource serialization more generic (for caching purposes).
refs https://github.com/TryGhost/Toolbox/issues/515
- We don't have a good way to test TTL caches without setting up Redis in the environment
- Adding in-memory cache adapter with TTL allows to run tests on CI without having to install Redis
- Also, TTL in memory cache can be a great substitution for Redis-based caches on instances that
have a lot of spare RAM and don't need to use Redis necessarily
- MemoryTTL cache accepts two parameters "TTL" and "max"
- TTL - is time in milliseconds to hold the value for in cache
- max - is the maximum amount of items to keep in the cache
- To use MemoryTTL cache specify following config in the cache section:
```
"adapters": {
"cache": {
"imageSizes": {
"adapter": "MemoryTTL",
"ttl": 3600
}
}
}
```
- Above config would apply MemoryTTL cache to imageSizes feature with TTL fo 3600 ms
Removed the wrapper class for the email service from coverage, because this only wires up a lot of dependencies, which is hard to test in a unit test because we also have to init all the dependencies in a unit test. It is already covered by E2E tests.
refs https://github.com/TryGhost/Team/issues/2550
By using cheerio to parse the HTML we can correctly look for elements
which use the target URL as the href attribute, rather than doing a
plaintext search. This closer to what the spec says.
refs https://github.com/TryGhost/Team/issues/2550
Whilst this isn't ideal making multiple requests for the same site
it's a first step towards verification properties, and can be
refactored to improve performance later. With the current volume of
incoming Webmentions we've seen so far this shouldn't be a problem.
refs https://github.com/TryGhost/Team/issues/2548
We need to be able to do this outside of the verify method so that
repository implementations have the ability to create existing
instances without running expensive verification checks!
[Added initial mentions-jobs
service](3656190114)
This is the result of running `cp -r jobs mentions-jobs` in the services
directory.
[Waited for mentions-jobs queue before
shutdown](2bb1a12a89)
This matches the functionality of the existing jobs service where we
will wait
for jobs to complete before closing the process.
[Used mentions-jobs service in the mentions
service](4e4f9fdd00)
This ensures that any delays in the mentions jobs queue does not effect
other
parts of the application.
refs
https://www.notion.so/ghost/Marketing-Milestone-email-campaigns-1d2c9dee3cfa4029863edb16092ad5c4?pvs=4
- Added a `slack-notifications` repository which handles sending Slack
messages to a URL as defined in our Ghost(Pro) config (also includes a
global switch to disable the feature if needed) and listens to
`MilestoneCreatedEvents`.
- Added a `slack-notification` service which listens to the events on
boot.
- In order to have access to further information such as the reason why
a Milestone email hasn't been sent, or the current ARR or Member value
as comparison to the achieved milestone, I added a `meta` object to the
`MilestoneCreatedEvent` which then gets accessible by the event
subscriber. This avoid doing further requests to the DB as we need to
have this information in relation to the event occurred.
---------
Co-authored-by: Fabien "egg" O'Carroll <fabien@allou.is>
refs https://github.com/TryGhost/Team/issues/2561
- added simple socket-io implementation to Ghost server
- added alpha flag for websockets
- added route in admin to test websockets using a simple counter stored in server local memory (refreshes on reboot)
refs https://github.com/TryGhost/Toolbox/issues/522
- Browse endpoints for Posts and Pages are creating the most database traffic in the system. These are read-only endpoints that don't have to be fresh 100% of the time. Having optional cache allows to offload some of the database querying to more efficient storage.
- To enable cache for Posts/Pages browse endpoints there are two prerequisites:
- set 'hostSettings:postsPublicCache:enabled' to 'true' in the configuration file
- add 'postsPublic' cache adapter in cache configuration
- Example config for adapters with 60s TTL for a cached resource:
```
"adapters": {
"cache": {
"postsPublic": {
"adapter": "Redis",
"ttl": 60,
"keyPrefix": "site_id_here:posts-content-api:"
}
}
},
```
refs https://github.com/TryGhost/Toolbox/issues/522
- The caching rules for Content API resources like Tags / Posts / Pages / Authors are the same, so it makes sense to make a common repository which would take in a model as a parameter and perform the queries on that level instead of implementing repository-per-model.
- With this refactor getting Posts Content API caching would be much simpler change.
no issue
This should massively increase the speed of importing for the large dataset, which is important as the time to import it on Pro is >10 minutes at the moment
fixes https://github.com/TryGhost/Ghost/issues/16278
`urlToPath` is not a method that is defined on the StorageBase, and thus is not implemented on any custom storage adapter.
We need this method to prevent uploading a file in two different directories. Currently this is an edge case:
- If you upload a file at midnight between a month shift, it is possible that we upload one file at `/2023/02/file.png` and the original at `/2023/03/file_o.png`
- To make sure we upload the second at the same directory, we need the `urlToPath` implementation. The save method sadly only returns the URL, which could be different than the stored file path, so we cannot use that return value.
This fix only uses `urlToPath` if available, to prevent above issue.
fixes https://github.com/TryGhost/Team/issues/2433
- Moved all outbound link tagging code to separate OutboundLinkTagger
- Because a site can easily enable/disable this feature, we don't store
the ?refs in the HTML but add them on the fly for now in the Content
API.
closes https://github.com/TryGhost/Team/issues/2551
Rather than blindly passing all data through the API we explicitly include each
new property. This allows us to make changes to the core entities without
affecting the API. The verified property is being added now to give design the
ability to display these mentions differently.
We also needed to include the verified property in the return value of toJSON,
this was missed as part of the original entity changes
refs https://github.com/TryGhost/Toolbox/issues/520
- The cluster config is taking over local adapter ttl configuration - the priority should be reverse adapter config first followed by cluster config
- In addition if we add nested config merging to adapter manager we could achieve the same effect by having per-adapter configuraiton with "clusterConfig.options.ttl" value
no issue
- With the switch of using a `MilestoneCreatedEvent` we'll be decoupling the mailing functionality and not need `GhostMailer` as dependency in the package anymore
no issue
- In preparation of using event emitting for Milestone achievements, we needed to add a dedicated `MilestoneCreatedEvent` to the `Milestone` entity.
- The event will be emitted using `DomainEvents` when a new milesteone is saved, which will allow us to listen to these events.
no issue
- The way we're going to implement milestones diverged from the original idea of handling email sending within the milestone-emails package, as we'll be sending events instead and will utilise the StaffService to listen to them and send the emails
- This renames the package as well as the service in core itself and all relevant tests
no issue
- For better testability with in-memory repository, refactor the
milestones service to preserve the API instance
- Fetching the information about Stripe live mode from Stripe service
was causing difficulties when testing. As a workaround we switched to
reading the live mode keys and determine it that way.
---------
Co-authored-by: Fabien "egg" O'Carroll <fabien@allou.is>
refs https://github.com/TryGhost/Toolbox/issues/520
- The cluster config is taking over local adapter ttl configuration - the priority should be reverse adapter config first followed by cluster config
- In addition if we add nested config merging to adapter manager we could achieve the same effect by having per-adapter configuraiton with "clusterConfig.options.ttl" value
refs c489343831
- `lexicalLib.lexicalHtmlRenderer` can't be used directly because it's missing required config for nodes to render properly
- updated email serializers to use `lexicalLib.render()` instead
closes https://github.com/TryGhost/Team/issues/2558
- bumped `kg-lexical` packages so we're working with latest suite of default nodes and renderer
- added a `render()` method directly to our `lexicalLib` object
- allows us to pass through all of Ghost's config for image transforms etc in one place rather than every time we want to render something
no issue
- local development Ghost installs should not default to a locally served `koenig-lexical.umd.js` file but should use the default CDN version
- a locally served file should be set in `config.local.json` when you want to develop against an unreleased version
no issue
- added `requestMethod` to `fileTypes` map
- added pass-through of `formData` options to `upload(file, options)` so `url` property can be passed to map uploaded image file to media file
- fixed `resourceName` for media thumbnail uploads
fixes https://github.com/TryGhost/Team/issues/2557
When a member doen't have a name, and the first_name replacement doesn't have a fallback, we did show %recipient.first_name% instead of an empty string.
no issue
- When we receive an email failure with an empty message, the saving of
the model would fail because of schema validation that requires strings
to be non-empty.
- This adds more logging to the email analytics service to help debug
future issues
- Performance improvement to storing delivered, opened and failed emails
by replacing COALESCE with WHERE X IS NULL (tested and should give a
decent performance boost locally).
refs https://github.com/TryGhost/Koenig/pull/491
- `<KoenigComposer>` now takes a `fileUpoader` object in place of `imageUploadFunction`
- updated the upload functions in `koenig-lexical-editor.js` to match expected patterns, handle multiple files and file types, and return expected upload progress, result, and error details
closes https://github.com/TryGhost/Team/issues/2548
Rather than use a setter here we've used a verify method which takes the HTML
string and naively validates that the target URL is present. This is so that the
logic of verification is encapsulated in the Mention, and should mean that
erroneous verification doesn't happen.
We could consider down the line that the verify method fetches the content
itself, but if we're to do that we should pass in `got` as a param, so that it's
possible to stub in tests.
One thing to think about when it comes time to making this as performant as
possible is doing a single fetch of the source document and using that for
verification and metadata extraction. At that point we should probably
consolidate both of those operations, either moving the metadata extraction into
the Mention entity (passing in any necessary deps) OR we move the verification
out to the same layer as metadata extraction.
closes https://github.com/TryGhost/Team/issues/2552
We send a Webmention for the same URL twice, but change the contents
of the source document, and we check that the source metadata is
updated appropriately.
We should consider extending all of these tests to include featured
images and logos etc...
fixes https://github.com/TryGhost/Team/issues/2542
fixes https://github.com/TryGhost/Team/issues/2543
fixes https://github.com/TryGhost/Team/issues/2544
- Hides incomplete subscriptions
- Shows Past Due subscriptions
- Fixed UI issues with 3+ subscriptions
- Fixed missing complimentary subscription when one subscription was
incomplete/inactive
- Fixed sending a paid subscription started email for incomplete
subscriptions. This change also required us to actually send the email
when the incomplete subscription eventually becomes active. So the
introduction of a new `SubscriptionActivatedEvent` made sense/was
required (because sending a SubscriptionCreatedEvent again would cause
other issues).
refs https://github.com/TryGhost/Toolbox/issues/515
- This implementation allows to use Redis cluster as a caching adapter. The cache adapter can be configured through same adapter configuration interface as others. It accepts following config values:
- "ttl" - time in SECONDS for stored entity to live in Redis cache
- "keyPrefix" - special cache key prefix to use with stored entities
- "host" / "port" / "password" / "clusterConfig" - Redis instance specific configs
- Set test coverage to non-standard 75% because the code mostly consists of the glue code that would require unnecessary Redis server mocking and would be a bad ROI. This module has been used in production for a long time already, so can be considered pretty stable.
refs https://github.com/TryGhost/Toolbox/issues/515
- Redis-based caches can be used on hosted-environments to store information with high memory impact - when in-memory caches would be too impractical to use
- This is a placeholder package for a cluster-aware Redis cache implementation compatible with Ghost's cache adapter interface (a41d351f16/packages/adapter-base-cache)
refs https://github.com/TryGhost/Toolbox/issues/515
- The caches relying on external storage like e.g.: Redis, the get/set operations are usually async. The tags repository should be working with these as caching is expected to be non-in-memory for these data.
refs https://github.com/TryGhost/Toolbox/issues/515
- By moving the cache initialization behind the hostSettings configuration we can limit the experimental feature to only hosted environments with special capacities
- An example configuration to enable Tags caching looks like this:
```
"hostSettings": {
"tagsPublicCache": {
"enabled": true
},
```
In addition to have the caching backed by a Redis backend or even InMemoryTTL cache the site configuration should include a cache adapter configuration like this:
```
"adapters": {
"cache": {
"active": "Memory",
"tagsPublic": {
"adapter": "TTL",
"ttl": 60000 // 60 * 1000 minute
},
```
refs https://github.com/TryGhost/Toolbox/issues/515
- There are a lot of repeated cacheable tag-related queries coming from
{{get}} helpers in themes that can be cached.
- Having a repository layer deal with very specific type of query allows
to add extra functionality, like caching, on top of the database query
- This commit is wiring code that addds a default in-memory cache to
all db queries. Note, it lasts forever and has no "reset" listeners. The
production cache is mean to have a short time-to-live (TTL) - removes a need
to keep the cache always fresh.
- Kept the cache key shortened. Without a "context" and any other non-model options the cache-key can store more variations of queries. For example, there is no member-specific or integration-specific query results, so having those in the cache key would only partition the cache and use up more memory.
closes https://github.com/TryGhost/Team/issues/2547
Changed the configuration for testing to be a bit more strict, by slowing down the amount of requests it can handle to give CI enough time to kick in the rate limiter. Before this, CI simply wasn't hitting the API fast enough to trigger the rate limiter.
Co-authored-by: Ronald Langeveld <hi@ronaldlangeveld.com>
refs https://github.com/TryGhost/Team/issues/2535
At the moment we only update the metadata of the Webmention source, so
that we can capture update titles, excerpts, etc... when a post is updated.
refs https://github.com/TryGhost/Team/issues/2535
Moving this into a separate method allows us to set the metadata externally from
the Mention entity and keep all of the validation, without having duplicate code
refs https://ghost.slack.com/archives/C04MSE4MKJT/p1675948815531779
- running at a fixed hh:mm every day means a platform with a large number
of Ghost sites will get hammered with DB requests when they all start
up
- this reconfigures the cron to run at a random minute and second
between 0am and 5am, which gives a 6 hour window
refs https://github.com/TryGhost/Team/issues/2534
As we're using soft deletes for mentions we need to store the `deleted` column
as well as enforce a `'deleted:false'` filter on the bookshelf model.
We've also implemented the handling for deleting mentions. Where we remove a
mention anytime we receive and update from or to a page which no longer exists.
Co-authored-by: Steve Larson <9larsons@gmail.com>
refs https://github.com/TryGhost/Team/issues/2534
This is so that we can support soft deletes for Mentions.
We need to add the defaults to the model so that write continue to work.
Co-authored-by: Fabien "egg" O'Carroll <fabien@allou.is>
closes https://github.com/TryGhost/Team/issues/2526
- Mention emails can now be toggled inside staff user' profiles, if they
have the webmention flag enabled on their Ghost site.
- Removed the flag dedicated to webmention email notifications and is
now handled by the `webmention` flag.
- Does not send email notification if `webmention` flag is not enabled.
- Updated tests.
---------
Co-authored-by: Fabien "egg" O'Carroll <fabien@allou.is>
refs https://github.com/TryGhost/Team/issues/2482
This moves the processing of Mailgun events to the main thread. By using a simple approach where we emit a start event on the worker thread (via the job manager) and listen for it on the main thread. This is needed because for now the job manager doesn't support scheduling periodic jobs on the main thread (not offloaded).
Apart from that, the email processor now uses the email event storage directy instead of emitting events (it is still emitting event for now). This makes sure we await for the event to be processed before continuing with the next event.
refs https://github.com/TryGhost/Ghost/issues/16125
The previous fix for incorrect recipient details being shown when
re-sending a failed email introduced another bug that prevented the
"Match post visibility" default recipients setting from working.
- the server always sets `post.emailSegment` to `'all'` for new posts so
the publish flow recipient filter logic that checked for
`post.emailSegment` being present always defaulted to `'all'` rather
than falling back to the selected default recipients setting
- when a post has been published but the email failed it will have its
`newsletter` value set so we can use that as a check for using the
`post.emailSegment` value in place of the default recipients setting
refs https://github.com/TryGhost/Team/issues/2526
- created a migration for a new boolean column in users that would
determine if the staff user gets an email when the publication receive a
new mention.
closes https://github.com/TryGhost/Team/issues/2419
- adds a rate limiter implementation to the mentions receiving
endpoint.
- Current configuration is `{"minWait": 10,
"maxWait": 100,
"lifetime": 1000,
"freeRetries": 100}` which is still very open and almost unrestricted.
- currently makes use of database storage to track the limits, but can be relatively easily swapped out to something eg Redis should we find this endpoint getting hit too often and maliciously.
no issue
Free and premium newsletters were the other way around in the demo-data. This was a good opportunity to stop the email table importer from relying on the newsletter name, and use the order alone.
refs https://github.com/TryGhost/Ghost/issues/16125
The previous fix for incorrect recipient details being shown when
re-sending a failed email introduced another bug that prevented the
"Match post visibility" default recipients setting from working.
- the server always sets `post.emailSegment` to `'all'` for new posts so
the publish flow recipient filter logic that checked for
`post.emailSegment` being present always defaulted to `'all'` rather
than falling back to the selected default recipients setting
- when a post has been published but the email failed it will have its
`newsletter` value set so we can use that as a check for using the
`post.emailSegment` value in place of the default recipients setting
refs
https://www.notion.so/ghost/Marketing-Milestone-email-campaigns-1d2c9dee3cfa4029863edb16092ad5c4
This adds a milestone entity and in-memory repository in a new
`milestone-emails` package. This also adds a first initial definition of
milestones and their types which is held in the default config to avoid
DB changes when, e. g. values change.
This should get everything in place to begin with the service
implementation.
fixes https://github.com/TryGhost/Team/issues/2522
When sending an email for multiple batches at the same time, we now
reuse the same email body for each batch in the same segment. This
reduces the amount of database queries and makes the sending more
reliable in case of database failures.
The cache is short lived. After sending the email it is automatically
garbage collected.
fixes https://github.com/TryGhost/Team/issues/2522
When sending an email for multiple batches at the same time, we now
reuse the same email body for each batch in the same segment. This
reduces the amount of database queries and makes the sending more
reliable in case of database failures.
The cache is short lived. After sending the email it is automatically
garbage collected.
refs TryGhost/Team#2508
-added sending service e2e tests
-should job off this sending service for better tests
-and for ghost to finish processing the job before shutdown
refs https://github.com/TryGhost/Toolbox/issues/515
- The link redirect handled was querying database on every single frontend request causing a significant amount of unnecessary traffic
- The optimization is returning early if the incoming URL does not start with a common "r/" prefix
fixes https://github.com/TryGhost/Team/issues/2512
The email sending is a crucial flow that should not be interrupted. If
there are database connection issues, we should try to recover from them
automatically by retrying individual database queries and transactions
for a limited amount of time.
This adds a helper method to retry database operations. It limits all
database queries before sending the actual email to maximum 10 minutes.
Database operations that happen after sending have a higher retry time
because they are more crucial to prevent data loss (e.g. saving that an
email was sent).
refs 7f1e970a0b
- `koenig-card-callout.hbs` was touched without fixing the lint errors or updating to todo list causing the lint todos to become out of sync
- fixed the lint error and updated the todo list
refs https://github.com/TryGhost/Team/issues/2503
This is in the MentionController atm as it's considered a presentation
concern. We might want to consider moving this into the MentionsAPI in
future so that we can simplify the controller and even remove it
completely in favour of putting the data-mapping in the endpoint definition.
refs https://github.com/TryGhost/Toolbox/issues/515
- The link redirect handled was querying database on every single frontend request causing a significant amount of unnecessary traffic
- The optimization is returning early if the incoming URL does not start with a common "r/" prefix
refs TryGhost/Team#2477
-removed post.edited as it was too inclusive
-changed to post.published, post.published.edited, post.unpublished
-blocked import and internal data from triggering mentions
refs https://github.com/TryGhost/Team/issues/2506
`email` objects no longer contain a `html` field and the fallback logic in the email preview modal was failing resulting in a 404 from trying to fetch an email preview using the email id rather than a post id.
- added quick-fix to the preview modal logic to use `data.post_id || data.id` for generating the preview URL (previous logic never expected to reach the fallback when working with an email record)
This is a pretty simple way for us to track which webmentions are sent
by Ghost. Although it's easily spoofed, so are other approaches like
using a header (e.g. User-Agent). If we find that this data is being
spoofed we can look at different approach.
Becuase our receiving implementation stores the payload of the
Webmention, we'll be able to know inside Ghost which Mentions
originated from another Ghost installation, which is useful for stats
and gives us the possibility to display that information in the feed.
Longer term we might want to consider storing this data in a separate
column for Mentions, rather than the `payload` column - but that is
outside the scope of this change.
refs https://github.com/TryGhost/Toolbox/issues/497
refs fb7532bf5d
- We downgraded the 'GS090-NO-PRICE-DATA-CURRENCY-CONTEXT' rule in gscan to non-fatal, meaning Ghost should not be throwing an error but instead render an empty value for {{price}} helper when price data is empty.
- For example, a legacy syntax like this: '{{price currency=@price.currency}}' should not cause a page render error but return an empty price string.
- The pattern of returning an empty string instead of crashing is used in other helpers like {{img_url}} and and {{url}}
refs https://github.com/TryGhost/Toolbox/issues/497
refs fb7532bf5d
- We downgraded the 'GS090-NO-PRICE-DATA-CURRENCY-CONTEXT' rule in gscan to non-fatal, meaning Ghost should not be throwing an error but instead render an empty value for {{price}} helper when price data is empty.
- For example, a legacy syntax like this: '{{price currency=@price.currency}}' should not cause a page render error but return an empty price string.
- The pattern of returning an empty string instead of crashing is used in other helpers like {{img_url}} and and {{url}}
