refs https://github.com/TryGhost/Ghost/issues/12420
- updated `order` bookshelf plugin's `parseOrderOption()` method to return multiple order-related properties
- `order` same as before, a key-value object of property-direction
- `orderRaw` new property that is a raw SQL order string generated from `orderRawQuery()` method in models
- `eagerLoad` new property that is an array of properties the `eagerLoad` plugin should use to join across
- updated `pagination.fetchAll()` to apply normal order + raw order if both are available and to handle eager loading / joins when `options.eagerLoad` is populated
- updated post model to include details for email relationship and to add `orderRawQuery()` that allows `email.open_rate` to be used as an order option
no issue
- job registration was checking for submitted emails in it's email count but the job registration method is called as soon as an email is created meaning the email has a status of 'pending' which prevented the analytics job from being started until a second email was sent
no issue
- email analytics may be desirable to fully switch off in certain circumstances, when that happens we want to prevent related background jobs from running and expose the feature flag via the config endpoint in the Admin API so that clients can adjust accordingly
no issue
- if emails are older than 30 days we wouldn't be able to fetch any analytics for them and if a site used emails in the past but is no longer using them it doesn't make sense to keep potentially expensive background worker threads spinning up
no issue
- recurring jobs spin up worker threads which can be quite CPU intensive even when not performing much processing, this can be problematic in environments where there are many Ghost instances running
- updated the email job scheduling to be skipped on bootup when there are no emails in the database and to be started when the first email is created as long as we're not in testing env
- increase analytics job schedule from every 2 minutes to every 5 minutes to help spread the load further across instances
no issue
- typically cron/later schedules will schedule for :00 on the minute which would create API spikes with every members-email-using Ghost site hitting the API at the same time
- adjusted the scheduling to use cron syntax with job runs every 2 minutes on 1,3,5... or 2,4,6... and a random seconds value to smooth usage across sites
no refs
Ghost's Portal script is loaded via unpkg which was till now pinned to load `@latest` version, which unpkg auto-resolved to the latest released Portal version. This allowed fast iterations on Portal while still in active beta development to test latest Portal releases.
Going forward, Portal will be pinned to latest specific minor version that allows releasing new features that are not backward compatible without affecting older Ghost releases.
Note: All previous Ghost releases with Portal `@latest` will continue to resolve to latest version and will need to update to latest Ghost 3.x to use all Portal features.
no issue
- it's possible background jobs may cause unintended side-effects so it's useful to have a kill-switch to disable them individually to keep sites working
- users imported from CSV with no created_at date where having their created_at date being stored as an int rather than a datetime.
- this was causing parsing issues with the graph so this commit fixes the formatting
closes#12083
- fixes a parsing issue where negative offset values were incorrectly having the + sign added regardless of actual offset for sqlite databases.
- for mysql databases absolute values of offset were taken with sign applied where appropriate to stop issues where both hours and minutes could be negative which would cause both an issue with offsets that could present as -2:30 and by the look of the code also trigger extra padding to result in -2:-030 rather than the expected -2:30
no issue
- the 4.2.0 version of `sqlite3` that we're using is not compatible with `worker_threads`
- 5.0.0 should add support this but there are other errors
- 5.0.1 is released but not published (https://github.com/mapbox/node-sqlite3/issues/1386)
- we export i18n from `core/frontend/services/proxy` and this is used in
the most of the places in the frontend code
- this commit aligns the rest of the code in core/frontend to use the
proxy too
- unfortunately core/frontend/services/themes/i18n.js loops back to the
proxy so we have a circular dependency
no issue
- added `EmailAnalyticsService`
- `.fetchAll()` grabs and processes all available events
- `.fetchLatest()` grabs and processes all events since the last seen event timestamp
- `EventProcessor` passed event objects and updates `email_recipients` or `members` records depending on the event being analytics or list hygiene
- always returns a `EventProcessingResult` instance so that progress can be tracked and merged across individual events, batches (pages of events), and total runs
- adds email_id and member_id to the returned result where appropriate so that the stats aggregator can limit processing to data that has changed
- sets `email_recipients.{delivered_at, opened_at, failed_at}` for analytics events
- sets `members.subscribed = false` for permanent failure/unsubscribed/complained list hygiene events
- `StatsAggregator` takes an `EventProcessingResult`-like object containing arrays of email ids and member ids on which to aggregate statistics.
- jobs for `fetch-latest` and `fetch-all` ready for use with the JobsService
- added `initialiseRecurringJobs()` function to Ghost bootup procedure that schedules the email analytics "fetch latest" job to run every minute
no issue
- cleans up unused tables `emails.{meta,stats}`
- adds timestamp columns `email_recipients.{delivered_at,opened_at,failed_at}` that can be used for event timelines and basic stats aggregation
- indexed because we want to sort by these columns to find the "latest event" when limiting Mailgun events API requests
- adds aggregated stats columns `emails.{delivered_count,opened_count,failed_count}`
- adds a composite index on `email_recipients.[email_id,member_email]` to dramatically speed up `email_recipient` update queries when processing events
- modifies the db initialisation to support an `'@@INDEXES@@'` key in table schema definition for composite indexes
closes https://github.com/TryGhost/Ghost/issues/12416
This fixes compatibility for the `send_email_when_published` option for
the Posts API.
The model layer only allows setting the `email_recipient_filter` column
when the `status` is being changed. Because of this we need to withhold
the `status` change until after we have determined the
`email_recipient_filter`.
- this helps simplify the code and gets rid of Promise chaining
- apparently I can't easily use an async function within filter, so I've
left it for now
refs https://github.com/TryGhost/Ghost/issues/12256
We no longer want to filter out cancelled subscriptions, so we are able
to remove the whereIn clause of the relation.
* Fixed paid flag on member
* Fixed content gating for members
Now that the subscriptions for a member include all of them, we must
explicitly check that the member has an active subscription in order to
consider them "paid"
- this helps bring all the code together so we can extract it in the
future
- turning it into a class also lets us easily inject the i18n instance
and store it locally
no issue
- the `'bulk-email`' tag was only being added to bulk emails if another more specific tag was set up via config
- we always want the `'bulk-email'` tag to be present for better event filtering
refs #12402
- With bumped version of job-manager it offloads job procesing into separate worker thread. Having jobs run out of main Ghost process even loop allows for safe job execution, which does not block Ghost from serving requests or performing other functions without a delay
- Added experimental data access to 'testmode' jobs. This should serve as an illustration of how to access data from the job layer
refs https://github.com/TryGhost/Ghost/issues/10354
- The serializer wasn't hooked up properly during the implementation (1a4497fc9a). It is not possible to hook it back in now as that would introduce breaking change to now stable v2 API.
- v3 (canary) has is properly hooked in, so there should not be a problem going forward
refs #12366
This implements redirection based on the settings for successful member sign up!
- Removes support for redirecting to `req.path` afterwards, this was never used and
we now have a more configurable implementation.
- Retains redirection to the homepage for unsuccessful sign up (invalid/expired token)
no-issue
This logic would assume that the option was always passed at the point
of publishing the post, which is not the case for scheduled posts.
Instead of setting the property to 'none' when the option is not
present, we take the approach of ONLY setting the propery when
1. It is present and not 'none'
2. The post is being published or scheduled
This means that scheduled posts will have the property set correctly,
and any future publishing will leave the it in the original state
no refs
- Migrations were using previous default for badge which was set to `false`
- Default config for badge was `false` which caused migration to switch off the badge, updated to `true`
no issue
- The logic inside the if/esle conditions was duplicated and made it confusing to reason about.
- This change is not effecting the underlying logic at all
- these helpers remove a lot of the duplicated code that we had when doing up/down column migrations, and provides a much shorter way of doing this in the future
no issue
Updated newsletter badge to be shown by default which was previously set to be hidden. Users can still control the visibility via Email settings in Ghost-Admin
no-issue
We want to give users to ability to customise the content of their newsletter, and the first step
toward that is a setting in which we can store text or html to embed in the template
no-issue
* Handled send_email_when_published in Posts API
This restores backwards compatibility of the Posts API allowing existing
clients to continue to use the `send_email_when_published` flag. This
change uses two edits, which is unfortunate. The reason being is that
this is an API compatibility issue, not a model issue, so we shouldn't
introduce code to the model layer to handle it. The visibility property
of the model is used to determine how to fall back, and because it can
be left out of the API request, and relies on a default in the settings,
we require that the model decide on the `visibility` before we run our
fallback logic (or we duplicate the `visibility` default at the cost of
maintenance in the future)
* Dropped send_email_when_published column from posts
Since this column is not used any more, we can drop it from the table.
We include an extra migration to repopulate the column in the event of
a rollback
* Updated importer to handle send_email_when_published
Because we currently export this value from Ghost, we should correctly
import it. This follows the same logic as the migrations for this value.
* Included send_email_when_published in API response
As our v3 API documentation includes `send_email_when_published` we must
retain backward compatibility by calculating the property.
* Fixed fields filter with send_email_when_published
* Added safety checks to frame properties
Some parts of the code pass a manually created "frame" which is missing
lots of properties, so we check for the existence of all of them before
using them.
* Fixed 3.1 migration to include columnDefinition
We require that migrations have all the information they need contained
within them as they run in an unknown state of the codebase, which could
be from the commit they are introduced, to any future commit. In this
case the column definition is removed from the schema in 3.38 and the
migration would fail when run in this version or later.
refs https://github.com/TryGhost/Ghost/issues/12355
- Adds new default settings for newsletter customisations - `newsletter_show_badge`, `newsletter_show_header` and `newsletter_body_font_category`
- Adds migrations to update group for new settings
- Add migration to update settings based on existing config value for newsletter settings
- Passes new newsletter settings to newsletter template and updates design based on them
- Fix tests
no issue
- These changes should allow easy testing of scheduled jobs and provide an implementation example for future jobs within Ghost
- Added experimental support to run jobs based on functions defined in modules
- Extendeded testmode "schedule" endpoint with an optional job name parameter which picks up job from "jobs" folder
- Adjusted scheduleJob method use and passed "schedule" as first parameter
- Bumped job-manager version to allow for all new functionality
no issue
- Living in separate folder will allow adding more testmode specific modules (for example test jobs)
- Mimicks the folder structure of how v2/canary routes are structured better
no-issue
* Used email_recipient_filter in MEGA
This officially decouples the newsletter recipients from the post
visibility allowing us to send emails to free members only
* Supported enum for send_email_when_published in model
This allows us to migrate from the previously used boolean to an enum
when we eventually rename the email_recipient_filter column to
send_email_when_published
* Updated the posts API to handle email_recipient_filter
We now no longer rely on the send_email_when_published property to send
newsletters, meaning we can remove the column and start cleaning up the
new columns name
* Handled draft status changes when emails not sent
We want to reset any concept of sending an email when a post is
transition to the draft status, if and only if, and email has not
already been sent. If an email has been sent, we should leave the email
related fields as they were.
* Removed send_email_when_published from add method
This is not supported at the model layer
* Removed email_recipient_filter from v2&Content API
This should not be exposed on previous api versions, or publicly
* Removed reference to send_email_when_published
This allows us to move completely to the email_recipient_filter
property, keeping the code clean and allowing us to delete the
send_email_when_published column in the database. We plan to then
migrate _back_ to the send_email_when_published name at both the
database and api level.
no issue
- set `emails.track_opens` to `true` when the `enableDeveloperExperiments` flag is set
- update mailgun bulk-email provider to pass the open-tracking header to Mailgun when the email's `track_opens` flag is set
- this check should never really be hit unless the DB has missing data,
but the code tries to read from an undefined variable and crashes, which we
can protect against
- this can be triggered by doing a `knex-migrator init` followed by a
`knex-migrator rollback --v 3.0 --force`
- 504509bb6 removed the global override for Promise
- there are a bunch of places in code that use Bluebird Promise methods,
but Bluebird wasn't being imported in these places
- this would have thrown errors all over the place
no issue
- tracking of bulk email opens can be enabled/disabled over time, if we're calculating analytics for emails we don't want emails which didn't have tracking enabled skewing the results so we need a record of whether tracking was enabled for each email
no issue
- We need an ability to early test scheduled jobs to refine the new feature and it's API. Should be used with caution
- To schedule an example scheduled job every 30 seconds run following request: `curl http://localhost:2368/ghost/api/schedule/every%2030%20seconds`
refs #11085
- Incorrect usage error was logged to the output when there was no recirecst configuration file present in the system. Previously an empty string was returned in such situation, resulting in "ENOENT" error, which was ignored through special handling.
- The fix resembles logic in redirects async getter function where empty array is returned when the config file does not exits.
- Attempting to read unexistent config should not ever happen and will be handled on the config service layer, this is why special "ENOENT" handling has been removed
no-issue
This column will allow us to store the canonical recipient filter on the
email resource giving us a detailed log of which members an email was
intended for
no-issue
This column will allow us to decouple the recipients of newsletter from
the `visibility` of a post, allowing us to send emails to specifically
free members, or to send paid posts as newsletters to all members.
closes#11085
- Ghost has been using YAML format for other configurations (e.g. routes). The plan is to move to this format for all user-edited settings files. By default JSON format is still used in Ghost Admin API v2/v3, but will be changed to YAML in API v4. Check referenced issue for more context.
- New format supports all the features available before. The main noticeable change is the structure of config file. It is now grouped by redirect HTTP code instead of specifying `"permanent": true | false` attribute for each config property. Example format for YAML config:
```
302:
/from-url/: /to-url/
301:
/category/([a-z0-9\-]+)/i: /tag/$1/
/v([0-9\.]+)/docs/([a-z0-9\-]+)/i: /docs/$2/
```
- Added 2 new endpoints: `POST redirects/upload` and `GET redirects/download`. These serve as an alias to current GET/POST `/redirects/json. "upload/download" naming pattern is introduced to match the convention with other resources that can be uploaded and downloaded (images, themes etc.). `/redirects/json` endpoints will be removed in Admin API v4
- The parsing code from `custom-redirects.js` has been moved to `frontend/services/redirects/settings.js`. This location is more appropriate for this logic and eventually `custom-redirects.js` middlewear might be moved into "frontend" as this middlewear plays a role mostly effecting that area.
no refs
[Portal](https://github.com/TryGhost/Portal) is a new drop-in script to make the bulk of Ghost membership features work on any theme out of the box, which was under a developer flag so far. This release removes the flag for Portal and makes it included as default for any members-enabled Ghost site. The Portal script is backward compatible with old public members script and existing Members-enabled themes should notice no change.
- Removes Portal config flag as Portal is now enabled by default
- Removes old members script as Portal is backward compatible with it
- Changes `{{content}}` helper to show default CTA in case of restricted content access
- `accent_color` setting is no more behind the dev experiment flag and included by default
- Adds migration to switch off Portal button setting for all existing sites which don't have Portal enabled in beta
closes#12263
- Express parses repeated query parameters as an array (req.query properties). Because there is no clear reason on why not to support this behavior extended order parameter parsing logic to handle arrays. This follows the rule of "liberal inputs, conservative outputs"
- Example supported query string for ordering can now look like: `?order=featured&order=published_at asc`, the priority of the order stays the same with the most significant appearing first and least significant last
refs #11572
- Filtering by fields coming from posts_meta table did not work for post resources. This was due to lack of support for these types of operations on NQL layer. The approach taken here is using same way filtering was done for many:many relations and generates a `WHERE IN` filtering clause. In the future we could look into adding preloading of 1:1 relations which should allow getting rid of `WHERE IN` in favor of `JOIN` and filtering directly by field names.
- Changed structure of `EXPANSIONS` filter configuration. Current approach was based on "bag of all the things". Such structure will become problematic as more fields are added. For example, adding all the fields from 1:1 relation posts:posts_meta might collide with any other relations that would have similar naming like meta_description from tags table (if it were was added).
- Bumped nql version to 0.5.0. This adds filtering support to 1:1 relations
- Added filter expansions which can be unique per model Previous approach with single global expansions lookup wasn't working in case different models would need to declare expansion for same field names. Having a `filterExpansion` method per model works in a similar convention other filter related model methods do (e.g. enforcedFilters, defaultFilters)
no issue
- adding user variables via the mailgun API when sending emails means that events related to email have those variables attached to them
- adding the `email.id` value to user variables means we can easily associate mailgun events to emails, otherwise we'd have look up the batch via `email_batches.provider_id` then use a join to get back to the associated email
- currently if Ghost has a version of something like
`3.37.0-pre.abc.def`, Ghost will return `3.37.0-pre.abc` as the full
version
- this hides parts of the version which are useful for debugging
- this commit updates the logic to join together all prerelease elements
so we keep the full string
no issue
The email change verification template was using the same as for `subscribe`, which did not have the right messaging that needs to be communicated about the action thats happening in this case. Updates the email template to same as what we use for email verification for support/newsletter address
closes https://github.com/TryGhost/Ghost/issues/12253
- Allows using custom action param for requests from Portal by using a new `requestSrc` option that is passed down when a request for magic link is made via Portal
refs https://github.com/TryGhost/Ghost/issues/12256 , https://github.com/TryGhost/Ghost/issues/12255
Currently when listing subscriptions for Members in both the Admin and the Theme, we only show the subscriptions which have a status of trialing or active.
Based on discussion, the `unpaid` and `past_due` states on Stripe also represent owner's intention of considering a subscription as active instead of `cancelled`, so we allow any subscriptions under these 2 states to be also listed for a member and consider them as `paid`.
- Subscriptions will go into a past_due state if the payment is missed, this should be considered a grace period where the member still has access.
- After this the subscriptions will either go to the unpaid or the cancelled state - this can be configured on an account by account basis in the Stripe dashboard. `unpaid` is considered as an intention to keep the subscription to allow for re-activation later.
no refs.
- Members settings were moved to a new page [here](https://github.com/TryGhost/Ghost-Admin/pull/1736)
- updated members related redirect for from/newsletter email address change to point to new page
no refs
- The token generation logic for single use token was replacing only the first instance of + or / to make the token URL safe, instead of replacing all instances which caused a bug where token was not validated properly in case it included multiple + or / in it.
- The fix ensures replacing all the + or / in the token with URL safe _ or - so it can be properly validated via magic link
closes https://github.com/TryGhost/Ghost/issues/11548
- bumps @tryghost/kg-default-cards
- introduces a breaking change where `imageOptimization` is passed as an options argument instead of separating out individual config options
- fixes width/height attributes for images in gallery cards being larger than the max default width used when resizing images
no issue
- centralises definition of max width and allows customisation if needed
- allows for passing of the config value through to rendering libraries
no issue
- standard browse/read/add/edit/destroy API endpoints for snippets resource
- updates `@tryghost/admin-api-schema` dependency to version that includes snippet definition and schemas
no issue
- all staff users can browse/read snippets so they're usable in the editor for everyone
- only administrators, editors, and admin integrations are able to create/edit/delete snippets
no issue
- minimal table structure required for the first iteration of content snippets
- snippets are stored pieces of re-usable content that could effectively be entire posts so the `mobiledoc` field length matches the `posts.mobiledoc` field length
no issue
- Member email settings - newsletter and support - was previously being set to null for all cases, which can lead to `null` values being set for those addresses on a successful import whereas the expected behavior is to retain the old values.
- Adds the 2 email settings to ignored settings list so they are not changed during import process
no issue
- In a case where support address is null, member's site data won't load as there was no fallback.
- The fix adds default `noreply` as the support address for site data
no-issue
The Action model loops through all registered models when the file is
loaded, by loading the model last, we ensure that it can read all
models, rather than an arbitrary selection which come before it.
closes#12273
- `comped` field has been allowed when editing a member or importing from a CSV. There has been a usecase (Zapier Integration) for API client to create a member with "Complimentary" plan, which made this change necessary
- Previously the logic for comped field was to skip and continue member record creation if Stripe was not connected. Now we throw an error - same as the one we have been throwing before when stripe_customer_id field was passed in. The implication of this change is that we won't be creating any record now if comped === true and Stripe is disabled.
- Bumped admin-api-schema-package. Contains `comped` schema change so this field gets passed through to controller
no issue
When scheduling a post to publish+send the "view online" link was pointing at https://site.com/404/ rather than the published post's url.
The problem occurred because the `/schedules/` endpoint wraps it's post read+edit calls in a transaction. Context:
- when a post is published with with the "send email" option the email record is immediately generated and added to the API response, as part of the email record generation we render the email content including fetching the url for the "view online" link
- urls for all resources are handled by our `url` service, that service updates it's internal cache based upon model events such as the "edited" event triggered when a post is published
- if the posts API controller is given a transaction, the email record is also generated inside of that transaction however at this point the `url` service will not have been updated because the post record hasn't been committed meaning it has no available url for the post
Fix:
- removed the `models.Base.transaction()` wrapper around the post read+update in the `/schedules/` API controllers
- we don't need a transaction here. It was added as protection against another write request coming in between the `/schedules/` controller reading a post and publishing a post but we already have protection against that in the form of collision detection - if a write request comes in and commits between the schedules controller reading the post and updating it, the scheduler's update call will fail with a collision error at which point the scheduler itself should retry the request which could then publish the post successfully if everything else is in order
closes https://github.com/TryGhost/Ghost/issues/12260
- if a card type was not explicitly chosen (i.e. a url was pasted into the editor) then abort fetching the oembed endpoint if we detect it's a `wp-json` oembed and return a bookmark card payload instead
- cleaned up an unused argument in the internal `fetchBookmarkData()` method
closes https://github.com/TryGhost/Ghost/issues/12257
- there was a destructuring problem introduced in the recent email refactor which meant the correct replacement data was not being passed over to the Mailgun provider when sending email
closes https://github.com/TryGhost/Ghost/issues/12259
- adds a `DISTINCT` to the query used to fetch member rows when generating an email recipient list
- this increases query time 2.7s vs 1.6s locally with ~94k paid members but once the `members.paid` column is implemented this slow query can be removed
closes https://github.com/TryGhost/Ghost/issues/12247
- Internal preview controller was lacking "mapping" call to post object which handled not only missing meta attribute information but lots of other mappings (e.g. users, tags, etc.)
- Have added a regression test to catch issues like this in the future
no issue
- fixed passing of errors up through send/processBatch/processEmail
- fixed errant overwrite of email status with a "submitted" status after a failure had occurred
no issue
- wrap email batch/recipient record creation in a transaction so if an error occurs during creation we're not left with a partially created batch/recipient set in the database
no issue
- if an error occurred whilst creating email batch/recipient records the email status was never updated and was left in the 'pending' status
- adjusted the error handling to update the email status and record the error message if such a scenario occurs
no issue
- the paid-member SQL query that is obtained using `models.Member.getFilteredCollectionQuery({paid: true})` can return multiple columns with the same name (eg, `email`, `name`), when that happens the last column with duplicate names "wins" and it's value is used in the resulting knex row instance
- in the `mega` service when fetching email recipient rows we ran into this problem, to avoid it we adjust the query to explicitly select only the data from the `members` table
no issue
- we've had an issue with emails failing due to unexpectedly missing data when inserting email recipient rows
- added a validation check before adding recipient details along with a log so that invalid data can be investigated
no issue
- We had previously allowed accent_color setting for member site settings behind portal flag, but Ghost Admin also needs the public site setting with accent color to correctly reflect the accent color when flag is switched on
- Removes deletion of accent color setting when behind the Portal flag OR dev experiment flag
