refs https://github.com/TryGhost/Team/issues/579
Currently the members signup setting is explicitly yes/no to allowing free members signup, with the implication that when set to "no" members is still active but members have to be created via Stripe or the admin API.
This change renames the setting and changes its type to allow more than a binary option.
- migration to create/update the new setting based on the old value
- free signup = "all", no free signup = "invite"; matches the current UI for this setting
- rename setting everywhere it's used/tested against
- modify `getAllowSelfSignup()` used to configure members packages to only return `true` when the new setting is set to `'all'` to match behaviour to the older setting
- update importer to rename the setting when importing from an older Ghost version
refs https://github.com/TryGhost/Team/issues/586
- Passes new Product, Stripe Price and Stripe Product models to members API service
- Allows members service to populate the tables for existing plans and products
https://github.com/TryGhost/Team/issues/599
- When custom integration limit is enabled all webhooks belonging to integrations have to be disabled as well. The result is the webhook would stop working and to discover that a user would need to navigate to Admin UI (this changes is yet to come, see refed issue)
https://github.com/TryGhost/Team/issues/599
- When custom integration limit is enabled all requests from existing integrations should not be accepted. With the exception of internal integrations like backup and scheduler
refs https://github.com/TryGhost/Team/issues/599
- This is a precursor change to tests which verify the hostSettings limits are working correctly
- Bumped limits-service version which allows for multiple calls of loadLimits on the same service instance
refs https://github.com/TryGhost/Team/issues/510
- This version bump includes follwing interface improvements of the limit-service package: passing in errors as a parameter to "loadLimits" and allowing for custom "currentCountQuery" method implementations per limit
refs https://github.com/TryGhost/Team/issues/397
The default newsletter/support email address for a site is currently setup as noreply@DOMAIN , which means for a custom domain setup with www the email address becomes noreply@www.somesite.com which is not the expected behavior normally. This removes the `www` subdomain if present for those email addresses, but doesn't change any other subdomain
refs https://github.com/TryGhost/Team/issues/536
From 4.0, we ensure and require that accent colour is always set. This change removes hardcoded accent color fallbacks to avoid confusion as well as cause accidental fallback that is undesired causing themes to look different
closes https://github.com/TryGhost/Team/issues/570
When site owner/admin updates their newsletter/support email address from settings, they receive an email with confirmation link which on success takes them to Ghost Admin on email settings screen with a toast about success. Since the path for email settings in Ghost Admin changed in v4, the fix updates the redirect link to new Admin settings URL.
refs 4dc413d6a1
- Fixed failing test cases which were designed to check non-major upgrade messages. There's no clear use cases for those, but still worth keeping such case in mind
closes https://github.com/TryGhost/Team/issues/564
refs https://github.com/TryGhost/Ghost/issues/10236
- The notification to upgrade to new 4.0 Ghost version was still visible to users after upgrading the instance to 4.0. This was caused by notification filtering not taking into account 3.x or 4.x versions.
- The fix filters out notifications that detect a major version notification using `x.0 is now available` pattern and compares current version to that major. This should future proof the issue from happening in Ghost 5.0 (but a proper holistic fix is preferable!)
no issue
Comped members were not able to view paid-member content because content gating was only looking for `member.status === 'paid'` which doesn't take into consideration members on a "complimentary" plan.
- added front-end acceptance tests for member access to posts
- updated content-gating check to take comped members into consideration
refs: https://github.com/TryGhost/Team/issues/510
- The current member limit was implemented as a member-specific concept
- The new limit service is much more generic, here we are swapping old for new
- The updated concept here is blocking all publishing, not just email sending, when a site is over its member limit
- To determine that we are publishing a post, we must be in the model layer. The code has been moved to the permissible function which makes sense as this is a permissions error that we are throwing
- I've left the extra check for email retries in, in case there is some loophole here (but we may wish to change it)
refs https://github.com/TryGhost/Team/issues/510
- When the host config was introduced it was incorrectly introduced as host_settings instead of hostSettings
- All other Ghost config uses camelCase, so changing this now before it becomes a problem
- Note: Also removed some rogue return awaits that don't make sense. It's not possible to hit that case, but cleaning up anyway
refs https://github.com/TryGhost/Team/issues/509
- During 3.x we standardised on "lang" instead of "default_locale" for the site setting, which was an assumption based on an earlier change to @site.lang to make <html lang="@site.lang"> read nicer. This was a mistake as the field represents more than "lang" its a "locale". With this changeset we introduce a transition to use "locale" name for the value instead of "lang"
- Adds `@site.locale" value in as well as new 'locale' property in Content API's response
- "lang" will be considered as deprecated starting with API v4 and will be dropped completely with API v5
refs https://github.com/TryGhost/Team/issues/509
- Flipping around key/newKey pair allows to map multiple keys to the same field in the settings
- This becomes handy when there's a need to deprecate a field. For example, we are about to introduce a 'locale' setting which would need to map to 'lang' db key, with current structure it's impossible to have many:1 mapping because it's impossible to have duplicate keys in the JS object ("hash")
refs: https://github.com/TryGhost/Team/issues/510
- added and wired up the new limit service, which is a lazy-loaded service
- this handles the case that there are host limits set in config, and wraps all the logic needed for detecting exceeded limits & throwing limit errors
- expects limits to be set in config under `host_settings.limits`
- supported limits are managed in the limit service, outside of core
refs https://github.com/TryGhost/Ghost/issues/12610
refs https://github.com/mailgun/mailgun-js-boland/blob/v0.22.0/lib/request.js#L285-L333
The mailgun domain is used by the mailgun API to construct the URL for
the API. e.g for a domain of "mg.example.com" the URL for the API
messages would look like:
https://api.mailgun.net/v3/mg.example.com/messages
One weird thing about the mailgun API is that if the path does not map
to an API endpoint, then instead of a 404, we get a 200, with a body of
"Mailgun Magnificent API".
The `mailgun-js` library which we use, expects a JSON response, and will
return a body of undefined if it does not get one.
This all resulted in us trying to read the property `id` of an undefined
`body` variable. The fix here is to reject the containing Promise, if
there is no body. So that the default error handling will kick in.
refs https://github.com/TryGhost/Team/issues/513
- Bumping version as this is now the latest stable API.
- The change might cause this particular side effect (acceptable for major version): if a member requests a login URL when the instance is on 3.x version and site owner upgrades do 4.x before member authenticates through login link, the login will fail and the member will have to request a new login URL.
refs https://github.com/TryGhost/Ghost/pull/12541
refs https://github.com/TryGhost/Ghost/pull/12689
- the analytics job had been switched to create it's own instance of EmailAnalyticsService to avoid requiring logging but the analytics extraction branch was created before this change and wasn't picked up when merging
- pulled `queries` option object into a separate file for re-use
- updated `fetchLatest` job to conform to extracted library interface
closes https://github.com/TryGhost/Team/issues/493
- all functionality except that directly related to Ghost's database and business logic now lives in external packages
- @tryghost/email-analytics-service
- @tryghost/email-analytics-provider-mailgun
refs https://github.com/TryGhost/Ghost/issues/12716
- This change is made to avoid using now deprecated 'v2' API anywhere in the codebase.
- Switching to 'v4' should not cause sideeffects as this parameter is always present within the URL when Admin API is used
refs https://github.com/TryGhost/Ghost/issues/12716
- The code in serializePostModel was broken and always defaulted to 'v3'! It refered to non-existent `model.get('api_version')` there's no such field in posts model! Changed the implementation so that the API version is passed in as a parameter to the method instead
- The style of providing "defaults" everywhere creates a need for future maintenance when we bump the version e.g in Ghost v5. Maybe reworking these methods to require a passed version and throwing an error instead would be more maintainable long-term?
refs https://github.com/TryGhost/Ghost/issues/12716
- The default API in Ghost v4 will be `'v4'`. When a new webhook is created or any data passes through serialization it should assume `v4` as a fallback if not specified.
refs https://github.com/TryGhost/Team/issues/498
Requests to fetch the data of the logged in member made without a
session cookie were responsing with 400 Bad Request. This was incorrect
and always should have been a 401 Unauthorized.
refs https://github.com/TryGhost/Team/issues/498
This was filling up logs with less than useful information - for every
single request made by a non-member to the frontend. Be gone!
refs https://github.com/TryGhost/Ghost/issues/12602
As part of the member events, we added a third status of 'comped'.
Members with a status of 'comped' should still be considered paid, so
this fixes the definition of the paid flag to take that into account.
refs https://github.com/TryGhost/Team/issues/221#issuecomment-759105424
- Mailgun responds to an email send with a provider id in the format `<x@y.com>` but everywhere else it's used in their API it uses the format `x@y.com`
- updates email batch save to strip the brackets, and migration removes brackets from existing records so we no longer have to add special handling for the stored id any time we use it
refs https://github.com/TryGhost/Ghost/issues/12496
By requiring the models layer the shared logging util was being required as a side-effect causing the open file descriptors problem to continue. Removing logging from the models layer isn't feasible due to deep require chains spreading across the codebase, it's much quicker to remove the need for models in the analytics job.
- models layer was only needed because it's used by the session service
- updated analytics job to create it's own instance of `EmailAnalyticsService` rather than the default instance in order to pass in custom dependencies
- pass in custom `logging` object that uses `parentPort.postMessage` as a way of writing log output
- pass in custom `settings` object that returns settings that have been manually fetched and cached during job instantiation
refs https://github.com/TryGhost/Ghost/issues/12496
- Handling logging in the main thread avoids file handle leaks which happen due to leaky implementation of bunyan logger (see referenced issue for more context)
- Bumped job-manager version to allow for `workerMessageHandler` callback funciton
refs https://github.com/TryGhost/Ghost/issues/12496
- Using ghost-ignition logging caused file handle leaks. As there is no straight-forward way to handle write streams with bunyan (ghost-ignition's underlying logging library) this method of logging was chosen as an alternative to keep the amount of open file handles to minimum
- The follow up changes will include custom formatter for jobs service which should make logging match the same format as has been used inside the jobs
refs https://github.com/TryGhost/Ghost/issues/10318
- Because members is effectively "enabled" by default starting Ghost 4.0 have hardcoded labs setting to be such. The alternative of removing this key from labs would be equivalent to `labs.members === false` which is undesireable and would mean additional work on theme developer's side.
refs https://github.com/TryGhost/Ghost/issues/12602
* Updated members_status_events table
By replacing the `status` column with a `from_status` and `to_status`
column, we are able to track the changes between multiple statuses
easier, and accumulate the data. e.g. the delta of paid members in a
given time range is the sum of the `to_status` columns set to 'paid'
minus the sum of the `from_status` columns set to 'paid' within that
time range
* Updated MEGA to handle addition of 'comped' status
With the addition of the 'comped' status, we need to ensure that MEGA
will still send emails to the correct recipients. I've opted to use an
"inverse" filter, as that is the intention of the free/paid split in
MEGA - as far as MEGA is concerned, "free" is the opposite of "paid"
* Updated customQuery for MemberStatusEvent
With the `status` column replaced with `from_status` and `to_status`
this allows us to fix and update the customQuery to correctly accumulate
the data into deltas over time, broken down by day.
* Populated members_status_events table
As the table will be used to generate deltas, we need to backfill the
data so that existing sites will be able to sum up the deltas and
calculate correct data.
The assumptions used in backfilling is that a Member's current status,
is their only status.
no-issue
This removes all references to the members labs setting, any code that was run conditionally behind this flag now runs unconditionally.
* Removed usage of Members labs flag
* Removed tests for Members disabled
* Added dynamic keypair generation for when setting is missing
no refs
- Updates member model serializer to directly set subscriptions on member object instead of `stripe.subscriptions`
- Updates all references to members subscriptions from nested `stripe.subscriptions` to `subscriptions`
- Updates v3 API serializer to still use `stripe.subscriptions`
- Updates tests
no-issue
* Removed support for paid param from v3 & canary API
* Updated active subscription checks to use status flag
* Updated MEGA to use status filter over paid flag
* Removed support for paid option at model level
* Installed @tryghost/members-api@1.0.0-rc.0
* Updated members fixtures
no issue
We added `portal-action` and `requestSrc` in 3.x to allow Portal to handle notifications only for auth actions trigged while using it directly, so that existing themes are not affected in any way. Going forward in 4.0, we don't want to have any special handling in backend for Portal but instead expect themes to handle any Portal specific behavior directly.
- Removes setting of `portal-action` for auth actions like signup
- Removes `requestSrc` being passed through to determine portal actions
refs #12537
- Refactored overuse of rewire mocking blog-version.
- Fixed a bug introduced along the way when duplicate notifications errored instead of returning empty result
refs #12537
- notifications controllers were overbloated with non controller related code and were identical. It is important to reduce unnecessary code ahead of v4 API introduction
- Follow up commit will transform newly created module into a class following DI pattern
refs #12537
- Class syntax is preferred over module functions because of constructor parameter injection (DI) which allows for easier module decoupling and testing
refs #12537
- `stats` method in members controller is quite big and does much more then controller method code should - few calls to relevant modules
- Extracted code "as is" into members serivce
- Next step will be to refactor this module as a class pattern with DI parameters
refs https://github.com/TryGhost/Team/issues/446
Mailgun permanent failure events do not always correspond to unsubscribe-level events as originally thought, meaning some members could be unsubscribed unexpectedly due to delivery hiccups.
- disabled auto-unsubscribe on permanent failure events in the analytics event processor
- list maintenance will be added back in the future via alternative means
closes https://github.com/TryGhost/Ghost/issues/12492
The changes to email processing models had set replyTo address for an email batch as `reply_to` instead of `replyTo` which was not picked by mailgun service for setting newsletter reply address
refs https://github.com/TryGhost/Ghost-Utils/issues/118
- Duplicating error handling across jobs is not best developer experience. Also, having custom error handling logic did not allow for recommended worker script behavior: allowing for unhandled exceptions to bubble up and be managed by parent process
no issue
- sqlite will store a float in an integer column due to it's type affinity resulting in long decimal numbers in the UI when we're expecting an integer
- use the `ROUND()` function to ensure we're storing integers in place of floats when performing open rate average calculations
no-issue
This module encapsulates the work around performing imports, it
currently uses the concept of a "Job" which at the moment is not
persisted to the database, however when we want to look at resuming
imports after a server restart, this should give us the flexibility to
do it.
refs https://github.com/TryGhost/Ghost/issues/12461
- added two default aggregations for overall email count and opened email count
- when number of tracked emails is sufficient add the open rate aggregation to the update query
refs https://github.com/TryGhost/Ghost/issues/12421
requires https://github.com/TryGhost/Ghost/pull/12457
- updates stats aggregator to calculate and store an open rate for each member
- uses two queries because I couldn't find a reasonable approach to perform the update in a single query as per the email aggregation
- benchmarked locally at <1sec/1000members
- will not store an open rate unless the number of tracked emails sent to a member is above a certain threshold (defaults to 5) to avoid new members being heavily weighted
- fixes typo in EmailAnalytics that was stopping member stats from being aggregated
no issue
- job registration was checking for submitted emails in it's email count but the job registration method is called as soon as an email is created meaning the email has a status of 'pending' which prevented the analytics job from being started until a second email was sent
no issue
- email analytics may be desirable to fully switch off in certain circumstances, when that happens we want to prevent related background jobs from running and expose the feature flag via the config endpoint in the Admin API so that clients can adjust accordingly
no issue
- if emails are older than 30 days we wouldn't be able to fetch any analytics for them and if a site used emails in the past but is no longer using them it doesn't make sense to keep potentially expensive background worker threads spinning up
no issue
- recurring jobs spin up worker threads which can be quite CPU intensive even when not performing much processing, this can be problematic in environments where there are many Ghost instances running
- updated the email job scheduling to be skipped on bootup when there are no emails in the database and to be started when the first email is created as long as we're not in testing env
- increase analytics job schedule from every 2 minutes to every 5 minutes to help spread the load further across instances
- users imported from CSV with no created_at date where having their created_at date being stored as an int rather than a datetime.
- this was causing parsing issues with the graph so this commit fixes the formatting
no issue
- the 4.2.0 version of `sqlite3` that we're using is not compatible with `worker_threads`
- 5.0.0 should add support this but there are other errors
- 5.0.1 is released but not published (https://github.com/mapbox/node-sqlite3/issues/1386)
no issue
- added `EmailAnalyticsService`
- `.fetchAll()` grabs and processes all available events
- `.fetchLatest()` grabs and processes all events since the last seen event timestamp
- `EventProcessor` passed event objects and updates `email_recipients` or `members` records depending on the event being analytics or list hygiene
- always returns a `EventProcessingResult` instance so that progress can be tracked and merged across individual events, batches (pages of events), and total runs
- adds email_id and member_id to the returned result where appropriate so that the stats aggregator can limit processing to data that has changed
- sets `email_recipients.{delivered_at, opened_at, failed_at}` for analytics events
- sets `members.subscribed = false` for permanent failure/unsubscribed/complained list hygiene events
- `StatsAggregator` takes an `EventProcessingResult`-like object containing arrays of email ids and member ids on which to aggregate statistics.
- jobs for `fetch-latest` and `fetch-all` ready for use with the JobsService
- added `initialiseRecurringJobs()` function to Ghost bootup procedure that schedules the email analytics "fetch latest" job to run every minute
refs https://github.com/TryGhost/Ghost/issues/12256
We no longer want to filter out cancelled subscriptions, so we are able
to remove the whereIn clause of the relation.
* Fixed paid flag on member
* Fixed content gating for members
Now that the subscriptions for a member include all of them, we must
explicitly check that the member has an active subscription in order to
consider them "paid"
no issue
- the `'bulk-email`' tag was only being added to bulk emails if another more specific tag was set up via config
- we always want the `'bulk-email'` tag to be present for better event filtering
refs #12366
This implements redirection based on the settings for successful member sign up!
- Removes support for redirecting to `req.path` afterwards, this was never used and
we now have a more configurable implementation.
- Retains redirection to the homepage for unsuccessful sign up (invalid/expired token)
no-issue
We want to give users to ability to customise the content of their newsletter, and the first step
toward that is a setting in which we can store text or html to embed in the template
refs https://github.com/TryGhost/Ghost/issues/12355
- Adds new default settings for newsletter customisations - `newsletter_show_badge`, `newsletter_show_header` and `newsletter_body_font_category`
- Adds migrations to update group for new settings
- Add migration to update settings based on existing config value for newsletter settings
- Passes new newsletter settings to newsletter template and updates design based on them
- Fix tests
no-issue
* Used email_recipient_filter in MEGA
This officially decouples the newsletter recipients from the post
visibility allowing us to send emails to free members only
* Supported enum for send_email_when_published in model
This allows us to migrate from the previously used boolean to an enum
when we eventually rename the email_recipient_filter column to
send_email_when_published
* Updated the posts API to handle email_recipient_filter
We now no longer rely on the send_email_when_published property to send
newsletters, meaning we can remove the column and start cleaning up the
new columns name
* Handled draft status changes when emails not sent
We want to reset any concept of sending an email when a post is
transition to the draft status, if and only if, and email has not
already been sent. If an email has been sent, we should leave the email
related fields as they were.
* Removed send_email_when_published from add method
This is not supported at the model layer
* Removed email_recipient_filter from v2&Content API
This should not be exposed on previous api versions, or publicly
* Removed reference to send_email_when_published
This allows us to move completely to the email_recipient_filter
property, keeping the code clean and allowing us to delete the
send_email_when_published column in the database. We plan to then
migrate _back_ to the send_email_when_published name at both the
database and api level.
no issue
- set `emails.track_opens` to `true` when the `enableDeveloperExperiments` flag is set
- update mailgun bulk-email provider to pass the open-tracking header to Mailgun when the email's `track_opens` flag is set
no issue
- adding user variables via the mailgun API when sending emails means that events related to email have those variables attached to them
- adding the `email.id` value to user variables means we can easily associate mailgun events to emails, otherwise we'd have look up the batch via `email_batches.provider_id` then use a join to get back to the associated email
no issue
The email change verification template was using the same as for `subscribe`, which did not have the right messaging that needs to be communicated about the action thats happening in this case. Updates the email template to same as what we use for email verification for support/newsletter address
closes https://github.com/TryGhost/Ghost/issues/12253
- Allows using custom action param for requests from Portal by using a new `requestSrc` option that is passed down when a request for magic link is made via Portal
no refs.
- Members settings were moved to a new page [here](https://github.com/TryGhost/Ghost-Admin/pull/1736)
- updated members related redirect for from/newsletter email address change to point to new page
no issue
- In a case where support address is null, member's site data won't load as there was no fallback.
- The fix adds default `noreply` as the support address for site data
closes https://github.com/TryGhost/Ghost/issues/12257
- there was a destructuring problem introduced in the recent email refactor which meant the correct replacement data was not being passed over to the Mailgun provider when sending email
closes https://github.com/TryGhost/Ghost/issues/12259
- adds a `DISTINCT` to the query used to fetch member rows when generating an email recipient list
- this increases query time 2.7s vs 1.6s locally with ~94k paid members but once the `members.paid` column is implemented this slow query can be removed
no issue
- fixed passing of errors up through send/processBatch/processEmail
- fixed errant overwrite of email status with a "submitted" status after a failure had occurred
no issue
- wrap email batch/recipient record creation in a transaction so if an error occurs during creation we're not left with a partially created batch/recipient set in the database
no issue
- if an error occurred whilst creating email batch/recipient records the email status was never updated and was left in the 'pending' status
- adjusted the error handling to update the email status and record the error message if such a scenario occurs
no issue
- the paid-member SQL query that is obtained using `models.Member.getFilteredCollectionQuery({paid: true})` can return multiple columns with the same name (eg, `email`, `name`), when that happens the last column with duplicate names "wins" and it's value is used in the resulting knex row instance
- in the `mega` service when fetching email recipient rows we ran into this problem, to avoid it we adjust the query to explicitly select only the data from the `members` table
no issue
- we've had an issue with emails failing due to unexpectedly missing data when inserting email recipient rows
- added a validation check before adding recipient details along with a log so that invalid data can be investigated
no issue
- there was an typo in the recent email sending refactor that resulted in `Error: 'to' parameter is missing` errors when sending email previews and bulk emails
closes#12244
As per RFC 6454 the Origin header MUST be set to the string 'null' when
in a "privacy-sensitive" context. We were not handling this string and
this was causing errors. This commit updates all checks of the 'Origin'
header to treat the value 'null' as if the header was not present.
ref: https://tools.ietf.org/html/rfc6454#section-7.3
