no issue
- The accent_color setting was being removed from members site data when behind portal flag
- The accent color setting is now allowed in members site data for all cases as it doesn't make any sense to remove it specifically from here where we already have all the other Portal settings included which is a dev/portal flag feature anyways
no-issue
By using the "email" validation, we were validating emails in CSV
imports using a different validator to the rest of the API. AJV's built
in email validation was failing on emails with "special" characters,
such as letters with an umlaut above them.
This commit brings the validation for CSV imports in line with the rest
of the API.
no issue
The email table should be a reference for all data that was used when sending an email. From and Reply-to addresses can change over time and we don't have any other reference for their value at the time of sending an email so we should store them alongside the email content.
- schema updated with `from` and `reply_to` columns
- both are set to `nullable` because we don't have historic data (can be populated and changed in later migrations if needed)
- neither `from` or `reply_to` have `isEmail` validations because they can have name+email in an email-specific format
- will help keep concerns separated in the future. `mega` service can deal with all of the email contents/properties, and the `bulk-email` service's concerns are then only email sending and any provider-specific needs
refs #12033
- Allowing to change parent integration opens up possible security holes and has no clear usecase at the moment. After a webhook record is created it should not be possible to change parent integration.
- Had do partially duplicate JSON schema definition from webhooks definition as there is no proper composition technique available in current version of JSON Schema.
refs https://github.com/TryGhost/Ghost/issues/12033
refs https://github.com/TryGhost/Ghost/issues/10567
- Creating a webhook without valid parent integration leads to orphaned webhook records, which shoult not ever happen
- This scenario is only possible for non-integration authentication,
because in case of integration being authenticated it's id is
automatically assigned to creatd webhook
refs #11729
- When ordering is done by fields from a relation (like post's `meta_title` that comes form `posts_meta` table), Bookshelf does not include those relations in the original query which caused errors. To support this usecase added a mechanism to detect fields from a relation and load those relations into query.
- Extended ordering to include table name in ordered field name. The information about the table name is needed to avoid using `tableName` within pagination plugin and gives path to having other than original table ordering fields (e.g. order by posts_meta table fields)
- Added test case to check ordering on posts_meta fields
- Added support for "eager loading" relations. Allows to extend query builder object with joins to related tables,
which could be used in ordering (possibly in filtering later). Bookshelf does not support ordering/filtering by proprieties coming from relations, that's why this kind of plugin and query expansion is needed
- Added note about lack of support for child relations with same property names.
closes#12038
Previously we were emitting changed events for _all_ settings which would
cause any listeners for those to be triggered, this ensures that listeners are
only triggered if the corresponding setting, _did_ in fact change.
no-issue
This change allows themes to collect member names on signup with use of the
`data-members-name` attribute on an input element in the `data-members-form`
refs #11729
- Having a plugin allows to reason about ordering better and gives way
to further extraction away form the core.
- Just like with filtering, ordering falls into similar category of having effect on knex'es query builder object extension through additional statements - ORDER BY in this case.
- Because there were bugs associated with use of permittedAttributes inside of `parseOrderOption` method, introduced separate `orderAttributes` function which returns only those fields which are orderable (https://github.com/TryGhost/Ghost/issues/11729#issuecomment-685740836)
refs #11878
- When password reset link is invalid previous messaging left the user
without clear information about why the reset failed and what they could do about it.
- Updated messaging around password reset tokens including detection of
when password token has invalid structure, has expired or has already
been used
This reverts commit 80af56b530.
- reverting temporarily so that all associated functionality can be merged in a single release
- creating email batch/recipient records without using them would cause inconsistent data
no-issue
* Added SingleUseTokenProvider to members service
This implements the TokenProvider interface required by members-api to
generate magic links. It handles checking if the token is expired and
pulls out any associated data.
Future improvments may include the email in the error for expired
tokens, which would make resending a token simpler.
* Passed SingleUseTokenProvider to members-api
This sets up the members-api module to use the new single use tokens
* Installed @tryghost/members-api@0.30.0
This includes the change to allow us to pass a token provider to the members-api
no-issue
This is a model for the tokens table, which handles the single use
aspect by customising the `findOne` method to automatically destroy the
model after reading from it
no-issue
refs: https://github.com/TryGhost/Members/commit/63942f03
The above commit updated all magic-link methods to be async, this change
ensures that we will handle a Promise return when it's updated in Ghost.
`await` has no effect on non Promise return values so it's safe to add
this now.
no-issue
After discussion with Matt, we decided that 192 bits for the token is a
good number, as it has no padding when base64 encoded and is more secure
than 128 bits, whilst still a managable size.
no-issue
This is a table to store single use tokens for use in magic links, the
columns are as simple as possible at the moment and are designed as:
id - standard ObjectID like all of our tables
token - 128bit base64 encoded string
data - arbitrary data to store against the token
created_at - timestamp to allow for expiry to be implemented for tokens
no issue
- In a recent change to ownership verification email flow, we changed the FROM address of ownership verification mails to use the same email as the one we are verifying, aka TO address.
- Email clients like Gmail flags off such emails as possible spam
- Fix updates the `FROM` address to `noreply@domain.com` where domain.com is domain for TO address
- In case the TO is already noreply@domain.com, we use no-reply@domain.com to bypass the same address restriction.
no issue
- The new Portal config flag allows switching on Portal conditionally with config
- The dev experiment flag still works for enabling Portal
- The flag currently defaults to `false` as Portal is still a beta feature and switched off by default
- We expose it on the admin api config endpoint so that the Ghost-Admin client can use it to conditionally render Portal settings
no issue
- Members site data was not appending blog domain for default support address which is `noreply`
- The change allows Portal to use default support address correctly
requires https://github.com/TryGhost/Ghost/pull/12192
- added initial `EmailBatch` and `EmailRecipient` model definitions with defaults and relationships
- added missing `post` relationship function to email model
- fetch member list without bookshelf
- bookshelf can add around 3x overhead when fetching the members list for an email
- we don't need full members at this point, only having the data is fine
- if we need full models later on we can push the model hydration into background jobs where recipient batches are fetched ready for an email to be sent
- bookshelf model instantiation of many models blocks the event loop, using knex directly keeps concurrent requests fast
- store recipient list before sending email
- chunk already-fetched members list into batches and insert records into the `email_recipients` table via knex
- chunked into batches of 1000 to match the number of emails that Mailgun accepts in a single API request but this may not be the absolute fastest batch size for recipient insertion:
| Batch size | Batch time | Total time |
| ---------- | ---------- | ---------- |
| 500 | 20ms | 4142ms |
| 1000 | 50ms | 4651ms |
| 5000 | 170ms | 3540ms |
| 10000 | 370ms | 3684ms |
- create an email_batch record before inserting recipient rows so we can effeciently fetch recipients by batch and store the overall batch status
no issue
We want to store a list of recipients for each bulk email so that we have a consistent set of data that background processing/sending jobs can work from without worrying about moving large data sets around or member data changing mid-send.
- `email_batches` table acts as a join table with status for email<->email_recipient
- stores a provider-specific ID that we get back when submitting a batch for sending to the bulk email provider
- `status` allows for batch-specific status updates and picking up where we left off when submitting batches if needed
- explicitly tying a list of email recipients to a batch allows for partial retries
- `email_recipients` table acts as a join table for email<->member
- `member_id` does not have a foreign key constraint because members can be deleted but does have an index so that we can efficiently query which emails a member has received
- stores static copies of the member info present at the time of sending an email for consistency in background jobs and auditing/historical data
refs #2635
- Adds 'Location' header to endpoints which create new resources and have corresponding `GET` endpoint as speced in JSON API - https://jsonapi.org/format/#crud-creating-responses-201. Specifically:
/posts/
/pages/
/integrations/
/tags/
/members/
/labels/
/notifications/
/invites/
- Adding the header should allow for better resource discoverability and improved logging readability
- Added `url` property to the frame constructor. Data in `url` should give enough information to later build up the `Location` header URL for created resource.
- Added Location header to headers handler. The Location value is built up from a combination of request URL and the id that is present in the response for the resource. The header is automatically added to requests coming to `add` controller methods which return `id` property in the frame result
- Excluded Webhooks API as there is no "GET" endpoint available to fetch the resource
closes#12045
- When member's email is updated to an already existing email of different member it caused table's unique constraint error, which was not handled properly.
- Added handling for this error similar to one in members `add` method.
closes#11999
- When the routes.yaml file changes (manually or through API) we need
to store a checksum to be able to optimize routes reloads in the future
- Added mechanism to detect differences between stored and current routes.yaml hash value
- Added routes.yaml sync on server boot
- Added routes.yaml handling in controllers
- Added routes hash synchronization method in core settings. It lives in core settings
as it needs access to model layer. To avoid coupling with the frontend settings it accepts
a function which has to resolve to a routes hash
- Added note about settings validation side-effect. It mutates input!
- Added async check for currently loaded routes hash
- Extended frontend settings loader with async loader. The default behavior of the loader is
to load settings syncronously for reasons spelled in 0ac19dcf84
To avoid blocking the eventloop added async loading method
- Refactored frontend setting loader for reusability of settings file path
- Added integrity check test for routes.yaml file
closes#12060
- A 500 error what happening when invited user provided an email that is associated with an existing user
- Additional validation for existing email address was added to prevent invalid data hitting db constraint error
refs #11999
- The `routes_hash` setting will be used during the boot process to update the hash
of currently loaded routes.yaml file in case it's different from last restart
no issue
- The async/await syntax makes it easier to reason about the code. Because adding 'Location' header is in the works it's a prep-work in a sense
closes https://github.com/TryGhost/members.js/issues/94
- The member-api package was recently updated to work directly with models and needs explicit `withRelated` options to attack relations
- Without options, the endpoint was returning the default member data without subscriptions attached, which in Portal showed paid member as free
- Fix updates the middleware for updating member data to correctly pass the relations needed to populate the member
no issue
- By default for new sites, support address is set same as from address to `noreply` , with full email address using the domain for `@`
- For newsletter emails, the support address was missing the default site domain to be added to address if its `noreply`
- Fix updates the support address to use the same format as from address and add relevant domain for default case
no refs
- The `update` method in members-api package was edited to return Model object instead of JSON directly - TryGhost/Members@a28bcc5
- This unsubscribe handler was returning the raw member object returned from `update` method, which is now a model object and not able to access `member.email`
- Fix updates the unsubscribe request handler to return the member JSON again
no issue
- members who have trial subscriptions added directly via Stripe will have a status of `"trialed"` in their Ghost subscription
- the `paid: true` filter was not taking that into account meaning trial users were not receiving newsletters sent to paid members even though they have a "paid" subscription
no issue
- We used existing "from" address as sender for mails sent to new email address for verification, but that breaks the flow to update if the current "from" address has DMARC policy set.
- This updates the flow to always send the ownership verification email TO the new address and FROM the new address which both verifies the email deliverability for new address and ownership
no issue
- The newsletter emails are sent out with `from` address as sender
- The new `members_reply_address` setting is now used to set reply-to address for emails, which can be either newsletter or support address
no issue
- Member auth emails were previously using the `from` address as sender
- New `members_support_address` was introduced with default as original "from" address
- Auth emails use the new support address as sender
no issue
- Updated magic link generation and validation methods for email update API to handle new support address
- Updated importer to ignore the new support address as it can only be updated via verification
- Updated members service to listen on settings edit for new support/reply address fields as well
- Updated tests to include the new settings
no issue
- Added default settings for the two new setting fields - `members_support_address` and `members_reply_address`
- Added migrations for setting group for new email settings
- Migration sets current from address as new support address default
- Added migration to set new support address same as from address
- Updated tests for new settings
- `members_support_address` - How members can reach for help with their account, public setting
- `members_reply_address` - Where you receive responses to newsletters
closes#12167
- Tags API v2 was ignoring `count.posts` include parameter.
- Regression was introduced with a3f693b472
- Introduced regression tests across all Content API versions to avoid similar bug in the future
no issue
- Added default settings for the two new setting fields - `members_support_address` and `members_reply_address`
- Added migrations for setting group for new email settings
- Migration sets current from address as new support address default
- Added migration to set new support address same as from address
- Updated tests for new settings
- `members_support_address` - How members can reach for help with their account
- `members_reply_address` - Where you receive responses to newsletters
no issue
- When an import was done and there were no "global labels" present Ghost created generic `import-[data]` label which later helped to find a specific batch of imported data
- It did not make sense to create such generic label when user provided their own unique label
- The rules that work now are:
1. When there is no global provided Ghost generates on and removes it in case there are no imported records
2. When there is a unique new global label provided no new label is generated, but the label stays even if there are no imported records
no issue
- tested performance between knex raw, knex `count()` and bookshelf `count()` and found no difference over 1000 iterations of each (each ~19,500ms +- 500ms for 104k members locally)
- switched to using bookshelf as the code is the simplest
no issue
- extract filtering of an collection into a separate function
- use extracted function in `findAll()` so that it's query behaviour matches `findPage()`
no issue
- for large result sets or complex queries the count query itself can be quite time consuming
- when `limit: 'all'` is passed as an option there's no need to perform a separate count query because we can determine the pagination data from the final result set
- skipped count query when `limit: 'all'` option is present
- re-ordered comments to be closer to the code they reference (ie, why we have our own count query instead of Bookshelf's `.count()`
no issue
- This is handled on input sanitization layer with date
format check in JSON schema validation, so there's no need to do this
check again in the importer.
no issue
- When bulk insert fails there is no transactional logic to revert
related records form being inserted. Also, previously there were no
attempts to "retry" the insert.
- To avoid complex retry logic, an iterative one-by-one insert retry
approach was taken. If this becomes a bottleneck in the future, the
retry algorithm could be improved.
- To avoid a lot of code duplication refactored model's `bulkAdd` & `bulkDestroy`
methods to use 'bulk-operations' module.
- Updated error handling and logging for bulk delete operations. It's very
unlikely for error to happen here, but still need to make sure there is
a proper logging in place to trace back the failure.
- Added debug logs. This should improve debugging experience and
performance measurements.
- Added handling for unrecognized errors. Handling inspired by current unrecognized
error handling by ghost importer -10e5d5f3d4/core/server/data/importer/importers/data/base.js (L148-L154)
no issue
- When no members are succesfully imported through CSV import process the import label should not be created. Otherwise after multiple failed attempts to import there are orphaned labels in the system
no issue
- updates `@tryghost/kg-default-cards` which contains two fixes
- removes email-specific output being added to post html (had no visual impact due to use of conditional comments but keeps rendered html smaller+cleaner)
- adds a background-url style to the thumbnail container to give two options for styling
- updates member email template styling to hide the `<img>` element in bookmark cards and use a background image instead to get consistent rendering across email clients
no issue
- adds a `members:emailTemplate` config object
- `showSiteHeader` - defaults to `true`, shows the site title and icon in member emails
- `showPoweredBy` - defaults to `false`, adds a "Publish with Ghost" button to member email footer
- updates member newsletter email template with hideable site header and "powered by" badge
closes https://github.com/TryGhost/members.js/issues/87
- The `update` method in members-api package was edited to return Model object instead of JSON directly [here](a28bcc5b2a)
- This caused the update member API on member endpoint to return partial response only as most properties couldn't be fetched
- Fix updates the middleware to correctly call `toJSON` before formatting response
* 3.30.2:
Bumped @tryghost/members-api to 0.28.1 in lockfile
Bumbed @tryghost/members-api to 0.28.1
🐛 Fixed unable to delete member when stripe is connected
refs #12127
- Adds new `editSubscription` endpoint for members admin API which allows updating individual subscription for a member - `PUT /members/:id/subscriptions/:subscription_id/`
- `editSubscription` has same permissions as member's `edit` endpoint
- Currently allows toggling of cancellation at period end for an active subscription
no issue
- Similar handling to one introduced in 8418c829de
- Having granular tracking for failed to remove id's would make it possible to return more specific errors to the client
no issue
- When batch insert fails handling should be more granular and aim to retry and insert as many records from the batch as possible.
- Added retry logic for failed member's batch inserts. It's a sequential insert for each record in the batch. This implementation was chosen to keep it as simple as possible
- Added filtering of "toCreate" records when member fails to insert. We should not try inserting related members_labels/members_stripe_customers/members_stripe_customer_subscriptions records because they would definitely fail insertion without associated member record
no issue
- When stripe is disconnected and there are Stripe-connected records present in imported set they should not be processed and proper error should be thrown
closes https://github.com/TryGhost/Ghost/issues/12139
- once the email content has been rendered in the post serializer, perform some whole-content transformation of `figure` and `figcaption` to `div` using cheerio
- juiced will have already inlined the elements styles so there's no need to adjust the template's stylesheet
closes#12078
- Root cause was that pseudo class .kg-bookmark-author:after was not getting inlined to email newsletter or its preview.
- That is where the margin and bullet-point content are added between author and publisher.
- Dependency juice supports an option inlinePseudoElements which is false by default.
- Fix was to set inlinePseudoElements to true when serializing post email.
Co-authored-by: Jeremy Davidson <jeremy@crossingcontour.com>
- testmode allows us to test Ghost's behaviour with long running requests and jobs
- moved all the testmode logging to only happen if Ghost starts successfully to make it clearer what is happening
- added a (currently very dirty) bit of code to fetch the jobService and output the queue status giving us a similar
output for both open connections and jobs
- all of this allows for debugging exactly what Ghost is doing whilst it is processing a shutdown request
closes https://github.com/TryGhost/Ghost/issues/11536
- bumps `@tryghost/kg-default-cards`
- image and gallery cards now output `width/height` attributes on `img` elements with a max width of 600px
- uses resized images where possible to keep email weight down
- adds `height: auto` style to image card images so that the `height` attribute does not cause distortion at smaller screen widths
no-issue
* Added bulkAdd method to Member,Customer&Subscription model
This allows us to keep the db access in the model layer
* Updated @tryghost/members-api to 0.27.2
This includes fixes for rate-limiting of requests, and exposes necessary
Stripe methods for creating customers and complimentary subscriptions,
without affecting the database.
* Refactored importer to parallelise tasks where possible
By parallelising our tasks we are able to improve the speed at which the
entire import completes.
refs #12126
- Adds migration to add impersonation permission to administrators
- Adds default permission fixture to allow administrators to read member impersonation urls
- Allows administrators to create member impersonation magic links
closes https://github.com/TryGhost/Ghost/issues/12129
Migrations were failing when upgrading from a version that didn't include stripe member subscriptions table. The `up` migration was checking for the existence of a unique index that was added to the schema in 3.29 but it was using the wrong variable name which meant that it would never return true for an existing index. For most migrations this worked because the index existed but when through 2.34 the table was created from scratch and did included the index at that point.
- fixed variable name and re-ordered variable assignment for better code locality that would have made the typo more visible
no issue
- Additional validation is needed for imported data because in case of bulk insertions (through knex) we bypass model layer validation - this could lead to invalid data in the database, which would be hard to fix.
- Chose validation method we use for other endpoints - through JSON Schema. It proved to be very performant (200ms overhead for 50k records). When comparing it with iterative method (validating each record separately) this was adding about 17s of overhead.
- Refactored returned values from "sanitizeInput" method to encapsulate more logic so that the caller doesn't have to calculate amount of invalid records and deal with error types
- Whole sanitizeInput method could now be easily extracted into separate module (somewhere close to members importer)
- Bumped members-csv package. It is meant to handle empty string values - '' and null, which should allow validating member records more consistently!
- upon further testing, this started to not work again so I'm removing
this horrible hack
- it could be Bunyan playing around with us, and the server is actually
shutting down
- I have no idea why this is required and I really don't like it
- this change looks superfluous but I had problems with Ghost not
displaying shutting down messages properly and this is the only
thing that got it working
- the async-awaits were getting stuck in `await this._cleanup()`,
and then wouldn't enter into the `finally` block
- oddly, this was only reproducible when a job was running, it would
shut down normally otherwise
- if I find the solution in the mean time, I'll happily get rid of this
no issue
- Updated sanitization logic to be self contained and return sanitieze input along with error stats
- This should give a nice place for validations to fit in
no issue
- There were many failed import records due to rate-limit errors. With concurrency of 9 imports go through with 100% success
- Would need to verify these limits with live API to make the most of it
no issue
- we output the post excerpt in a hidden div in the email template so that email clients pick it up as the "preview" text when listing emails
- when no custom excerpt is provided the preview text is grabbed from post.excerpt which is the first 500 chars of the post.plaintext value
- post.plaintext formats links as "Link [http://url/]" which is unwanted in html email previews
- add a basic replacement to the post email serializer to remove any `[http://url/]` occurrences from the post excerpt before rendering the email content
refs https://github.com/TryGhost/Ghost/issues/11536
- Outlook supports `'` as a special char for apostrophes but not `&#apos;` which is what cheerio/juiced render
- adds a basic string placement to the email serializer to switch to the older style of special char
no issue
- moves the meat of `pendingEmailHandler()` code into a new function `sendEmailJob()` that is passed over to the new job service
- lets the server keep processing email generation and sending when it receives a shutdown request rather than halting processing mid-send and ending up in a partial state
no-issue
* Added stripeSubscriptions relation to member model
This allows us to fetch the subscriptions for a member via standard
model usage, e.g. `withRelated: ['stripeSubscriptions']` rather than
offloading to loops and `decorateWithSubscriptions` functions, this is
more performant and less non-standard than the existing method.
* Updated serialize methods to match existing format
The current usage of `decorateWithSubscriptions` and the usage of
members throughout the codebase has a subscriptions array on a stripe
object on the member, this ensures that when we serialize members to
JSON that we are using the same format.
There is definitely room to change this in future, but this is an
attempt to create as few breaking changes as possible.
* Installed @tryghost/members-api@0.26.0
This includes the required API changes so that everywhere can use
members-api directly rather than models and/or helper methods
no issue
- The code in controller was becoming hard to reason about.
- Having a single module shows exactly how many dependencies are there to do an import for single batch.
- Having a separate module would make it easier to extract into it's own package in Members monorepo
no issue
- Member's labels have to have sort_order assigned when added/edited. This was lacking from batched importer.
- Implementation is based on logic used in model's base - e484709e73/core/server/models/base/index.js (L81-L86)
- Bottom line - we need to manage shutting down gracefully when doing long-running tasks
- To achieve that, we're going to use job queues
In this commit:
- added new @tryghost/job-manager dependency
- added a minimal job service, that handles in passing things like logging and (maybe later) config
- job service is wired up to server shutdown, so that the queue finishes before the server exits
- also added a new job endpoint to testmode so that it's easy to test job behaviour without needing to do real work
- Using consistent patterns for shutdown and stop
- Make it clear what each method does
- Use async/await to make the code more readable and simple
- This lays groundwork for having more cleanup tasks in stop than just server.stop()
- deleted files under `core/server/lib/promise` and related test files
- added `@tryghost/promise` as a dependency
- fixed all local requires to point to the new package
closes#12118
- server.start was mistakenly removed in 71f02d25e9
- it is used for loading themes (and other things) and is critical
- added tests to prevent this regressing again in future
no issue
- New Member API batched import is meant to be a substitution to current import
with improved performance while keeping same behaviore. Current
import processes 1 record at a time using internal API calls and times
out consistently when large number of members has to be imported (~10k
records without Stripe).
- New import's aim is to improve performance and process >50K
records without timing out both with and without Stripe connected
members
- Batched import can be conceptually devided into 3 stages which have
their own ways to improve performance:
1. labels - can be at current performance as number of
labels is usually small, but could also be improved through batching
2. member records + member<->labels relations - these could
be performed as batched inserts into the database
3. Stripe connections - most challanging bottleneck to solve because
API request are slow by it's nature and have to deal with rate limits of
Stripe's API itself
- It's a heavy WIP, with lots of known pitfalls which are marked with
TODOs. Will be solved iteratively through time untill the method can be
declared stable
- The new batched import method will be hidden behind 'enableDeveloperExperiments' flag to
allow early testing
- A simple way to test behaviours without having to do complex interactions to e.g. generate errors or slow requests
- Makes it easier to test the new shutdown behaviour, among other things
- there can now be quite a big delay between SIGINT/TERM being received and shutdown finishing
- add an extra log message to acknowledge the SIGINT/TERM to facilitate debugging and just be clear
- stopppable is a dependency that handles closing connections properly, which server.close does not
- active connections are allowed to complete what they are doing
- idle connections are closed
- no new connections are allowed
- we call stoppable in stop() instead of server.close so that idle connections don't hold the server open
- calling await stop() from shutdown then ensures that we have a consistent experience of stop
- all together this allows ghost to shutdown gracefully when there are long-running requests
- @TODO: handle graceful shutdown of long-running processes
- @TODO: consider do we need to send 503s whilst the server is shutting down?
- changed method to logStopMessages, as we use start and stop, not start and shutdown
- changed logStopMesasges to output the "proper" messages and use this method consistently - the closing connections message isn't really useful
- changed uptime message to always be output cos I can't see a case where there isn't interesting/useful
- Connection handling is legacy code added in 1438278ce4
- Although we were tracking all connections in memory, we weren't actually closing any because stop isn't called
- This (and restart) were both added as part of the now long-deprecated system for using Ghost directly as an npm module
- If we want to close connections cleanly, we should use a tool to do this
- the announce functions exist for the purpose of communicating with Ghost CLI
- the functions were called announceServerStart and announceServerStopped, which implies they tell Ghost CLI when the server starts and stops
- however, the true intention / purpose of these functions is to:
- either tell Ghost CLI when Ghost has successfully booted (e.g. is ready to serve requests)
- or tell Ghost CLI when the server failed to boot, and report the error so that Ghost CLI can communicate it to the user
- therefore, I've refactored the old functions into 1 function to make it clearer they do the same job, but with 2 different states
- also added some tests :D
refs 022a433e56
- noticed I missed adding debug info in one place (not that it's used, just for consistency)
- also made the SIGINT/TERM code slightly more readable
no-issue
- switch from `membersService.api.members.list` to using bookshelf `Member.findPage()` with the `{paid: true}` filter to avoid per-member queries (N+1) to decorate members with subscriptions and a heavy post-fetch filter via `contentGating`
- add concurrency to the Mailgun API requests in `bulk-email` service to reduce overall time submitting API requests
- add debug statements with timing output for easier measurements
no issue
For large numbers of members we're unable to perform queries for free/paid members in a reasonable time without indexes. Bulk delete is also very slow when looping through bookshelf models and having bookshelf manage deletion of child records, adding `ON DELETE CASCADE` to the foreign key indexes moves child deletion to the DB level and allows for performant bulk delete queries.
- migrations only run on mysql because sqlite does not support altering tables
- adds foreign key indexes and constraints with 'ON DELETE CASCADE' for members_labels, members_stripe_customers, and members_stripe_customers_subscriptions
refs #12100
For performance reasons we want to add foreign key and unique constraints
to the members_stripe_* tables so we can utilised cascading deletes and
joins across the tables when querying.
In order to do this we must first ensure that:
- There are no duplicate entries in the `subscription_id` or `customer_id` columns
- There are no orphaned rows in the subscription or customers tables
If the first is not true, the unique constraint will fail, and if the second is not true,
the foreign key constraint will fail.
As we are only adding the indexes to existing MySQL databases at this point, the
cleanup migrations will also only be done for existing MySQL databases too.
The migrations for removing orphaned rows splits the deletion into a `SELECT`
followed by a `WHERE IN` to avoid the database "optimising" the query into a
`JOIN` which ends up taking much longer due to the lack of indexes.
no-issue
We are in the process of creating migrations to add foreign key constraints
and cascading deletes to the members_stripe_* tables to make listing members
and deleting members faster. As well as the migrations we need to update the
database schema so that new installations have the correct indexes and constraints.
Co-authored-by: Kevin Ansfield <kevin@lookingsideways.co.uk>
refs 173e3292fa
- The bug was initially introduced in referenced commit. When request is done with `api_key` context, there should always be an `integration` object associated with it - 71c17539d8/core/server/services/permissions/parse-context.js (L36) . An `id` from `context.integration` not `context.api_key` has to be assigned to newly created webhook!
- The webhooks API is about to be declared stable in upcoming release, so no migration will be done
closes#11907
The image in the bookmark card was being shown out of the bounds of
the card because of a general style `height: auto !important`.
I added a new `max-height` property to the image to avoid exceeding
parent height.
no issue
- Webhooks API has been stabilized with latest changes and there are no breaking changes planned for v3. The change has strictly "informative" purpose
- Changed variable naming from "whitelisted" to "allowlisted" to follow updated naming convention (refs. https://mysqlhighavailability.com/mysql-terminology-updates/)
refs #12064
- `critical` is meant to be something unpredictable like internal error, something worthy attention, as described in Ignition -3439456d94/README.md (list-of-errors)
- This error level was introduced with - this PR https://github.com/TryGhost/Ghost/pull/9426, but there is no context provided why this specific value was used. Assuming it's an outdated value as 'not found' is nowhere to be treated in any special way
no issue
- `mailgun()` expects the `host` option not to include a port but `url.host` will include the port, we instead want to use `url.hostname` which skips the port
no issue
- bookshelf adds `DISTINCT` to any relation query that does not have an explicit `columns` statement
- when measuring the impact of `DISTINCT` on the eager-loading association query when listing members using `{withRelated: 'labels'}`, it can be 2x slower with no index on the sort_order column or 4x slower with an index on sort_order
no issue
- When processing entries with new labels in parallel Bookshelf relations is trying to create them which caused unique key constraints to fail. To avoid the failure, all labels should be pre-created before proceeding with creating members
- Member limit code was duplicated in 2 places unnecessarily
- Also used member api code that fetched members and subscriptions fully hyrated when we only need a count
- Using a raw query significantly improves performance here
no-issue
This updates the Admin API Member resource to *not* cancel subscriptions
by default, and adds a `cancel` option. This can be used over HTTP by
including a `cancel=true` query parameter.
- mailgun has a testmode flag we can use to get email to be accepted but not delivered
- this is useful for developers testing general bulk email code - not for users - so it is only available via config
no-issue
Up until now we have left orphaned rows in members_stripe_* tables when
a member is deleted, this updates the destroy method so that we cascade
and remove any MemberStripeCustomer and StripeCustomerSubscription
models related to the Member.
This also adds regression tests for the new functionality as well as to
confirm the existing functionality of cascading to the members_labels
join table
This adds the relations of Subscription->Customer & Customer->Member
refs https://github.com/TryGhost/members.js/issues/72
- Portal is using using publication logo from settings for signup/signin pages
- Instead, we are switching to using publication icon from settings, which also needs to be passed in site data API
no issue
Having all members created during an import labelled with a specific "import label" is useful for later operations such as bulk delete/edit or simply recording how and when a member was created.
- automatically create a label with the date/time the members CSV import occurred and assign it to all imported members
- return the import label data in the API response so that clients can react accordingly such as automatically filtering the members list by the label once an import finishes
refs #12074
Since we've split members settings into multiple keys the
reconfiguration of the members-api has been happening in quick
succession as the stripe_connect_* settings are all set at once.
This debounces the call to reconfigure the members-api so that we only
need to instantiate it once.
no issue
- By default, GhostMailer throws EmailError with statusCode as `500` for any failure in sending mail
- In case of failure due to `RecipientError`, status code as now correctly sent as `400` as its a bad request and not an error we can't handle.
closes#12049
Stripe plans used to default to 0, and our new validation of plan
amounts were causing issues when importing from an older version of
Ghost, this updates the validation to be skipped when importing.
- Added regression test for importing plans
refs https://github.com/TryGhost/Ghost/issues/11971
- Added statusCode from bulk email provider to API response
- Updated error messages for different bulk email(mailgun) failure states
- Added `context` to preview mail API error message with mail provider's error message
closes#12033
- Added webhooks schemas and definitions.
- Added validation checking if integration_id is present when using session auth. This is needed to prevent orphan webhooks.
- Integrated webhook schemas into frame's validation layer.
- Added isLowerCase ajv keyword support. This is needed to be able to do isLowerCase validation using JSON Schema for webhooks.
closes#12001
* Moved settings validation to the model
This moves the settings validation out of the validation file and into
the model, as it is _only_ used there.
It also sets us up in the future for custom validators on individual
settings.
* Improved validation of stripe_plans setting
- Checks `interval` is a valid string
- Checks `name` & `currency` are strings
* Moved stripe key validation into model
The stripe key settings are all nullable and the regex validation fails
when the input is `null`. Rather than reworking the entirety of how we
validate with default-settings validation objects, this moves the
validation into methods on the Settings model.
* Added tests for new setting validations
Adds tests for both valid and invalid settings, as well as helpers
making future tests easier and less repetitive
closes#12015
refs 95880dddeb
- The bug was caused by falsy plaintext field assignment to empty string `''` when the html content was `null`. Because of the `setEmptyValuesToNull` function (referenced commit), there is no sense to assign empty string value to plaintext property, because it would still end up being `null`
- The `''` -> `null` conversion was confusing the model layer to think that some fields were changed, where in reality none did. This in turn lead to a bug with falsy cache invalidation
closes#12016
- The change detection didn't work when editing post_meta fileds because we only check current model's `_changed` fields when performing `wasChanged()` check
- A solution was adding change tracking of post_meta relation to currently edited post model and overloading `wasChanged` method to check these fields as well
refs #12043
- On updating From-address from Members settings in Labs, we send a confirmation email to the updated address with magic link for verification
- Previously, no explicit sender email was being set for this so fallback config address was used
- This updates the sender address to use the current from address for "from-address" update emails
no-issue
- Added breaking test for webhook url including subdirectory
- Previously the webhook handler URL was generated incorrectly when
running Ghost on a subdirectory, appending the path to the root of the
host, this fix ensures that the subdirectory is included before the
path.
no-issue
This version of members-api includes changes to how webhooks are
managed, previously they would be deleted and recreated on every boot of
Ghost. Now they are created and the secret is persisted, on boot the
webhook is updated to the most current url and events. If the api
version is wrong or the update fails, the webhook is deleted and
recreated and the settings updated.
- Installed @tryghost/members-api@0.24.0
- Updated config to work with 0.24.0
no-issue
They will be used to store webhook information so that we can persist it between
boots and simplify the creation process of webhooks in members
closes https://github.com/TryGhost/Ghost/issues/12026
- 3 new portal settings were added in `3.23` in default settings - `portal_button_style`, `portal_button_icon`, `portal_button_signup_text`
- New settings default to group `core` on migrating from pre 3.22 versions due to missing columns in DB
- Migration here updates the new settings to correct group
closes#11994
- Adds support for ordering based on slug filter that contains a slug-is-in filter. It is applied only to Content API's resources - post, page, tag, author. The order is applied in the same order in which slugs appear in the filter.
- For, example providing following query parameter filter for any of the above resources: `?filter=slug:[kitchen-sink,bacon,chorizo]`, would filter them by these slugs and order in the same way defined in the filter
- Can be used in handlebars templates in following way: `{{#get "tags" filter="slug:[slugs,of,the,tags,in,order]"}}`
- The property conteining this new order is assigned to `autoOrder` instead of `rawOrder` intentionally. This explicit asstignment would allow distinguishing where the 'orderRaw' comes from the model or the API layer. Apart from adding necessary context this separation makes it easier to refactor separately model layer and API specific ordering in the future
- This commit also fixes default filtering for `author` resource in Content API. The serializer was never used before as it was missing from `serializers/index.js` module.
no issue
- Changes introduced to both API v3 and v2
- Makes sure to use the same integration_id as authenticated integration for the webhook's data.
- Makde it is impossible to create orphaned webhooks using token authentication
- Allowed only parent integration to edit it's children webhooks. Throwing permission error otherwise
no-issue
There is concern that the settings cache can return `null` for values
which it cannot parse correctly, this just ensures that we always have
an array where we expect one
no issue
- Existing and new sites should have portal button hidden by default and need explicit switch on from Admin.
- Default value for new sites has been updated, but existing sites still have old default and portal button set as true
- Migration here resets portal button setting to false for all existing sites, and they'll need to be explicitly turn it on from admin again
no issue
- Adds new portal button settings to members site data for portal script
- Updates settings input/output serializers to handle portal icon image url
no issue
- Adds new portal settings - `portal_button_style`, `portal_button_icon` and `portal_button_signup_text`
- New settings allows customization of portal button
- Updates tests to include new settings
* tag '3.22.2':
v3.22.2
Updated Ghost-Admin to v3.22.2
Emitted all settings events on reinit of cache (#12012)
🐛 Updated access to be true by default in v3 API
Hardened members subscription migration against missing data (#12009)
closes#12003
There are a few parts of Ghost that rely on the settings events being
emitted anytime a setting is changed, so that the data is kept in sync.
When a setting is renamed in a migration essentially what happens is
that the settings value is changed from a default value to its actual
value, but this does no emit an event.
Anything that is initialised before migrations have run that relies on
the events to keep it up to date will have stale data - e.g. the themes
i18n service.
This change ensures that when we `reinit` after migrations have been
run, we emit events for every setting to tell the rest of Ghost that it
has changed.
closes#11990
- access should be a members feature, but it was already accidentally exposed to the theme layer
- it has now been added to the API even if members is disabled
- access defaults to true, unless members is enabled
- when members is enabled, access is set to the currently logged in members' access
closes#11993
We had some issues with some databases being in an unexpected state this
check for each property before using it add uses defaults when it is
missing.
no issue
- when searching for paid/free members, the `members_stripe_customers`
table would be joined into the query on `members`
- this table also has a `name` and `email` field, so both MySQL and
SQLite would complain about ambiguous fields in the query
- the result of this would be a 500 error thrown inside Ghost, and no
useful response to the user
- this commit explicitly chooses the `members` table to check against,
and also adds a test for this
no issue
- The membersjs default trigger button was set to be visible by default, even behind dev flag
- This changes the default visibility to false so it needs to be switched on from Admin to be visible on screen
closes https://github.com/TryGhost/Ghost/issues/11944
- updates `@tryghost/image-transform` to version that exposes `canTransformFiles()` which checks for `sharp` availibility
- updates `@tryghost/kg-default-cards` to version that accepts a `canTransformImage()` method as an option
- updates our `mobiledoc` lib to pass a `canTransformImage()` function that returns false if sharp is unavailable, the image extension is not supported, or the storage engine in use does not support image transforms
- updates `populateImageSizes` to fetch image sizes when transforms are unavailable as the render/not-render is now handled in the renderer and we don't need to worry about adding size information to the mobiledoc source
refs https://github.com/TryGhost/Ghost/issues/10318
- re-initialize settings cache after migrations by shutting down to clean up event listeners then and calling `init` again
- important to ensure `db.ready` event is not emitted until settings have finished re-initializing to avoid problems with background processes using the db connection which is disconnected/re-connected or being kicked off with out-of-date settings
refs 05f6faf846
- The logic in "Stripe" error detection depended on error's message string matching, which is not a good practice in general. Had fixed it do to exact match on "context" of the error we throw internally and left more extensive comment about why things are implemented the way they are.
no issue
Output serializer's url util was expecting `og_image` and `twitter_image` to be top-level attributes in the `attrs` object but they are actually nested under `posts_meta`.
- updated the code to use lodash's `get/set()` so that we can work with paths for easier handling of nested objects
- fixed unit tests where the mocked data under test did not match real-world data
closes#11932
- as per the issue, there is no need to output a <link> tag if the favicon is the default /favicon.ico, as all browsers automatically check for this
- instead the favicon <link> is only output if a custom favicon has been set, telling the browser to look somewhere different to its default location
- some of the tests expected 3 links in the html head, but through the favicon change there are often only 2.
Co-authored-by: RenCloud <rencloud@pop-os.localdomain>
closes#11574
- the current implementation of the access property has it frontend only, and wired up only in one place
- this leaves it only available in a handful of places, e.g. can't use it in a post loop or get helper
- the current implementation also fails logically if the html content of the post is blank
This fix moves the behaviour to the API
- this ensures the field is always available no matter what context you are in
- it also updates the logic to use the same membersHasAccess logic as is used to gate the post, so it's always correct
TODO: should reconsider the location of this code
refs https://github.com/TryGhost/Ghost/issues/10318
- Skipped `members_subscription_settings` and `stripe_connect_integration` from settings import
- Fixed import data field filtering for core and theme groups
refs https://github.com/TryGhost/Ghost/issues/10318
- Updates `boolean` serialization in v2/canary serializers to apply only for `boolean` type settings
- Updates `boolean` transformation in model layer `format`/`parse` to check on `boolean` type setting
- Removes error thrown on Read-only setting for settings edit endpoint
- Updates v2/canary input serializers to remove any Read-only settings (using RO flag) to avoid edits
- Added type/group mappings in the importer when pre-migration settings table import data is present
- Updates tests
refs https://github.com/TryGhost/Ghost/issues/10318
- Updates default settings to contain correct type and validation for each setting
- Updates `populateDefaults` to correctly insert type value for new settings
- Updates settings schema to allow only select types - `array`, `number`, `boolean`, `string`
- `object` is a temporary type allowed till we get rid of all JSON object settings
refs #10318
Because settings are not populated with the correct group and flags, we
must _always_ set these. Then we can check to see if there are values
which need migrating, and if not, can safely exit and leave the values
as default.
refs #10318
If migrating from a previous version that does not include the setting
being migrated from we can safely not update the new setting, and just
rely on its default value being present. When rolling back we can use
defaults if the new setting does not exist.
refs https://github.com/TryGhost/Ghost/issues/10318
- Updates the type field for a setting to specify its type explicitly
- Type can be one of (string, number, boolean, array)
- `object` is allowed a temp type till all the object settings are removed
- existing `type` value for setting was misleading and referring to `group`
- we moved the current `type` value for a setting to `group` here - 4a9e57c170
* Updated members default settings
ref #10318
This pulls out the members_subscription_settings & stripe_connect_intgration settings into separate keys
* Updated usage of members_from_address
* Updated stripe_connect usage
* Updated members config to use new settings
* Updated members middleware to use isStripeConnected
* Updated members service to reload correctly
We reload the members-api instance when the related settings change, so
this makes sure we're listening to the correct settings changes
* Updated ghost_head helper to use new settings
* Updated theme middleware to use new settings
* Renamed members_allow_signup -> members_allow_free_signup
* Fixed tests after settings refactor
* Removed from direct key settings key
* Fixed regression tests for settings api
refs #10318
As populateDefaults is run _before_ migrations, the new settings will
already be inserted in the database, so we just need to update their
values and then delete the old settings.
refs https://github.com/TryGhost/Ghost/issues/10318
`Settings.populateDefaults()` is run before migrations during Ghost's startup. This can cause problems when new settings table columns are added (and populated in `default-settings.json`) because `populateDefaults()` was using the model layer which assumes that those columns are available, resulting in `ER_BAD_FIELD_ERROR: Unknown column` type errors.
- query the database for the available `settings` table columns
- switch to using raw knex queries without Bookshelf for insertions so that we're in control of the columns that are added
- use `_.pick` to skip any properties in `default-settings.json` that do not match to an available column - those columns will be added and populated by later migrations
- moving away from using the model to insert settings has the side-effect of not emitting `settings.added/edited` and `settings.x.added/edited` events, this should be fine because `populateDefaults()` is called before anything else is set up and listening
- added a call to `populateDefaults()` in our knex-migrator "before migration" hook so that we have consistent db state across both startup initialised migrations and manually triggered knex migrations
no issue
- Import error grouping wasn't working correctly when error didn't have defined context property
- The copy will be refined in final design review
no-issue
These utils are the first steps toward getting the models out of our
migrations! The utils here interact directly with the database and where
possible don't reinvent the wheel, by using smaller building blocks to
build more comples ones.
refs TryGhost/Ghost#10318
refs 8fc526ff6
- This is symetric change to one done for v3 API (commited as 8fc526ff6)
- Added 'core' filtering for v2 API controller
refs https://github.com/TryGhost/Ghost/issues/10318
- `shared_views` is always an array (Ghost-Admin has handling to revert it to an array if it's anything else) but it's default value was set to an empty object
refs https://github.com/TryGhost/Ghost/issues/10318
- `members_public_key` and `members_private_key` are now correctly grouped under `core` which is already filtered out so we don't need the extra filter to exclude them
refs https://github.com/TryGhost/Ghost/issues/10318
refs 1dc0405803
- Adds 1:1 mapping for filtering options to renamed settings "type" to "gorup"
- Ignores the name changes and any old types
- Detailsed type -> group mappings can be checked in the refereneced migration commit
refs https://github.com/TryGhost/Ghost/issues/10318
- maps old `settings.type` values to new `settings.type/group` values
- uses an explicit map so that we don't lose information and can safely roll back even though we're modifying `settings.type` too
- updates `settings.type` values too to keep code working while we switch to using `settings.group`
- sets the `settings.group` value for all settings which are keeping the same group as their current type
- adapts `settings.type` validations to match new groups
- adds flags to specific settings, both in the migration for existing settings records and in default-settings.json for new settings records
refs https://github.com/TryGhost/Ghost/issues/10318
- `group`
- to replace the `type` column, provides a more descriptive name for the columns use
- for existing sites it will be populated by migrating data from the `type` column in a later migration
- for new sites a minimal update has been added to `parseDefaultSettings()` to populate the `group` field when settings are created during startup - fixes the NOT NULL constraint on `settings.group`
- `flags`
- signifies special handling that is different to other settings in a group
- eg, `PUBLIC,RO` would indicate that the setting is available via unauthenticated endpoints and is read-only
refs https://github.com/TryGhost/Ghost/issues/10318
- There was a copy/paste error and we didn't have a test to pick it up. Will follow up with a regression test to make sure it doesn't happen again
refs #10318
refs 2614565d5a
- Adds importer mapping for fields as in referenced migration
- The intention is to allow exports from v2/v3 to still be compatilbe with current version
- Adds a mapper for deprecated fields and imports them with new settings "keys"
refs #10318
refs 2614565d5a
- Renamed ghost_head/ghost_foot in settings to match the new names
introduced in migrations
- Above change lead to reshufling in the mappings in input/output
serializers
- Makes sure change is compatible with v2 API
refs https://github.com/TryGhost/Ghost/issues/10318
refs 2614565d5a
- Renames to match referenced migration renames
- Fixed API responses so they are consistent with newly renamed fields
- Not returning lang and timezone keys from settings in API v2 ther rest should be returned in API v3/canary
refs https://github.com/TryGhost/Ghost/issues/10318
- precursor to migrating from `settings.type` to `settings.group`
- renames `blog` type to `site`
- renames `bulk_email` type to `email`
- moves settings out of `site` (previously `blog`) into more appropriate groups such as `core` or individual feature groups
no issue
- Previous migration in commit - 2614565d5a - was using incorrect json structure for brand -> primary_color
- Fixes the up/down migrations to work on correct existing `brand: primary_color` structure
refs https://github.com/TryGhost/Ghost/issues/10318
- Adds following renames to settings table keys:
'default_locale' -> 'lang'
'active_timezone' -> 'timezone'
'ghost_head' -> 'codeinjection_head'
'ghost_foot' -> 'codeinjection_foot'
'brand.publicationColor' -> 'accent_color'
- The renames are done to match revised naming conventions and naming
exposed through APIs
- Supersedes this revert - 1eeb5a60b8 and #11946
no issue
- when using subdirectories, images can be stored in the database both with and without the subdirectory prefix. We weren't taking that into account and so images without the subdirectory were not having the `/content/images/` prefix removed when passed to the storage adapter resulting in the storage adapter not finding the image
refs https://github.com/TryGhost/Ghost/issues/10318
- Adds following renames to settings table keys:
'default_locale' -> 'lang'
'active_timezone' -> 'timezone'
'ghost_head' -> 'codeinjection_head'
'ghost_foot' -> 'codeinjection_foot'
'brand.publicationColor' -> 'accent_color'
- The renames are done to match revised naming conventions and naming
exposed through APIs
refs #10318
refs https://github.com/TryGhost/Ghost/pull/11942
- Removes force_i18n, permalinks, and members_session_secret usage from the codebase
- We deprecated these flasgs and have not used since Ghost v2. It's good time to remove them before we introduce bigger changes to how `settings` table opeartes.
- Fixed importer test. The test was meant to check if string values were converted properly, the check agains boolean didn't make much sense in this context, so removed it.
- Following this change are going to come ralated migrations to fix existing data (see ref)
no issue
- reverts commit 87c31444fd but with modifications to settings naming
- Adds new settings for members modal customization to default settings
- `portal_button` controls the visibility of beacon in members modal
- `portal_name` controls the visibility of name field in signup
- `portal_plans` controls the visibility of plans allowed for member to signup with
- Adds stripe connect check to determine if stripe is setup or not
- Adds the 3 new settings to members site data
- Updates to snake case naming for members site API data
no issue
- Moves out CSV parsing and serialization related code into separate package as a part of push to modularize Ghost repo.
- Next up is to remove `csv-parser` dependency from this new package
refs 7904c303a7
- Model's validation errors are sometimes returned as an array of errors, in those cases they should be flattened to before returning to the client. This way there is more insight into what went wrong during the import.
no-issue
pr: https://github.com/TryGhost/Ghost/pull/11930
- Upgraded @tryghost/members-api to 0.23.0
This version includes a new method hasActiveStripeSubscriptions
- Added /admin/members/hasActiveStripeSubscriptions
This can be used to determine whether or not we should allow removing
the stripe keys.
- Added /admin/settings/stripe/connect
This can be used to delete a Stripe Connect integration, provided
there are not active subscriptions
no issue
- fixed incorrect method name when calling
- fixed problem with setting post.mobiledoc to a promise rather than waiting for the size population to finish and setting it to a mobiledoc string
no issue
- `?formats` is useful when you want to get `html` back from the Admin API when creating/editing posts, otherwise you're forced to make a following `GET` request
no issue
- Adds new settings for members modal customization to default settings
- `membersjs_show_beacon` controls the visibility of beacon in members modal
- `membersjs_show_signup_name` controls the visibility of name field in signup
- `membersjs_allowed_plans` controls the visibility of plans allowed for member to signup with
- Adds stripe connect check to determine if stripe is setup or not
- Adds the 3 new settings to members site data
no issue
- adds `populateImageSizes()` to our mobiledoc lib module
- uses `image-size` lib to speed up reading of image dimensions
- for local images, use storage adapter with same guards as used by `handle-image-sizes` middleware so that we don't insert srcsets for images that aren't transformable
- for unsplash images, remove any width and crop params from the url so it points to the full-size image
- use `populateImageSizes(mobiledoc)` to modify post model's mobiledoc when re-rendering
no issue
- known failing case was Medium. Some *.medium.com articles triggered redirects but without cookies being stored/sent across the redirect requests it would cause an infinite redirect loop and we'd abort after 10 redirects
- use `got`'s [cookie support](https://github.com/sindresorhus/got/tree/v9.6.0#cookies) via `tough-cookie` so that we can create bookmark cards for medium.com and other sites with similar problems
- There were various cases where it was possible to trigger a private site to display a 404 instead of redirecting to /private/
- Private mode was also not always displaying the correct robots.txt
- This PR includes tests for all cases in test/frontend-acceptance/default_routes_spec.js & where possible the unit tests have also been updated for completeness
- Fixing the 404 issues required
- Better handling of paths using req.path instead of req.url in filterPrivateRoutes
- Additional error handling, to cover the case that a tag/author RSS feed does not exist
- Fixing the robots.txt required the order of middleware to be changed, so that private blogging gets a chance to render first
- NOTE private blogging is the only app with a setupMiddleware function so nothing else is affected
no-issue
This issue only occurs when using custom redirects with a subdirectory
setup, and the path to be redirected from is expressed as a regex, and
the url that is being redirected to is not an external url.
The issue has a few components:
- Redirect paths as a regex generally use the ^ to ensure that they
match the beginning of the path.
- The path that the regex is matched against conditionally excludes the
subdirectory, specifically, the subdirectory is excluded for external
urls
These combined means you end up with a regex like /^\/custom-redirect/
and a path like /subdir/custom-redirect, these will not match/replace
correctly, and you'll end in an infinite redirect loop.
The fix here is to *always* remove the subdirectory when testing regex's
and then conditionally adding it back *only* for the redirect, and only
if it is an internal redirect
no issue
- localhost check was causing an immediate "no provider" error when fetching oembed for a localhost url and bypassing the bookmark fallback even when the configured site lives on localhost
- allow `localhost:port` through in oembed endpoints when it matches the configured url
no issue
- pass `srcsets` value through to mobiledoc renderer
- it will stop adding `srcset` attributes to images when explicitly set to `false`
- adds `reload()` method to `mobiledocLib` so that memoized instances of the renderer can be reset during tests that change config values
closes#11917
- Pass text-only version to mailgun as `text` not `plaintext`
- This ensures we send a text-only version of the email, and this in turn should help to improve spam scores
refs https://github.com/TryGhost/Ghost/issues/11414
- Importing data currently overwrites the existing "from address" with new value
- "from address" needs to go through email validation flow before update which was bypassed
- Updates importer to not allow overwrite for "from address" and use existing
- Adds test for "from address" overwrite
- We have many customers asking for INR as there are special rules in Stripe for this currency
- As well as a desire for local-selling
- Meaning it's not valid to use e.g. USD instead
no-issue
This was initially missed as local settings always had the original
stripeDirect keys, this ensures that regardless of Connect vs Direct vs
Both vs Neither that the config is correct.
Also ensures that the Members API instance is reloaded when the Stripe
Connect settings are changed.
no issue
- there are various situations where we adapt/fix/improve our mobiledoc->html output over time but we didn't have a way of updating old content without manually editing the mobiledoc and saving, or running an expensive migration to re-render all old content
- this adds a `?force_rerender=true` query param to the `PUT .../admin/posts/:id/` endpoint that allows the `html` field to be re-generated without modifying the `mobiledoc` field contents
no issue
- The intention is to move away from using file type names in URLs. This endpoint is meant to correspond to current `POST /members/csv` endpoint, that is planned to be renamed into `/members/upload`. And the `GET /members/csv` to be renamed to `/members/download` respectively.
no issue
- NQL does not support the relationship setup that members->stripe customer<->stripe subscriptions uses so it wasn't possible to use the `filter` param to query against having an active subscription
- adds `customQuery` bookshelf plugin that allows customisation of SQL query used in `findPage` method by individual models
- use `customQuery` in Member model to set up joins and conditionals to select free/paid members when `options.paid` is present
- allow `?paid` param through API and permitted options for member model
no issue
- This new format allows to return additional metadata with failed import records. The data for invalid records is returned in following format:
```
{
count: {count_of_invalid_records},
errors: [{
message: "Members not imported. Members with duplicate Stripe customer ids are not allowed." // message field of the error
context: "Attempting to import members with duplicate Stripe customer ids." // context field of the error
help: "Remove duplicate Stripe customer ids from the import file, and re-run the import." // help field of the error
count: 2 // count of this specific error
}]
};
- Errors are grouped by their context fields because message fields sometimes can contain unique information like Stripe customer id, which would produce too many errors in case of bigger datasets.
no issue
- This endpoint is meant to be used for validation of imported members
- Main function at the moment is to validate if stripe_customer_id present in the dataset exists in connected Stripe account
no issue
- Ghost-Admin redirects all paths to `/ghost/settings...` as `/ghost/#/settings/...`, this updates the admin redirect on successful magic link validation to directly use the latter to avoid extra redirect
no issue
- adds a set of hardcoded "content image sizes" to the base config
- adjusts `handle-image-sizes` middleware to always allow the hardcoded content image sizes to be genreated
- updates `@tryghost/kg-card-factory` to allow passthrough of options to card renderers
- updates `@tryghost/kg-default-cards` to add `srcset` output for image and gallery cards
closes#11008
- Updated @nexes/nql to 0.4.0
This version exports the mapKeyValues utility function
- Replaced nql-map-key-values with @nexes/nql util fn
Usage was found using `rg nql-map-key-values` and replaced globally.
- Deleted nql-map-key-values module in shared
Now that this module isn't referenced anywhere else, we can remove it,
relying solely on the util exported by @nexes/nql
refs https://github.com/TryGhost/members.js/issues/43
- Adds new `isStripeConfigured` flag to public members site data which denotes if stripe setup is completed
- Helps clients like members.js/themes to configure payment behavior based on this flag
no-issue
The service at stripe.ghost.org must know which client_secret to use,
either the test, or live one. By encoding a JSON object as the state we
are able to pass data through the flow to inform this decision at the
end.
Note, that we still keep a random value in the state to protect against
CSRF attacks.
no-issue
We've added a "mode" query param to the members_stripe_connect api auth
method, allowing the client to easily switch between live and test mode.
no-issue
This adds the ability to pass a `mode` param of 'test' to the members stripe
connect service, which will ensure we use a testmode client_id for the
Stripe Connect OAuth flow.
This will allow users to connect their account in testmode.
no issue
- When imported member contains stripe_customer_id data but there is no Stripe configured on the Ghost instance such import should faiil. The logic is consistent with one where import fails after not being able to find customer in linked Stripe account
- Fixed import stats to show import failures instead of "duplicate" when the validation error is of "Stripe" origin
no issue
- The aim was to extract format-csv to become an external dependency. After some analysis found out that native papaparse method `unparse` was achieving the same results with a lot better test coverage and stability. Because papaparse will become Ghost's default csv processor in near future decided to integrate papaparse instead of extracting module for the format-csv module's code, which would become redundant soon anyways.
- For reference papaparse will substitute current csv-parser lib because it's better performance and maturity.
- Performance comparison can be checked here - https://github.com/Keyang/csvbench#result . At the time of writing papaparse is rougly 40% faster than csv-parser
refs https://github.com/TryGhost/Ghost/pull/11807
- this was originally reported in the PR above, but we could not
reproduce it on master
- presumably the user had the latest version of moment installed for
other purposes and so they were seeing the issue
- between moment 2.24.0 and 2.26.0, something must have changed which
stopped the previous functionality working
refs https://github.com/TryGhost/Ghost/issues/11414
Confirms if the fromAddress for sending member emails is valid and accessible using magic link flow, allowing owners to update full from address including domain change.
- Extends member service to handle magic link generation and validation for email update
- Updates existing setting endpoint to not directly update from address
- Adds new endpoint to send magic link to new address
- Adds new endpoint for validating the magic link when clicked and update the new email for from address
- Adds new email template for from address update email
refs https://github.com/TryGhost/Ghost/issues/11414
We want to allow adding custom domains to member's from address as long as the new email is verified using the magic link flow and not saved directly to DB. Since we currently don't store domain as part of fromAddress, this PR -
- adds migration to update all existing fromAddress by appending site domain
closes https://github.com/TryGhost/Ghost/issues/11768
- Wraps from parameter in double quotes so mail clients can read it whole
- Escapes double quotes in site title to avoid clash with wrapping
refs https://github.com/TryGhost/Ghost/issues/11212
- if a bookmark card fetch is performed (either directly or from fallback) and the page does not have an extractable title, return a more specific error message than "No provider found for supplied URL."
no issue
- When the customer cannot be imported because they are missing from linked Stripe account or the linked account is incorrect one, these new messages should provide a better clue about what has caused the error and how to act on it.
no issue
- There is a need to be able to label certain import group of members
with custom labels. This will allow to distinguish/filter these newly
imported members.
- Allowed `POST /members/csv/` endpoint to accept `labels`
field parameter which assigns labels to every member from imported csv.
no issue
- Similarly to other additive api methods (e.g. members.add) returned more specific ValidationError with contex filled in with the reason why adding did not succed.
- This change is needed for more graceful label handling when adding new members through import
no issue
- Adding labels doesn't cause any content to invalidate, similarly to adding members. Unlike it's caunterpart - tags, there is no dependent "frontend" content that would become invalid
no-issue
This was because the API was recieving a default `null` value for the
stripe_connect_integration_token setting.
This also improves the logging for this error.
no issue
- added an `externalRequest` lib
- uses same underlying `got` module as our `request` lib
- uses `got`'s `beforeRequest` and `beforeRedirect` hooks to perform it's own dns resolution for each url that's encountered and aborts with an error if it resolves to a private IP address block
- includes a bypass for Ghost's configured url so that requests to it's own hostname+port are not blocked
- updated v2 and canary oembed controllers to use the `externalRequest` lib
no-issue
If the stripeDirect config value is NOT set / false, then connect token
should be used over the API keys if both are present, or else use
whichever is present. If the stripeDirect config value is set /
true,then API keys should be used and stripe connect token should be
completely ignored.
The reason for this is that we want to use Stripe Connect as the primary
auth method going forward, but we want to keep the direct version
availiable should the Ghost Foundation ever dismantle. This will allow
people to use all the same features with no dependency on an external
service.
no-issue
In order to issue a redirect we need access to the "raw" req/res
objects, which is why we must return the function which gets access to
them.
The members service is used to create the auth url and to update the
users session.
no-issue
Uses the members service to parse a stripe_connect_integration_token
setting and set the stripe_connect_integration based on that.
This change includes ignoring the stripe_connect_integration{,_token}
settings, as the token is never saved, and the integration can only be
set by using the token.
no-issue
This module handles the creation of a url used for authorization of
Stripe Connect, and also the parsing of the data eventually received
from the authorization flow.
no-issue
There was some unused code here, the variable was never used, also we
were looping and collecting a list of errors, but only every using the
first one, so switched to the `find` method which stops iteration after
an element has matched.
closes https://github.com/TryGhost/Ghost/issues/11841
- Migration adds mapping between permissions and roles for email_preview send test mail
- Only owner previously had correct permission to send test emails
- Fixture existed to allow Admin/Editor/Integrations to send test mails but had missing migration
- Adds tests for roles to send test email
refs https://github.com/TryGhost/members.js/issues/30
- Added new `updateEmail` type for sending email address update confirmation mail to member
- The link in email updates member's email address
no issue
- node was complaining the package was missing after the sanitize-html update [1]
- the update removed an old version of lodash which supported the dot importing
method
- our code relied on this subdependency to work, but the structure has since
been updated
[1]: https://github.com/TryGhost/Ghost/pull/11867
- Represents that logging is shared across all parts of Ghost at present
* moved core/server/lib/common/logging to core/shared/logging
* updated logging path for generic imports
* updated migration and schema imports of logging
* updated tests and index logging import
* 🔥 removed logging from common module
* fixed tests
refs https://github.com/TryGhost/members.js/issues/30
- Member cannot update their email directly but need to do it via magic link sent to new email address
- Previous profile update change had allowed email to be updated directly as well for authenticated member
no issue
- adds `search` bookshelf plugin that calls out to an optional `searchQuery()` method on individual models to apply model-specific SQL conditions to queries
- updated the base model's `findPage()` method to use the search plugin within `findPage` calls
- added a `searchQuery` method to the `member` model that performs a basic `LIKE %query%` for both `name` and `email` columns
- allowed the `?search=` parameter to pass through in the `options` object for member browse requests
* moved `server/config` to `shared/config`
* updated config import paths in server to use shared
* updated config import paths in frontend to use shared
* updated config import paths in test to use shared
* updated config import paths in root to use shared
* trigger regression tests
* of course the rebase broke tests
* refactored core/frontend/services/proxy to import common dependency like a normal person
* removed all imports of `common/errors`
* 🔥 removed common/errors module
Co-authored-by: Vikas Potluri <vikaspotluri123.github@gmail.com>
no issue
- moves members stats generation for the admin graph from the client to the server
- outputs a basic totals count across a requested date range of 30, 90, 365 days, or all time. See below for the response shape
- leaves heavy lifting of the counts to the SQL engines - tested on a dataset of 100k members and query performance is <100ms
```
GET /ghost/api/canary/members/stats/?days=30
{
total: 100000,
total_in_range: 20000,
total_on_date: {
'2020-04-25': 19000,
'2020-04-26': 19500,
// continues until today's date
},
new_today: 200
}
```
* refactored `core/frontend/apps` to destructure common imports
* refactored `core/frontend/services/{apps, redirects, routing}` to destructure common imports
* refactored `core/frontend/services/settings` to destructure common imports
* refactored remaining `core/frontend/services` to destructure common imports
* refactored `core/server/adapters` to destructure common imports
* refactored `core/server/data/{db, exporter, schema, validation}` to destructure common imports
* refactored `core/server/data/importer` to destructure common imports
* refactored `core/server/models/{base, plugins, relations}` to destructure common imports
* refactored remaining `core/server/models` to destructure common imports
* refactored `core/server/api/canary/utils/serializers/output` to destructure common imports
* refactored remaining `core/server/api/canary/utils` to destructure common imports
* refactored remaining `core/server/api/canary` to destructure common imports
* refactored `core/server/api/shared` to destructure common imports
* refactored `core/server/api/v2/utils` to destructure common imports
* refactored remaining `core/server/api/v2` to destructure common imports
* refactored `core/frontend/meta` to destructure common imports
* fixed some tests referencing `common.errors` instead of `@tryghost/errors`
- Not all of them need to be updated; only updating the ones that are
causing failures
* fixed errors import being shadowed by local scope
no issue
- Removes global bodyParser middleware for membersApp and adds it to specific endpoints
- Removes global boolParser middleware for membersApp
We added bodayParser middleware to memebrsApp in [this](fe3eab1836) commit to read json requests for members update endpoint, but that had issues with stripe webhook parsing for `/webhooks` endpoint as stripe expects raw data to be passed down.
no issue
- updated `filter` plugin with appropriate label relationship and replacement config for NQL
- fleshed out member and label fixture data and tools to facilitate tests
no-issue
The flag currently defaults to `true` as we are still using stripe
direct. We expose it on the admin api config endpoint so that the
Ghost-Admin client can use it to conditionally render
- Allows member logged in with valid session to update their profile info - name, email, subscribed(newsletter subscription status)
- Adds new util method for formatted member response on the endpoints
- Adds common middlewares for body/bool parser and maintenance
- Adds `subscribed` status to member response
no issue
- shows the full URL instead of a truncated link in password reset email body. This is required for security and usability reasons (copy/paste)
closes#11825
- The initial implementation had a typo in a role name which didn't allow "Administrator" to edit post's "visibility" attribute
- Added unit tests to check administrator specific role and visibility attribute permission
- Add a query param that indicates whether signin/up succeeded or failed
- Add unit tests for all 3 possible cases for the createSessionFromMagicLink middleware
- Added an acceptance test to show the behaviour works in principle
- This restores the functionality from 3.14 as follows:
/members/ -> (with no route) rendered 404 error
/members/ -> (with route) renders members template
/members/?token=invalidtoken&foo=bar -> redirects to /?foo=bar
/members/?token=validtoken&foo=bar -> redirects to /?foo=bar
refs https://github.com/TryGhost/Ghost/pull/11790
- reduced complexity by sticking to one email for both normal reset and forced reset (locked staff accounts)
- exposed `siteTitle` for use in any email templates
- updated email copy to be suitable for both types of password reset
- Fixed session invalidation for "locked" user
- Currently Ghost API was returning 404 for users having status set to "locked". This lead the user to be stuck in Ghost-Admin with "Rousource Not Found" error message.
- By returning 401 for non-"active" users it allows for the Ghost-Admin to redirect the user to "signin" screen where they would be instructed to reset their password
- Fixed error message returned by session API
- Instead of returning generic 'access' denied message when error happens during `User.check` we want to return more specific error thrown inside of the method, e.g.: 'accountLocked' or 'accountSuspended'
- Fixed messaging for 'accountLocked' i18n, which not corresponds to the
actual UI available to the end user
- Added automatic password reset email to locked users on sign-in
- uses alternative email for required password reset so it's clear that this is a security related reset and not a user-requested reset
- Backported the auto sending of required password reset email to v2 sign-in route
- used by 3rd party clients where the email is necessary for users to know why login is failing
Co-authored-by: Kevin Ansfield <kevin@lookingsideways.co.uk>
no issue
- Adds new endpoint on integration to refresh admin/content api key secret
- Allows owner/admin to refresh their content or admin API keys for an integration via Ghost Admin
- Adds a new `refreshed` event to actions table for anytime an api_key secret is refreshed
- Added a wrapper around express.Router to our shared/express util
- Also export static and _express
- Use this shared util everywhre, meaning express is only used directly in this one file
- ATM this file is mostly an experiment / debug helper, it might be removed again later
- The aim is to have a minimal framework wrapping express that allows us to:
- reduce our usage of express() in favour of Router()
- unify some of our duplicated logic
- fix some structural issues e.g. Sentry
- make it easier to understand the codebase
refs https://github.com/TryGhost/Ghost/issues/11756
- we removed the fixed `width: 600px` to fix gmail scaling but that means we need a new way of creating a 600px centre column in Outlook
refs https://github.com/TryGhost/Ghost/issues/11756
- fixed gmail scaling problems
- the `width: 600px` on `.container` was forcing gmail to always render at 600px wide and then use scaling to resize the email to fit the device width
- for most emails gmail would also apply their own font resizing to compensate so it didn't look _too_ bad
- some emails however would not trigger the font resizing, most notably when posts contained a feature image, which would result in very small text
- removing the fixed `width: 600px` resolves the scaling problem and lets the email be truly responsive
- removed attribute selectors in the media query CSS
- gmail does not support attribute selectors
- attribute selectors used to be necessary for Yahoo Mail but this is no longer the case
- tested using litmus.com for all popular email clients
- Use array destructuring
- Use @tryghost/errors
- Part of the big move towards decoupling, this gives visibility on what's being used where
- Biting off manageable chunks / fixing bits of code I'm refactoring for other reasons
- Meant to cleanup the old api/canary/members earlier, removed now as it's unused
- Also removed all the duplicate references to labs.members in various places
- My codebase-wide replacement of const/let yesterday went one step too far :)
- grunt uglify:prod fails on the use of const
- This file should continue to use var for the time being cos there's plans to upgrade this script already :)
- Clarify that the parent app has 2 distinct parts: backend and frontend
- Frontend app takes members and site apps + the frontend SSL redirect middleware
- Backend app already has admin + API (and the SSL redirect needs significant work)
- There's a lot more to do here, but this increases clarity
- create a new app for the /members/ endpoint
- moved all /members/ routes and middleware onto this app
- helps to separate members and frontend/site logic so we can start to decouple things more
refs https://github.com/TryGhost/Ghost/issues/11756
- updates `@tryghost/kg-default-cards` which includes a VML version of video embed card fallbacks
- fixes play button styling for Yahoo Mail
- adds a minimum height to video embeds so they appear more reasonable when images are not loaded
no issue
We changed the magic link route handling from setting global value to just redirecting to frontend in [this](d8d5d6b7d0 (diff-0d54454fd954b0203a71ec52df4bd4c0R96-R98)) commit, but missed removing `next()` call which attempts to send response again causing Unhandled rejection error. This change simply removes the extra `next()` call
refs #10898
- Execute string replacement on external paths
- Take non-top-level base URLs into consideration (to avoid #10776 dups)
- Added tests for all of the above cases
- Magic link token handling doesn't need to be global, this couples the system to the frontend, which isn't necessary
- Instead, we create a session from the token, and redirect to the frontend
- Move res.locals.members setting into existing middleware function instead of having it separate
- The existing createSessionFromToken was actually doing two things behind the scenes
1. Handling the ?token from the magic link and creating an actual session (mounted globally, which is not necessary)
2. Loading an existing session so that a member is logged in to the frontent
- IMO 1. is part of members, and doesn't need to be global
- IMO 2. is part of the frontend. It does need to be global but should NOT be hidden away behind the token middleware, as it wasn't clear what this was doing
- All var declarations are now const or let as per ES6
- All comma-separated lists / chained declarations are now one declaration per line
- This is for clarity/readability but also made running the var-to-const/let switch smoother
- ESLint rules updated to match
How this was done:
- npm install -g jscodeshift
- git clone https://github.com/cpojer/js-codemod.git
- git clone git@github.com:TryGhost/Ghost.git shallow-ghost
- cd shallow-ghost
- jscodeshift -t ../js-codemod/transforms/unchain-variables.js . -v=2
- jscodeshift -t ../js-codemod/transforms/no-vars.js . -v=2
- yarn
- yarn test
- yarn lint / fix various lint errors (almost all indent) by opening files and saving in vscode
- grunt test-regression
- sorted!
refs core/server/api/canary/oembed.js
- updated `kg-default-cards` to a version that will render a thumbnail and play button overlay for video embed cards when rendering to an email target
- added styling for video embed play button overlays to members email template
no issue
- embed cards now store metadata including thumbnail urls in their payload
- we want to use this metadata to render video cards in emails
- by default oembed endpoints return fairly small thumbnail images that don't look great when blown up
- oembed supports a `maxwidth` query param that will instruct oembed providers to return larger sizes of the content if available
no issue
- Handlebars now throws an Error for misuse errors within the if/unless
helpers, but our error handling checks for a TypeError
- this would skip using an IncorrectUsageError and ends up throwing a GhostError
- this commit removes the TypeError check and switches to using the
Handlebars error message
- Because we want devExperiments enabled when checking out from source, we have a committed config.development.json
- It works, but is a PITA if you want to have some local settings, as they have to be stashed (or get accidentally committed)
- This commit adds `config.local.json` as a local file that anyone can specify in any env, and it will be loaded
- Note that config.[specific env].json will trump it / overwrite it
- But you can still have settings alongside!
- added core/shared to watched folders in grunt
- moved sentry to shared
- moved express initialisation to a shared file
- always set trust proxy + sentry error handler
- use this new express init everywhere, and remove duplicate trust proxy and sentry error handler code
- renamed the parentApp in index.js to ghostApp, to reduce confusion with the layer that is named parentApp
- renamed the adminApp inside of parentApp to backendApp to reflect the fact it's both admin+api
- renamed a bunch more variables there to be backend, rather than admin
- renamed the api index.js file to app.js and created a new index which is an actual index
- put brand back, but only if dev experiments is enabled
- put members plans and allowSelfSignup back, but this is temporary as they need to live elsewhere
closes https://github.com/TryGhost/Ghost/issues/11659
- default `moment()` timezone is UTC and we store the `published_at` value in UTC
- fetch the configured timezone and convert the date into that timezone before formatting for inclusion in the email template
closes#11766, refs 7284227f1
- when we changed from host to hostname, more changed than just using the x-forwarded-host if trusted because express req.hostname does not return the port
- this causes issues with an infinite redirect if you try to set a different admin host with a port
- added a test to demonstrate the case, that didn't fail due to an error in the test logic
- switched from redirecting based on req.hostname to using req.vhost.host which has the correct trusted, requested value that we should rely on
- simplified the comparison logic to explicitly compare host with host
no issue
- the code didn't verify the existance of `timeoutInMS` before using it
- this caused `requestTimeout` to be `undefined`
- this commit adds the extra check so the fallback of 5000ms will be
used
- This code was a little verbose, which made it hard to see what was happening (it still is a bit)
- Used destructuring to reduce the code
- Renamed a few variables
- we had urlRedirects, urlRedirects.adminRedirect and adminRedirects
- all do kinda similar things, but for different contexts so for now I've done a minimal renaming for clarity
- and updated some comments!!
- also removed totally unnecessary if res.isAdmin clause, as we don't use that, and it was never true
- cache-control had some logic in it for private blogging + similar logic exists for members in site/app
- having it in 2 places is weird, and having it inside the mw makes the mw less generic/reusable
- instead of requiring config inside the middleware, we pass config in for the one case where this is used
- fixed tests that didn't test anything 🙈
- Moved normalize image mw from shared to api as it is not shared (except within the API)
- This file is only used in one part of the app, this updates the code structure to reflect this
- This is one of many similar changes needed to make it easier to refactor to the existing setup
- Moved upload validation mw from shared to api as it is not shared (except within the API)
- Co-located the code with the upload middleware, as it's small and gives us a nice API of .upload.single and .upload.validation
- This file is only used in one part of the app, this updates the code structure to reflect this
- This is one of many similar changes needed to make it easier to refactor to the existing setup
- These two functions have no dependencies and are only used in valiation/upload
- Co-locating the code makes it easier to move
- Exported them with a new module.exports._test pattern - we'll see about whether this is a good idea
- This is one of many similar changes needed to make it easier to refactor to the existing setup
- Moved upload mw from shared to api as it is not shared (except within the API)
- This file is only used in one part of the app, this updates the code structure to reflect this
- This is one of many similar changes needed to make it easier to refactor to the existing setup
- Moved update-user-last-seen from shared to api as it is not shared (except within the API)
- This file is only used in one part of the app, this updates the code structure to reflect this
- This is one of many similar changes needed to make it easier to refactor to the existing setup
- Moved api cors from shared to api as it is not shared (except within the API)
- This file is only used in one part of the app, this updates the code structure to reflect this
- This is one of many similar changes needed to make it easier to refactor to the existing setup
- Moved version-match from shared to api as it is not shared (except within the API)
- This file is only used in one part of the app, this updates the code structure to reflect this
- This is one of many similar changes needed to make it easier to refactor to the existing setup
refs https://github.com/TryGhost/members.js/issues/6
This exposes an endpoint on site url (`/members/ssr/member`) to get member's data in exchange for their session/identity on a theme when they are logged in. It essentially uses the same logic and data which is passed down to theme through handlebar helpers, and is used by members.js script to load member data.
no issue
This adds new public settings - description, logo, brand - and some public member settings - plans, allowSelfSignup to the open site endpoint which will be used by members.js for data initialization
no issue
This allows anyone using members service to fetch public membership plans for a site including currency and monthly/yearly charges. This is currently duplicated from theme service where we create the price helper, but will be cleaned up to keep logic in one single place.
- Moved ghost-locals from shared to parent as it is not shared
- This file is only used in one place, this updates the code structure to reflect this
- This is one of many similar changes needed to make it easier to refactor to the existing setup
- This allows shared middleware to be unhooked from the parent app
- Moved emit-events from shared to parent as it is not shared
- This file is only used in one place, this updates the code structure to reflect this
- This is one of many similar changes needed to make it easier to refactor to the existing setup
- Moved log-request from shared to parent as it is not shared
- This file is only used in one place, this updates the code structure to reflect this
- This is one of many similar changes needed to make it easier to refactor to the existing setup
- Moved request-id from shared to parent as it is not shared
- This file is only used in one place, this updates the code structure to reflect this
- This is one of many similar changes needed to make it easier to refactor to the existing setup
- Have a consistent structure so we can co-locate an app with its own middleware
- This is another small step in sorting out the giant mess that was web/shared/middleware
- Moved serve-public-file from shared to site as it is not shared
- This file is only used in one place, this updates the code structure to reflect this
- This is one of many similar changes needed to make it easier to refactor to the existing setup
refs 717567995b
- We should not be using relative URLs inside the admin panel anymore
- Removes a usage of "shared" middleware which isn't truly shared
- Moved handle-image-sizes from shared to site as it is not shared
- This file is only used in one place, this updates the code structure to reflect this
- This is one of many similar changes needed to make it easier to refactor to the existing setup
- Moved static-theme from shared to site as it is not shared
- This file is only used in one place, this updates the code structure to reflect this
- This is one of many similar changes needed to make it easier to refactor to the existing setup
- Moved serve-favicon from shared to site as it is not shared
- This file is only used in one place, this updates the code structure to reflect this
- This is one of many similar changes needed to make it easier to refactor to the existing setup
- Moved admin-redirects from shared to site as it is not shared
- This file is only used in one place, this updates the code structure to reflect this
- This is one of many similar changes needed to make it easier to refactor to the existing setup
no issue
- the `email.{html,plaintext}` fields are only used to display what was sent in the email so it doesn't make sense to store the mailgun-specific content which can be confusing when viewing in the admin area
- store the raw serialized post content with a basic no-data replacement of replacement strings rather than the output of full data fetching and mailgun transformation
no issue
- fixed plaintext templates being word wrapped and breaking across replacement strings
- updated `postEmailSerializer.serialize` to return the email template plus a replacements array that can be used for creating Mailgun-like recipient variable objects or more straight forward replacement
- updated email-preview API to work with the replacements data to show fallback data when previewing
no issue
- with the email replacements feature it's useful to have real member data when sending test emails from the PSM
- if the supplied email address matches a member then that member's data will be used for any replacements
no issue
- in our replacements the member properties are prefixed with `subscriber_` but this wasn't taken into account when requesting data from the member object
no issue
- the value of `mobiledoc` when submitting a page/post via the API must
be JSON, but we don't validate this
- this results in url-utils throwing an error, which ends up being a 500
- this commit adds a custom format to AJV to validate it is valid JSON
- also updates tests with bad JSON - 'a'
no issue
- adjusted mega's post serializer to get full email contents
- fetch `mobiledoc` from the API rather than the pre-rendered `html` and `plaintext`
- re-generate `html` using the mobiledoc renderer with an "email" target so that the email-only card content is included
- re-generate `plaintext` from the newly generated email html
- added replacement handling to mega's `getEmailData` function
- find all of our `%%{replacement "fallback"}%%` instances in the html template and push them into a replacements array with the respective property on the member instance and desired fallback
- transform the replacement for Mailgun compatibility. Mailgun uses `%recipient.variable_name%` for its template variables so we need to replace our custom replacement string with the compatible version. Our replacements system allows for the same replacement (`{subscriber_name}`) to be used multiple times and have different fallbacks, Mailgun doesn't support fallbacks so for each replacement we also need an indexed `variable_name` part so that we can put our fallbacks in the correct place
- perform the same Mailgun template transformation for the plaintext version except we re-use the replacements array to avoid bloating the API request to Mailgun with duplicate template variables for every recipient
- swapped `reduce` for a plain loop for easier readability
fixes#11740
- there was a discrepancy in the use of `private_blog` within the
code, and `private_block` in the default config
- this commit switches the code to `block` in order to avoid breaking
existing configs
- in 3.13.2 the importer always throws the error "The "path" argument must be of type string. Received an instance of Object"
- this is due to a change in method signature that wasn't accounted for
- added a test to catch similar changes to this code in future
fixes#11723
- when deleting an invite/label/tag/webhook that doesn't
exist, Ghost would throw a 500 error
- this commit catches the NotFoundError
- also rejects from model if nothing was found
- spotted in Sentry
- the helper dir also contained some code used with helpers - utils and helper-helpers?
- the goal here was for helpers to be the only thing in their folder so we can look at moving them out
- all other code has been moved to services/themes for now, which is not the right place either
- services/themes is a catch-all for theme storage, loading, validation, rendering and more, needs to be broken down
no issue
- moved `mobiledoc.renderers.mobiledocHtmlRenderer` to `mobiledoc.mobiledocHtmlRenderer` so that it's easier for the getter to access the parent objects getters
- removed all tests and dependencies that now live in @tryghost/mobiledoc-dom-renderer
- kept the `mobiledocHtmlRenderer` test because that's testing that we've correctly wired up our cards and atoms and the output is what we expect
no issue
- the blank document we use in Ghost is not specific to the html renderer
- renamed from `structure` to `document` to better represent its intent
- allows for easier extraction of `mobiledocHtmlRenderer`
no issue
- importing an LTS export would cause Ghost to throw a 500 error because
it used InternalServerError.
- an IncorrectUsageError is more applicable here
- this commit also updates the code comment and error message
- note: removed comment about WP exports because the plugin has been updated
to support the v2 & v3 format
- spotted in Sentry
no issue
- added the same 2sec timeout and `Ghost` user-agent header to the `rel="alternate"` oembed request that we use for the initial html page request
no issue
- Knex removed their use of several Bluebird methods, including `return`
- our code used `return`, but mostly to return null after a destroy action
- these uses have been replaced with `.then(() => null)` in order to
continue returning null and to avoid breaking anything
no-issue
* Added default for getting origin of request
This function is used to attach the origin of the request to the
session, and later check that requests using the session are coming from
the same origin. This protects us against CSRF attacks as requests in
the browser MUST originate from the same origin on which the user
logged in.
Previously, when we could not determine the origin we would return
null, as a "safety" net.
This updates the function to use a secure and sensible default - which
is the origin of the Ghost-Admin application, and if that's not set -
the origin of the Ghost application.
This will make dealing with magic links simpler as you can not always
guaruntee the existence of these headers when visiting via a hyperlink
* Removed init fns and getters from session service
This simplifies the code here, making it easier to read and maintain
* Moved express-session initialisation to own file
This is complex enough that it deserves its own module
* Added createSessionFromToken to session service
* Wired up the createSessionFromToken middleware
no-issue
This services handles the registration and retrieval of adapters,
it normalises the config to look like:
{
[adapterType]: {
active: adapterName,
[adapterName]: adapterConfig
}
}
no issue
- missing modules required by an adapter weren't flagged up as missing,
but that the entire adapter was missing
- therefore, it was difficult to see what you were missing
- this commit handles the case where a module is missing, and displays
an error
* Refactored SessionStore to use @tryghost/errors
no-issue
* Updated tests to test exposed API
no-issue
This will make refactoring easier, as we only have the "public" contract to maintain
* Refactored session functionality to SessionService
no-issue
This splits the session logic away from the HTTP responding logic,
which will allows us to decouple session creation/modification from the
API. Eventually this can be used to create sessions based on magiclink
style tokens.
* Instantiated and exported the new SessionService
no-issue
* Refactored session middleware to take session service
no-issue
This removes duplication of code and makes the middleware more explicit
that it's just a wrapper around the session service.
* Updated to use external @tryghost/session-service
no-issue
fixes#11694
- if the post contained no body, the `.replace` would throw an error
- converted to an if-statement instead of doing `|| ''` because there
would be a floating full-stop
no issue
- moved card definitions to a new library `@tryghost/kg-default-cards`
- moved `createCard` factory function to a new library `@tryghost/kg-card-factory`
- moved image.manipulation lib to a new package called @tryghost/image-transform
- new package has an updated API signature, so the method calls have changed but the underlying code is identical
- removed the optional sharp dependency from Ghost, as this is now optionally required by the image-transform module
- mock non existant module util was defined twice
- split it out properly from the rest of the utils, update all references
- this allows us to move this util out of the codebase along with other code, e.g. the image manipulation code
no issue
- Migrations within a minor have to be named with numbered prefixes like 01-, 02-, 03-.
- These two migrations were merged into master in the same time window which lead to having incorrect naming
- Moved zipFolder to a new package
- also exposing extract-zip from the new package
- new package has the API pre-promisified
- also uses @tryghost/extract-zip instead of extract-zip, which has bugfixes
- Apps are marked as removed in 3.0, never officially launched and have been deprecated for at least 2 years.
- We've slowly removed bits that got in our way or were insecure over time meaning they mostly didn't work
- This cleans up the remainder of the logic
- The tables should be cleaned up in a future major
- Apps are marked as removed in 3.0, never officially launched and have been deprecated for at least 2 years.
- We've slowly removed bits that got in our way or were insecure over time meaning they mostly didn't work
- This cleans up the remainder of the logic
- The tables should be cleaned up in a future major
- The existing common.i18n library contained code for core and theme translations
- There is some shared logic and some theme-specific logic, and the theme-specific logic has dependencies we don't want in lib/common
- This refactor introduces an I18n base class that does all the main shared logic, with no dependencies on other parts of the codebase
- ThemeI18n then extends this logic, and replaces the functions it needs to handle differently and adds it's dependencies on config and settingsCache
- The class has several methods broken down into smaller pieces to make it easier to extend only the necessary parts
- The class also encapsulates all of its logic, without external functions or variables
- The function loadThemeTranslations becomes the 'init()' function overridden in themeI18n.
