- With 4.0 we have a brand new version of Casper, new fixtures and new default settings
- Fixture posts cover the key features and give users an introduction to how to use their site
- This all comes from the marketing and design teams to refresh the look and feel of Ghost and give users the best possible onboarding experience
Note: this fixture overhaul includes
- new content for new 4.0 features
- regenerated post content using our updated mobiledoc structure
- a switch from British to US English
refs 2bba9989db
- Note: this will require new fixtures so that the navigation links actually work
- These updates are all in aid of getting the best possible default setup and onboarding experinence for new Ghost users
- With 4.0 we have a brand new version of Casper, new fixtures and new default settings
- This all comes from the marketing and design teams to refresh the look and feel of Ghost
Note on accent color:
This commit changes the default accent colour again.
The intention is that new sites should get #FF1A75 (pink) as their default.
Any existing sites that do not have an accent colour set yet, should get #15171A (black) on upgrading to 4.0.
These are different as they are different experinces. Fresh sites will be guided to pick a color, so
a bright color is more visible and helps to see what can be done, whilst existing sites get a muted
black, that should be a sensible fall back color.
refs TryGhost/Team#535
We want to ensure that a site will always have a default value of
`'#15171A'` for the accent_color setting.
Since the boot process changed we have three cases to account for:
1. Setting does not exist
2. Setting exists with no value
3. Setting exists with a value
It is only in the case of 2. that we want the migration to update the
database with a default value.
In the case of 3. the site owner has already set a value, which we do
not want to override.
In the case of 1. the setting will be created (and populated with
default value) from the default-settings.json file, by the
populateDefaults method called from the settings service
We also update the accent_color setting to include a non-empty
validation, to ensure that the setting will always have a value, as
sites before 4.x may have an empty accent_color, we must update the
importer to set the default value if one is not present. Otherwise we
would run into validation errors and even if we didn't would have an
invalid database state.
no-issue
We were originally checking the state of the database, e.g. if a foreign
key constraint existed, so that we could conditionally act upon it. This
was to ensure that our migrations are idempotent.
Some database configurations, for example if you have many databases on
a single MySQL instance, would cause these information_schema queries to
take an exceptionally long time.
In order to speed up migrations, we instead attempt the action we want
to apply to the database, and then catch relevant errors to ensure the
migration is idempotent.
SQLite does not error when adding duplicate foreign or primary key
constraints, meaning that we must keep in pre-checks for these
operations, when running on SQLite
Co-authored-by: Daniel Lockyer <hi@daniellockyer.com>
no issue
Upgrading from v1 or v2 can result in successful upgrades but with mobiledoc errors showing in the logs:
```
NAME: InternalServerError
MESSAGE: Mobiledoc card 'card-markdown' not found.
```
The errors do not signify a problem as long as the 4.0 migrations run because those rename the deprecated card before re-rendering.
- `@tryghost/kg-default-cards` dropped support for `card-markdown` cards. 4.0 migrations handled this by renaming all `card-markdown` cards to `markdown` before re-generating any content
- 2.0 and 3.0 also had migrations that re-generated content but they are run before the 4.0 card rename migration meaning that the mobiledoc renderer sees cards that it doesn't know about. The behaviour for unknown cards is to log an error and skip rendering of that card
- by NOOPing the 2.0 and 3.0 migrations we eliminate the incompatibility errors and reduce the amount of processing the upgrade needs to perform
refs https://github.com/TryGhost/Ghost/pull/12736
refs https://github.com/TryGhost/Team/issues/467
knex's `parse()` method is only called on data when directly fetched from the db. This was causing problems when model instances are passed around via events for example because `.get('key')` will return data that was directly set on the model without having gone through the `parse()` transformations. The result of this inconsistency was settings appearing correct when Ghost started up but then being broken as soon as a setting was changed.
- moved absolute/relative->transform-ready URL transformations from the API input serializers to the model's `format()` method and replaced with a relative->absolute transform in API input serializers
- results in consistency because `.get()` on a settings model will always return an URL
- removed transform-ready->absolute transforms from the API output serializers as that is now handled at the model-layer
refs https://github.com/TryGhost/Team/issues/467
refs https://github.com/TryGhost/Ghost/pull/12731
- AMP helper fetches HTML directly from the database rather than fetching via the API so we can't rely on the API serializers to perform transforms for us
- switched the `relativeToAbsolute(html)` call to `transformReadyToAbsolute(html)` to match the new `__GHOST_URL__` storage format
refs https://github.com/TryGhost/Team/issues/467
refs https://github.com/TryGhost/Ghost/pull/12731
- settings are mostly fetched directly from the settings cache rather than via the API so they aren't subject to the API-level output serializers that transform URLs meaning that URLs in the front-end ended up with raw `__GHOST_URL__` replacement strings
- added images to the Settings model's `parse()` method so they are transformed immediately when fetching from the database
closes https://github.com/TryGhost/Team/issues/467
- switches to storing "transform-ready" URLs in the database
- transform-ready URLs contain a `__GHOST_URL__` placeholder that corresponds to the configured url that gives a few benefits
- much faster and less memory intensive output transformations through not needing to parse html or markdown - the transform can be achieved using a straightforward regex find+replace
- ability to change to/from or rename subdirectory without any manual updates to the database
- modified existing 4.0 url-transformation migration rather than adding another one and repeating the transformation on posts rows
refs https://github.com/TryGhost/Casper/pull/741
closes https://github.com/TryGhost/Team/issues/524
- Use a local-based format as the default format as suggested in https://github.com/TryGhost/Casper/pull/741
- reworked the helper to be easier to read and follow the different use cases
- introduced setting and resetting locale in tests via settingsCache and themei18n
- updated tests to cover more cases e.g. passing a date, this.published_at and no date
- added validation for user inputted dates because they could literally be anything
Co-authored-by: Hannah Wolfe <erisds@gmail.com>
refs https://github.com/TryGhost/Team/issues/525
We expect the member event tables to be large, and they contain what is
considered metadata. For this reason we do not want to include them in
the export of sites.
refs https://github.com/TryGhost/Team/issues/469
- Cleans up response format for mrr and volume stats endpoint to more consistent pattern
- Removes `unit` attribute for now as its not used
closes: https://github.com/TryGhost/Ghost/issues/12413
- portal should be included if there is no context so that it appears on 404 pages
- having this inside the if (context) block was an oversight
- we want to chain the accent color style block, so we move this to the end to make that possible
refs: https://github.com/TryGhost/Team/issues/510
- Changed to return the full hostSettings key, not just the billing URL
- We are introducing several more settings that are needed by Admin including limits
- Passing the whole object makes this much easier to reason about as Admin has the exact same config as the server
refs: https://github.com/TryGhost/Team/issues/510
- The current member limit was implemented as a member-specific concept
- The new limit service is much more generic, here we are swapping old for new
- The updated concept here is blocking all publishing, not just email sending, when a site is over its member limit
- To determine that we are publishing a post, we must be in the model layer. The code has been moved to the permissible function which makes sense as this is a permissions error that we are throwing
- I've left the extra check for email retries in, in case there is some loophole here (but we may wish to change it)
refs: https://github.com/TryGhost/Team/issues/510
- In the case that host config is provided, keep staff users within the limiti
- The definition of a staff user is a user with a role other than Contributor, and whose status is not inactive
- Contributors don't count
- Suspended (status inactive) users don't count
- Locked users DO count
- Invited users DO count
- You can't invite more staff users whilst there are pending invites
- You can't unsuspend a user, or change the role on a user in such a way as will take you over your limit
- You can't import staff users - all imported users are automatically set to Contributors
- As part of this work, we are changing the default Ghost user to a Contributor otherwise it uses up a staff user
Note: there is one known active bug with this commit.
- Assume you have one remaining user within your limit. You send an invite, this works.
- You cannot "resend" that invite, it will think you're sending a new invite and hit the limit
- You must "revoke" that invite first, and create a new one
- This bug exists because the resend function uses the add endpoint & does a delete+add, but this hits the permission check before the delete
refs https://github.com/TryGhost/Team/issues/510
- When the host config was introduced it was incorrectly introduced as host_settings instead of hostSettings
- All other Ghost config uses camelCase, so changing this now before it becomes a problem
- Note: Also removed some rogue return awaits that don't make sense. It's not possible to hit that case, but cleaning up anyway
refs: refs 74fe765410
- Some pages, like error pages have no context.
- In that case there is also no previous style or script tag and so the existingScriptIndex is -1, not 0/falsy :D
- This ensures we always add this style tag
refs https://github.com/TryGhost/Team/issues/509
- Allows to update and read 'locale' key along with the deprecated 'lang'
- In Ghost v5 the 'lang' key will be dropped and the migration in settings table will clean up the key name to match the one exposed through the APIs
refs https://github.com/TryGhost/Team/issues/509
- During 3.x we standardised on "lang" instead of "default_locale" for the site setting, which was an assumption based on an earlier change to @site.lang to make <html lang="@site.lang"> read nicer. This was a mistake as the field represents more than "lang" its a "locale". With this changeset we introduce a transition to use "locale" name for the value instead of "lang"
- Adds `@site.locale" value in as well as new 'locale' property in Content API's response
- "lang" will be considered as deprecated starting with API v4 and will be dropped completely with API v5
refs https://github.com/TryGhost/Team/issues/509
- Flipping around key/newKey pair allows to map multiple keys to the same field in the settings
- This becomes handy when there's a need to deprecate a field. For example, we are about to introduce a 'locale' setting which would need to map to 'lang' db key, with current structure it's impossible to have many:1 mapping because it's impossible to have duplicate keys in the JS object ("hash")
refs https://github.com/TryGhost/Team/issues/513
refs https://github.com/TryGhost/Team/issues/477
- We have skipped work to improve the 'defaultTo' value when working on Ghost 4.0 release, so adding this comment while context loaded
- defaultTo should not be set to anything as it leads to more maintenance work during major version bump
- having validation might make sense but could lead to similar maintenance work unless it's linked to some global nosion of "supported API versions" used everywhere
refs https://github.com/TryGhost/Team/issues/513
- Having exports in no particular order was making it hard to spot if there's anything missing
- Having a DESC order on the exported API versions makes it very easy to modify and maintain in the future
refs: https://github.com/TryGhost/Team/issues/510
- added and wired up the new limit service, which is a lazy-loaded service
- this handles the case that there are host limits set in config, and wraps all the logic needed for detecting exceeded limits & throwing limit errors
- expects limits to be set in config under `host_settings.limits`
- supported limits are managed in the limit service, outside of core
refs https://github.com/TryGhost/Ghost/issues/12610
refs https://github.com/mailgun/mailgun-js-boland/blob/v0.22.0/lib/request.js#L285-L333
The mailgun domain is used by the mailgun API to construct the URL for
the API. e.g for a domain of "mg.example.com" the URL for the API
messages would look like:
https://api.mailgun.net/v3/mg.example.com/messages
One weird thing about the mailgun API is that if the path does not map
to an API endpoint, then instead of a 404, we get a 200, with a body of
"Mailgun Magnificent API".
The `mailgun-js` library which we use, expects a JSON response, and will
return a body of undefined if it does not get one.
This all resulted in us trying to read the property `id` of an undefined
`body` variable. The fix here is to reject the containing Promise, if
there is no body. So that the default error handling will kick in.
refs https://github.com/TryGhost/Team/issues/513
- The bump does not effect scheduled post as URL with authentication token is generated during runtime for DefaultScheduler.
- This needs to be checked for SchedulerAdapter implementations which use persistant storage
refs https://github.com/TryGhost/Team/issues/513
- Bumping version as this is now the latest stable API.
- The change might cause this particular side effect (acceptable for major version): if a member requests a login URL when the instance is on 3.x version and site owner upgrades do 4.x before member authenticates through login link, the login will fail and the member will have to request a new login URL.
closes https://github.com/TryGhost/Team/issues/508
- if an accent colour is set in site settings, output a `--accent-color` CSS variable in a `<style>` tag through `{{ghost_head}}`
- allows themes to use the accent colour without adding an additional conditional with CSS variable declaration to their default template
closes https://github.com/TryGhost/Team/issues/485
In order to loop over all posts, we currently need to pass a visibility="all" flag to a foreach as default for all items in current `visibility` helper is set to `public`. For a post, this behaviour is unintuitive, and inconsistent with the API. Instead, the default visibility should be "all" for the posts. The update allows themes to get all posts directly without passing in visibility -
```
{{#get "posts"}}
{{#foreach posts}}
//Loops over all posts, not just `public`
{{/foreach}}
{{/get}}
```
refs https://github.com/TryGhost/Ghost/pull/12541
refs https://github.com/TryGhost/Ghost/pull/12689
- the analytics job had been switched to create it's own instance of EmailAnalyticsService to avoid requiring logging but the analytics extraction branch was created before this change and wasn't picked up when merging
- pulled `queries` option object into a separate file for re-use
- updated `fetchLatest` job to conform to extracted library interface
closes https://github.com/TryGhost/Team/issues/493
- all functionality except that directly related to Ghost's database and business logic now lives in external packages
- @tryghost/email-analytics-service
- @tryghost/email-analytics-provider-mailgun
refs https://github.com/TryGhost/Ghost/issues/12716
- This change is made to avoid using now deprecated 'v2' API anywhere in the codebase.
- Switching to 'v4' should not cause sideeffects as this parameter is always present within the URL when Admin API is used
refs https://github.com/TryGhost/Ghost/issues/12716
- The code in serializePostModel was broken and always defaulted to 'v3'! It refered to non-existent `model.get('api_version')` there's no such field in posts model! Changed the implementation so that the API version is passed in as a parameter to the method instead
- The style of providing "defaults" everywhere creates a need for future maintenance when we bump the version e.g in Ghost v5. Maybe reworking these methods to require a passed version and throwing an error instead would be more maintainable long-term?
refs https://github.com/TryGhost/Ghost/issues/12716
- The default API in Ghost v4 will be `'v4'`. When a new webhook is created or any data passes through serialization it should assume `v4` as a fallback if not specified.
refs https://github.com/TryGhost/Team/issues/498
Requests to fetch the data of the logged in member made without a
session cookie were responsing with 400 Bad Request. This was incorrect
and always should have been a 401 Unauthorized.
refs https://github.com/TryGhost/Team/issues/498
This was filling up logs with less than useful information - for every
single request made by a non-member to the frontend. Be gone!
no-issue
The slack setting in 1.x did not have a `username` property. When
updating from 1.x, the migration to move the slack setting from a JSON
blob to two individual settings, would assume that the setting in the
database would have a value for the `username` property. This resulted
in errors on SQLite "sqlite does not support inserting default values."
The fix here is to add defaults when reading from the database, meaning
that we will _never_ attempt to insert `undefined`
refs https://github.com/TryGhost/Team/issues/476
* Moved paid subscription events population migration
This migration relies on the members_stripe_customers and
members_stripe_customer_subscriptions tables having no orphaned records
in order for it to correcly generate its data.
The migration to clean up orphaned records in those tables has not been
implemented yet, moving this migration free's up the "14" slot
* Removed orphaned stripe data from SQLite3
SQLite databases do not handle removing orphaned stripe records after a
member has been deleted. Our migration to populate the paid subscription
events relies on each customer and subscription being associated with a
member.
refs https://github.com/TryGhost/Team/issues/494
The migrations in 3.1.0 which added email permissions did not add those
permissions to the roles. This means that whilst we have the permissions
in the database, only the Owner role could use any of them.
This migration ensures that the email related permissions are added to
the correct roles.
- Theme preview was not showing the same behaviour as a real theme because nulls were being encoded and decoded incorrectly causing nulls/empty strings to be treasted as truthy values
- Swap from using split to using proper query param parsing so that the code is more robust
- this still creates empty strings and the string 'null' so added a small function to decode these back to real nulls
- moved to its own file ready to be split out - there needs to be a bigger picture plan for this
- added unit tests to cover the known issues + some potential breakages from converting the header string to a query param object
no issue
Ghost Ignition 4.4.4 includes ElasticSearch logging, this PR enables that feature by adding the config into the shared logger used within Ghost.
refs https://github.com/TryGhost/Ghost/issues/12602
As part of the member events, we added a third status of 'comped'.
Members with a status of 'comped' should still be considered paid, so
this fixes the definition of the paid flag to take that into account.
refs 17feb14e4a
- The original commit adding this intended to add transactions, following the pattern of always forcing a transaction when we use bookshelf-relations
- (We use bookshelf-relations here because integrations have api-keys and webhooks associated wtih them and we upsert as one)
- These add and edit methods were inadvertently added to the wrong argument object/section of bookshelf (really fucking easily done, one day we will fix bookshelf so its easier to work with)
- Bottom line: these methods have never been called
- I tried moving them to the right section, but this created test failures throughout our acceptance tests:
- Error: Transaction query already complete, run with DEBUG=knex:tx for more info
- This is likely because we need to account for integrations being used as part of the auth step in the before part of tests
- In terms of yak-shaving, fixing these tests is one step too far right now. I think not having this code here at all is a better state than having it look like it works when it doesn't
closes https://github.com/TryGhost/Team/issues/464
Currently, if the last4 value for a customer is empty, we end up showing `null` in theme where the card details are shown. The fix updates the last4 to return a masked value of **** instead of null if it's empty, ensuring themes using the last4 values don't have a `null` value shown to the user.
refs https://github.com/TryGhost/Team/issues/472
The current `{{price}}` helper only works with `amount` to convert it into right value but doesn't allow any formatting with currency etc, leaving most of the work to theme. We want to be able to output well formatted prices. E.g. the API returns 5000 for 5 EUR but we want to output €5.
The updated {{price}} helper can take a plan object or plan amount currency and use them to output a well formatted price. It works with JS's built in Intl.NumberFormat behaviour to return output in different formats, also allowing theme devs to override formatting with options.
Examples:
With Plan object => `{{price plan}} → "€5"`
With Plan object and custom number format => `{{price plan numberFormat="long"}} → "€5.00"`
Output only currency symbol => `{{price currency='EUR'}} → "€"`
no issue
- the analytice-node v4 update contains a breaking change that throws an
exception if the message is over 32kb
- I'm pretty sure we won't hit this, but it's good to track these errors
anyway and stop Ghost from bombing out if there's an issue
- this commit wraps the tracking call and adds Sentry + logging to the
error
refs https://github.com/TryGhost/Team/issues/467
refs https://github.com/TryGhost/Team/issues/221
- we've introduced backwards-compatible changes to rendering in 3.0 such as srcset and sizes which will only have taken effect on posts created or edited since the changes were made
- 4.0 brings additional changes such as image card width/height
- re-generating the `html` field of all posts from the `mobiledoc` brings all content up to latest rendering output
- Continuing cleanup and clarification of the boot process
- By locating requires and inits more closely together its easier to see what happens where and to get timings
- Improved consistency of the debug begin/end statements
- Added "Step" comments to the main process
- Inlined the mountGhost function, for readability and consistency with the other steps
refs https://github.com/TryGhost/Team/issues/469
The initial implementation was just to get us going with the dashboard
but the requirements have changed now, rather than updating the code we
allow to pass the `limit` options so the Admin can choose how many
events to display.
- There is now one true way to start Ghost - you create a server, and then call start with an express app
- We may well expand this again to improve testing pathways in future, but it will be done with a bit more clarity about expectations
- moved logging variable outside of the try-catch block, so it doesn't need a 2nd require
- I was considering having config,logging etc shared components required in a function together, but it's less readable
- Having a try-catch for loading shared components before main boot is also confusing
- This is simplest and most readable IMO
refs dd715b33dc
- this is the last event that is used to trigger part of the standard boot process
- events make the code harder to read/reason about
- the urlservice is one of the most core and critical components in Ghost, possibly the biggest consumer of time and memory
- we want to have the work it is doing front and center so that we can improve it
closes https://github.com/TryGhost/Ghost/issues/12506
- adds an error handler that will rename the backup folder to the original name if the newly uploaded theme wasn't saved successfully
refs https://github.com/TryGhost/Team/issues/221#issuecomment-759105424
- Mailgun responds to an email send with a provider id in the format `<x@y.com>` but everywhere else it's used in their API it uses the format `x@y.com`
- updates email batch save to strip the brackets, and migration removes brackets from existing records so we no longer have to add special handling for the stored id any time we use it
refs b1a98b0b67
- note: I already replaced server.start with themes.ready in the above commit
- events make the code harder to read/reason about
- long term it would be nice to have a concept of hooks for services, but for now explicit is clearer
refs https://github.com/TryGhost/Ghost/issues/12496
- having the logging require here means that workers wanting to use the db are unable to do so without requiring logging as a side-effect
- `connection.loggingHook` does not appear to be widely used for anything outside of specific debugging scenarios when using MySQL so it should be safe to disable until a proper fix is found for workers+logging leaking file descriptors
refs https://github.com/TryGhost/Ghost/issues/12496
By requiring the models layer the shared logging util was being required as a side-effect causing the open file descriptors problem to continue. Removing logging from the models layer isn't feasible due to deep require chains spreading across the codebase, it's much quicker to remove the need for models in the analytics job.
- models layer was only needed because it's used by the session service
- updated analytics job to create it's own instance of `EmailAnalyticsService` rather than the default instance in order to pass in custom dependencies
- pass in custom `logging` object that uses `parentPort.postMessage` as a way of writing log output
- pass in custom `settings` object that returns settings that have been manually fetched and cached during job instantiation
refs https://github.com/TryGhost/Ghost/issues/12633
Adds new `browse` endpoint for emails that allows Admin to check performance of newsletters over time and show stats on dashboard as primary usecase
refs https://github.com/TryGhost/Ghost/issues/12496
- Handling logging in the main thread avoids file handle leaks which happen due to leaky implementation of bunyan logger (see referenced issue for more context)
- Bumped job-manager version to allow for `workerMessageHandler` callback funciton
refs https://github.com/TryGhost/Ghost/issues/12496
- Using ghost-ignition logging caused file handle leaks. As there is no straight-forward way to handle write streams with bunyan (ghost-ignition's underlying logging library) this method of logging was chosen as an alternative to keep the amount of open file handles to minimum
- The follow up changes will include custom formatter for jobs service which should make logging match the same format as has been used inside the jobs
refs https://github.com/TryGhost/Ghost/pull/12598
- This changeset adds idepmotence to situations where unique contraint has to be dropped or added to the table
- Note '4.0/07-alter-unique-constraint-for-posts-slug.js` was migration that was effected by lack of idempotence
refs https://github.com/TryGhost/Ghost/issues/10318
- Storing JSON object in settings has caused multiple bugs in the past and was considered an antipattern
- This is a last bit in the long process of getting rid of "object" settings in the database. At this point there should be no settings with this type. Yey!
- In the old boot the server wasn't started til we were ready
- In new boot, we start the server immediately and send the old started event
- Then, when we are ready to accept some traffic, we send a ready event
- At the moment, ready isn't quite sent at the right time:
- It _should_ be when we're ready to serve real traffic, not just send 503s
- This is after the URL generation has finished
- But this requires more refactoring work :(
- So for now we send when everything else is ready
- This really needs some tests
- Notify is a more familiar name e.g. systemd has the sd_notify system which this is similar to
- We're actually announcing the server started, it's not actually ready for traffic (will serve 503s)
- use theme.ready for loading themes instead of server.start and properly clean this up
- remove server.start and server.stop as they are no longer used (only server.start was used, and only for themes)
- we're moving away from the pattern of using global events like this as they are hard to reason about
- This commit removes the old boot process and any files that are no longer needed as a result
- Remove the duplicate event for triggering inactive themes to load
- Tidied up a few other bits
- moved the DB init back into the boot file so everything is colocated - want it to be EASY to reason about whats happening
- added some more debug, and made the method signatures all the same
- expanded on comments to make it clear what the intentions are here, as there's a lot more work to do
no issue
- previously for SQLite tests we were pointing to a DB file within the
content/data folder, which would be stored on the filesystem
- by pointing this file to be under `/tmp`, the file is stored in memory
and should be a lot quicker to access
- this works great for me! - 2x faster test suite for SQLite
- however, Windows doesn't have a `/tmp` folder so we need to rewrite that
part to replace with `os.tmpdir()`
- With the new boot finalised it's clearer to see where this needs to go
- Update comments to add more clarity around what's happening and why
- Cleaned up the version require, as this was prep for maybe using version via config, which won't work in MigratorConfig
- We currently have 2 boot systems in the codebase, until now the tests were still running against the old one
- Regression tests using the forceStart + subdir flag no longer work. These need reworking, but this _should_ be easier later
- Note those tests work fine if they were the first tests run, it is only the case of trying to restart the server with a subdirectory after starting without that doesn't work
no issue
- all migrations should contain a logging message
- info for successful things, or warn if we deviated from what was
expected for some reason
- also added some spacing to make them easier to read
closes https://github.com/TryGhost/Team/issues/468
- updated post-gating
- clears excerpt if there's no access
- rebuilds excerpt from free preview if paywall card is used and there's no custom excerpt
closes https://github.com/TryGhost/Team/issues/466
- upgraded kg-default-cards to include paywall card
- extracted `htmlToPlaintext` from post model to shared util for re-use
- updated post-gating to set html+plaintext to the free preview if a paywall card has been used
- re-generates plaintext from the truncated html using `htmlToPlaintext` util
- display free content in the `{{content}}` helper via the default CTA template
refs https://github.com/TryGhost/Ghost/issues/10318
- API changes introduced:
canary/v4 Admin API
GET /settings/ (browse)
+ "unsplash" present in response as boolean value
GET /settings/:settingName (read)
+ "unsplash" present in response as boolean value
PUT /settings/ (edit)
+ "unsplash" updates setting, accepts ONLY boolean format
v3 Admin API
GET /settings/ (browse)
+ "unsplash" present in response with object value
GET /settings/:settingName (read)
+ "unsplash" present in response with object value
PUT /settings/ (edit)
+ "unsplash" updates setting, accepts either boolean or object formats
v2 Admin API
GET /settings/ (browse)
+ "unsplash" present in response with object value
GET /settings/:settingName (read)
+ "unsplash" present in response with object value
PUT /settings/ (edit)
+ "unsplash" updates setting, accepts object format
refs https://github.com/TryGhost/Ghost/pull/12631
- Initialy writen migration would break if executed twice
- Also accounted for situation if only part of the migration executed leaving only one of the values migrated
- If we are in preview mode, we output an additional string to set the accent color for portal
- This is semi-temporary, as we will probably have a proper way to pass portal settings later
- Allow the frontend to accept post messages to generate previews of the frontend
- Created a new endpoint in admin we can use to render these previews, which is possibly not necessary
- Supports a limited group of settings, which can easily be expanded, but care should be taken if expanding to use user-provided strings
refs https://github.com/TryGhost/Ghost/issues/10318
- Because members is effectively "enabled" by default starting Ghost 4.0 have hardcoded labs setting to be such. The alternative of removing this key from labs would be equivalent to `labs.members === false` which is undesireable and would mean additional work on theme developer's side.
refs https://github.com/TryGhost/Ghost/issues/10318
- `labs` setting is dropped from setting values as the use of JSON objec
to sore settings has been deprecated
- `labs` setting is no longer accepted as a paramter in the Settings API nor the
impoprter. The value is ignored if present in the POST/PUT requests and
returns 404 in case it is requested by key at `GET /settings/:key`
refs https://github.com/TryGhost/Team/issues/332
- The last value that has been used in the code was "members"
- By default members will be always "on" starting Ghost 4.0, so there's no need for this flag anymore
- Therefore there's no real need to keep "labs" around
refs https://github.com/TryGhost/Team/issues/467
refs https://github.com/TryGhost/Team/issues/221
- we added robust url transformation in Ghost 3.0 which ensures newly created or edited content is stored with relative URLs in the database
- this migration loops over all posts (and posts_meta) in the database performing the absolute->relative transformation to normalise data for old posts
closes https://github.com/TryGhost/Ghost#12562
- From NodeJS v13 `error.errno` returns error code instead of a string. Because of that use friendly "port is already in use" message did not work anymore.
- Changed to use `error.code` which acts the same way as `error.errno` in older NodeJS versions.
refs https://github.com/TryGhost/Ghost/issues/12646
- adds `ghostVersion` property to mobiledoc of all posts so that we can pin the rendering output of existing content to protect against future breaking changes
- cleans any deprecated markdown card names from old Ghost 1.0 content
- cleans up HR cards that had an [accidental payload added](610d801bcf)
refs https://github.com/TryGhost/Ghost/issues/12646
- if the version is missing then the content was created before Ghost 4.0
- setting the version to `'3.0'` means it will continue to use the same rendering output so there are no unexpected breaking changes when migrating content
refs https://github.com/TryGhost/Ghost/issues/12602
* Updated members_status_events table
By replacing the `status` column with a `from_status` and `to_status`
column, we are able to track the changes between multiple statuses
easier, and accumulate the data. e.g. the delta of paid members in a
given time range is the sum of the `to_status` columns set to 'paid'
minus the sum of the `from_status` columns set to 'paid' within that
time range
* Updated MEGA to handle addition of 'comped' status
With the addition of the 'comped' status, we need to ensure that MEGA
will still send emails to the correct recipients. I've opted to use an
"inverse" filter, as that is the intention of the free/paid split in
MEGA - as far as MEGA is concerned, "free" is the opposite of "paid"
* Updated customQuery for MemberStatusEvent
With the `status` column replaced with `from_status` and `to_status`
this allows us to fix and update the customQuery to correctly accumulate
the data into deltas over time, broken down by day.
* Populated members_status_events table
As the table will be used to generate deltas, we need to backfill the
data so that existing sites will be able to sum up the deltas and
calculate correct data.
The assumptions used in backfilling is that a Member's current status,
is their only status.
refs https://github.com/TryGhost/Ghost/issues/12602
As part of collecting Member event data, we have added a third status
for members "comped" - this fixes the population of the column to handle
this
refs https://github.com/TryGhost/Ghost/issues/12602
The gross volume aggregate query was incorrectly naming the selected `SUM(amount)` value as `gross_volume` instead of expected `volume_delta`
refs https://github.com/TryGhost/Ghost/issues/12633
Adds new stats endpoints for showing member dashboard, which fetches aggregate date over member events tables to form relevant datasets -
- Adds new endpoint for tracking MRR of site over time - `/members/stats/mrr`
- Adds new endpoint for tracking subscribers on site over time - `/members/stats/subscribers`
- Adds new endpoint for tracking gross volume of site over time - `/members/stats/gross_volume`
- Adds new endpoint for tracking member count on site over time - `/members/stats/count`
refs https://github.com/TryGhost/Ghost/issues/12646
- when a blank mobiledoc document is created, ensure the correct Ghost version is added so content output is pinned to the version of Ghost that created it
- Currently the new boot process breaks the loading of inactive themes because the theme loader event isn't wired until after the server.start event is fired
- This is a WIP fix, there needs to be some bigger refactoring of the boot process once the old process is gone
refs https://github.com/TryGhost/Ghost/issues/12646
- `card-markdown` support is being removed from the default mobiledoc cards so we need to make sure imported content doesn't fail
- removed long-unused `cardName` payload property from tests
refs https://github.com/TryGhost/Ghost/issues/10318
- Object format used in previous "slack" setting was considered an
anti-pattern. Flag structure of separate slack_url and slack_username
values was extracted out of the "slack" JSON.
no refs
The Portal URL that is shipped with every Ghost version is so far hardcoded in `ghost_head` and updated every-time we ship a new Portal minor/major change. This change brings the Portal URL inside the default Ghost config, which has few advantages -
- Allows easier access/managing of active Portal url/version
- Allows override for Portal URL for development/other purposes, where `config.*.json` allows using a Portal URL pointing to locally built copy for testing
refs https://github.com/TryGhost/Ghost/issues/12602
As we only want to add events for when an email is changed, we need to store the previous email if we want a complete log of all emails for a member
refs https://github.com/TryGhost/Ghost/issues/12608
- adds `admin/canary/themes/install` endpoint to the Admin API
- requires two query params. `source` must be set to "github". `ref` should refer to a GitHub repo in the format "{org}/{repo}"
- downloads zip archive for the repo from github
- runs downloaded zip through the same process as uploaded zips
refs https://github.com/TryGhost/Ghost/issues/12602
* Added members_payment_events table
This table will store successful and unsuccessful payment attempts, and
can be used to calculate gross volume over time.
* Added members_login_events table
This table can be used to audit member logins
* Added members_email_change_events table
This table will allow us to store a history of email addresses associated with a member
* Added members_status_events table
This table will allow us to track the change in status over time for members, as well
as calculate aggregates over time, e.g. paid members over time
* Added members_paid_subscription_events
This table will allow us to track subscriptions changes for members, as well as
calculating MRR over time
refs 0b79abf5b2
- in the referenced commit, we refactored the boot code to simplify it
- one of the required files was not updated with the change, so the path
was wrong
- this wouldn't have been easily visible from local testing because it's
based upon a config flag that isn't on by default
- this commit updates the path to the file
no issue
refs e3a0bb535f
Previously, {{content}} helper was updated use default or custom template to show a CTA when trying to use helper without access to content.
While the change was expected to not affect any existing themes as `{{content}}` helper is not supposed to be used on member sites without `access` check, we found quite a few themes use a pattern of using `content.hbs` with `{{content}}` helper inside them as default, which causes infinite loop in how content helper works in certain cases.
Whilst this followed the pattern of the other helpers using a partial with the same name, there are 2 key differences:
- this partial template pattern is being introduced way after the inception of themes with a fairly generic name “content”
- this partial template isn’t used to render the helper all the time - just the CTA part under certain circumstances.
This change updates the template name to `content-cta.hbs` to which makes it less likely to clash, and makes more sense as to what it is.
refs https://github.com/TryGhost/Ghost/issues/12567
- Changing unique constraint from slug to slug+type should allow for posts and pages to be created with the same slug
- The constraint will be present on application layer for API v4 while we figure out how to deal with it in API v5
Background:
- Ghosts existing boot process is split across multiple files, has affordances for outdated ways of running Ghost and is generally non-linear making it nigh-impossible to follow
- The web of dependencies that are loaded on boot are also impossible to unpick, which makes it really hard to decouple Ghost
- With 4.0 we want to introduce a new, linear, simpler, clearer way to boot up Ghost to unlock decoupling Ghost into much smaller pieces
This commit:
- adds a new ghost.js file which switches between boot mode with `node index` or `node index old` so that if we find bugs we can work around them this week
- Note: the old boot process will go away very soon, but ghost.js will remain as the interface between the command to start Ghost and the application code
- reworks the database migration process into a standalone utility, so that the DB is handled as one simple step of the boot process, decoupled from everything else
- is missing tests for this new db utility
- leaves a lot of work to do around loading core code, services, express apps in a sensible order, as work to fix this would start to break the old boot process
- doesn't use the new maintenance app because we aren't restarting the server here, instead we have the concept of a "core app" that starts in maintenance mode - need to think about how apps will be decoupled in the near future
closes https://github.com/TryGhost/Ghost/issues/12600
The bookshelf-relations plugin which we use will **remove** all
relations when they are set to an empty array, but will leave them alone
if it's set to undefined.
Our logic to deduplicate uppercase & lowercase version of the same label
was in advertently always setting the labels to an array, but when the
model was saved without passing the labels, this array would be empty.
Here we've added a check which will skip all label handling, if there
are no labels set.
refs https://github.com/TryGhost/Ghost/issues/12602
In order to build up a list of un/subscribes over time we have to use
the customQuery functionality to run SQL aggregates - this is "hidden"
behind an option, so that we can find{All,Page} as usual.
refs https://github.com/TryGhost/Ghost/issues/12602
Member Events in general are read-only after they've been created, so
we've explicitly disallowed destroying and editing via the model
refs https://github.com/TryGhost/Ghost/issues/12602
This allows us to backfill the data for testing purposes as well as for
use by existing sites. The mechanism for determining the date of an
unsubscribe event is crude, we use the date of the last update, which we
might expect to be _sometimes_ correct, but not always. We store the
`source` as NULL for unsubscribes as we cannot determine that with any
certainty whatsoever.
refs https://github.com/TryGhost/Ghost/issues/10921
refs https://github.com/TryGhost/Ghost/pull/11068
- When there is no twitter or og image specified for a post or a page the new default falls back to appropriate site-wide twitter/og image or site's cover image.
- New rules of follback follow this logic:
post/page/amp = post.twitter_image || post.feature_image || settings.twitter_image || settings.cover_image;
post/page/amp = post.og_image || post.feature_image || settings.og_image || settings.cover_image;
refs https://github.com/TryGhost/Ghost/issues/12567
- The method was using unneeded lodash dependency and was too complex for what it was doing
- Reshuffled internal code to use native JS filter/forEach iterators
refs https://github.com/TryGhost/Ghost/issues/12567
- Introduced here @@UNIQUE_CONSTRAINTS@@ notation allows to create unique contraints over multiple database fields. This will be needed to change posts' table unique constraint from `slug` to `slug+type`.
- The notation is equivalent to SQL's: UNIQUE(column_name1,column_name2)
- Example use in schema:
posts: {
slug: {type: 'string', maxlength: 191, nullable: false},
type: {type: 'string', maxlength: 50, nullable: false, defaultTo: 'post', validations: {isIn: [['post', 'page']]}},
'@@UNIQUES@@': [
['slug', 'type']
]
}
no issue
- when saving Stripe keys with `stripeDirect: true` config, if either key didn't match the key format the returned validation error always contained `stripe_secret_key`
- updated to output `stripe_publishable_key` if it was the publishable key that was invalid
refs https://github.com/TryGhost/Ghost/issues/12565
- Dropping tables happens sporadicaly, usually during major version releases. It made sense to create a utility based on previous migrations of this type (e.g.: 3.0/03-drop-client-auth-tables.js migration) to avoid code duplication in the future
no-issue
This removes all references to the members labs setting, any code that was run conditionally behind this flag now runs unconditionally.
* Removed usage of Members labs flag
* Removed tests for Members disabled
* Added dynamic keypair generation for when setting is missing
no refs
- Updates member model serializer to directly set subscriptions on member object instead of `stripe.subscriptions`
- Updates all references to members subscriptions from nested `stripe.subscriptions` to `subscriptions`
- Updates v3 API serializer to still use `stripe.subscriptions`
- Updates tests
no-issue
* Removed support for paid param from v3 & canary API
* Updated active subscription checks to use status flag
* Updated MEGA to use status filter over paid flag
* Removed support for paid option at model level
* Installed @tryghost/members-api@1.0.0-rc.0
* Updated members fixtures
no-issue
This value is nullable from the Stripe API so we're making sure that we
can store exact values locally.
SQLite3 does not supports altering columns so instead we have to:
1. Create a temp table and copy the data to it
2. Delete the original table
3. Recreate the original table with the necessary modifications
4. Copy the data from the temp table
5. Drop the temp table
refs #12568
- DI pattern through constructor allows for very clear separation of concerns and makes it easy to extract the class into a separate module
refs #12568
- When the server is in maintenance mode (e.g. during migration) it needs to serve a 503 page. Previously this role was delegated to the "frontend" which tightly coupled server bootup and frontend.
- With a dedicated HTTP application serving up 503 server is no longer coupled to the frontend during maintenance
refs #12568
- Allowing express app (frontend) to be booted after the migrations
decouples server maintenance mode from a need to have the "frontend"
loaded beforehand
refs #12160
This flag will allow us easier filtering of members via the API
* Added status column to members table
This flag will be used to determine if a member is free or paid, rather
than relying on joins with the customers and subscriptions tables.
* Added migration to populate members.status
As we add the column with a default value of "free" we only need to care
about the paid members here. We also preemptively handle migrations for
SQLite where there are > 998 paid members.
refs #12568
- The readivility of multiple nested promises is very poor
- Refactored this piece of code before doing a bigger reshuffle of frontend/server boot process
no issue
We added `portal-action` and `requestSrc` in 3.x to allow Portal to handle notifications only for auth actions trigged while using it directly, so that existing themes are not affected in any way. Going forward in 4.0, we don't want to have any special handling in backend for Portal but instead expect themes to handle any Portal specific behavior directly.
- Removes setting of `portal-action` for auth actions like signup
- Removes `requestSrc` being passed through to determine portal actions
closes https://github.com/TryGhost/Ghost/issues/12449
We’re starting to bump into errors with our current exporter due to the size of some of the tables in the db and hitting an issue with Ghost running out of memory during export. The intention for the export/import is not to be backup/restore functionality, but for exporting content and authors.
In addition, exporting and re-importing Stripe secret/publishable keys can cause unexpected side-effects for sites that can has major side-effects. This change -
- Removes `email_batches` and `email_recipients` tables from export data to reduce export size due to large amount of analytics data
- Removes stripe's secret/publishable/webhook keys to avoid unexpected issues with re-import
closes https://github.com/TryGhost/Ghost/issues/12347
This change allows a token to be used multiple times for the first 10
seconds after its initial use, this will stop dynamic link checking
software from invaliding magic links.
no refs
Adds new FirstPromoter settings similar to amp, which allows sites to take advantage of FirstPromoter to launch their own member referral program natively.
- Adds new firstpromoter settings group
- Adds `firstpromoter` setting to group
- Adds `firstpromoter_id` setting to group for FirstPromoter referral tracking id
- Updated tests
refs #12537
- Refactored overuse of rewire mocking blog-version.
- Fixed a bug introduced along the way when duplicate notifications errored instead of returning empty result
refs #12537
- notifications controllers were overbloated with non controller related code and were identical. It is important to reduce unnecessary code ahead of v4 API introduction
- Follow up commit will transform newly created module into a class following DI pattern
refs #12537
- Class syntax is preferred over module functions because of constructor parameter injection (DI) which allows for easier module decoupling and testing
refs #12537
- `stats` method in members controller is quite big and does much more then controller method code should - few calls to relevant modules
- Extracted code "as is" into members serivce
- Next step will be to refactor this module as a class pattern with DI parameters
refs https://github.com/TryGhost/Team/issues/446
Mailgun permanent failure events do not always correspond to unsubscribe-level events as originally thought, meaning some members could be unsubscribed unexpectedly due to delivery hiccups.
- disabled auto-unsubscribe on permanent failure events in the analytics event processor
- list maintenance will be added back in the future via alternative means
closes https://github.com/TryGhost/Ghost/issues/12493
- bumps `bookshelf-relations` to new version that allows for `hasMany` children to be kept when a parent record is destroyed
- adds necessary config to the `Member.email_recipients` relationship to keep related records when a member is deleted
closes https://github.com/TryGhost/Ghost/issues/12492
The changes to email processing models had set replyTo address for an email batch as `reply_to` instead of `replyTo` which was not picked by mailgun service for setting newsletter reply address
closes#12503
Since we include _all_ subscriptions in the `stripeSubscriptions`
relation, we must check that for an existing active complimentary
subscription to decide whether or not we should skip creation of a
complimentary subscription
ref #10898
- The redirects configuration's `to` & `from` URL parameters used to ignore it's query string parameters, which resulted in unexpected behavior
- Current changeset only partially fixes the issue. Now `to` URL's query parameters always take precedence over incoming query parameters and the rest of query parameters are passed through.
closes#12271
- When previous active theme did not have locale data for certain language, loading a theme which has such data did not result in correct locale being loaded
- Underlying issue was in settings cache being outdated during theme change related i18n initialization
- Fix focuses on removing settings cache dependency and and rely on most up to date data about currently active theme
- The benefit of this approach is reduced coupling with settings cache
closes#12181
refs https://github.com/TryGhost/Ghost/pull/12265
- The 'core' value is invalid for settings 'type' column
- The 'core' default value for 'type' column in model is also invalid
- Both need to be removed as they are never used and only introduce confusion into the codebase
no issue
- Previously used notation was not a valid JSDoc one. With this correction IDEs will provide more accurate information about all furnction parameters
refs https://github.com/TryGhost/Ghost-Utils/issues/118
- Duplicating error handling across jobs is not best developer experience. Also, having custom error handling logic did not allow for recommended worker script behavior: allowing for unhandled exceptions to bubble up and be managed by parent process
refs c1d66f0b01
- fixed base model allowing '@@INDEXES@@' as a permitted attribute/order
- fixed base model automatically setting `@@INDEXES@@` to null on the model when creating
- added `doAuth('members:emails')`
- creates an `email_batch` record attached to the first email in the fixtures
- creates an `email_recipients` record for each member
- runs analytics aggregation so the email and member counts are as expected
- added acceptance test for `/member/:id/?include=email_recipients`
no-issue
We now include all subscriptions as part of the member, so we need to
ensure the comped flag is only true if the member has an **active**
complimentary plan
refs https://github.com/TryGhost/Ghost-Admin/pull/1796
We want to be able to display an email activity timeline in Ghost-Admin for each member. The quickest way to achieve that right now is to provide access to the `email_recipient` data for the member when fetching, this will allow clients to build up a timeline based on the event timestamps included with each email_recipient/email pair.
- sets up `email_recipients` relationship in `Member` model
- updates members API read endpoint to accept an `email_recipients` include parameter
- appends `email_recipients.email` to the `withRelated` array when `email_recipients` is included so that we have data available for email subject and html/plaintext for previews
- updates members API output serializer to include the email_recipients object in the output
no issue
- sqlite will store a float in an integer column due to it's type affinity resulting in long decimal numbers in the UI when we're expecting an integer
- use the `ROUND()` function to ensure we're storing integers in place of floats when performing open rate average calculations
no-issue
This module encapsulates the work around performing imports, it
currently uses the concept of a "Job" which at the moment is not
persisted to the database, however when we want to look at resuming
imports after a server restart, this should give us the flexibility to
do it.
refs https://github.com/TryGhost/Ghost/issues/12461
- added two default aggregations for overall email count and opened email count
- when number of tracked emails is sufficient add the open rate aggregation to the update query
refs https://github.com/TryGhost/Ghost/issues/12461
- adds `members.email_count` and `members.email_opened_count` columns to contain cached counts for faster queries when outputting member data via API
- adds migration to populate cached counts with existing data
- tested locally on ~50k members which took ~4sec on mysql
- updates members output serializer to include the new fields in API output
refs https://github.com/TryGhost/Ghost/issues/12421
- add `orderRawQuery` function to members model so that we can ensure members with an open rate are ordered before members without an open rate no matter the order direction chosen
- added `email_open_rate` to members in the test fixtures to allow testing of order
refs https://github.com/TryGhost/Ghost/issues/12421
requires https://github.com/TryGhost/Ghost/pull/12457
- updates stats aggregator to calculate and store an open rate for each member
- uses two queries because I couldn't find a reasonable approach to perform the update in a single query as per the email aggregation
- benchmarked locally at <1sec/1000members
- will not store an open rate unless the number of tracked emails sent to a member is above a certain threshold (defaults to 5) to avoid new members being heavily weighted
- fixes typo in EmailAnalytics that was stopping member stats from being aggregated
refs https://github.com/TryGhost/Ghost/issues/12421
- nullable so we can distinguish between members that have and haven't received any trackable emails
- indexed because we'll be using this column for sorting
