refs https://github.com/TryGhost/Team/issues/469
- Cleans up response format for mrr and volume stats endpoint to more consistent pattern
- Removes `unit` attribute for now as its not used
refs: https://github.com/TryGhost/Team/issues/510
- Changed to return the full hostSettings key, not just the billing URL
- We are introducing several more settings that are needed by Admin including limits
- Passing the whole object makes this much easier to reason about as Admin has the exact same config as the server
refs: https://github.com/TryGhost/Team/issues/510
- The current member limit was implemented as a member-specific concept
- The new limit service is much more generic, here we are swapping old for new
- The updated concept here is blocking all publishing, not just email sending, when a site is over its member limit
- To determine that we are publishing a post, we must be in the model layer. The code has been moved to the permissible function which makes sense as this is a permissions error that we are throwing
- I've left the extra check for email retries in, in case there is some loophole here (but we may wish to change it)
refs: https://github.com/TryGhost/Team/issues/510
- In the case that host config is provided, keep staff users within the limiti
- The definition of a staff user is a user with a role other than Contributor, and whose status is not inactive
- Contributors don't count
- Suspended (status inactive) users don't count
- Locked users DO count
- Invited users DO count
- You can't invite more staff users whilst there are pending invites
- You can't unsuspend a user, or change the role on a user in such a way as will take you over your limit
- You can't import staff users - all imported users are automatically set to Contributors
- As part of this work, we are changing the default Ghost user to a Contributor otherwise it uses up a staff user
Note: there is one known active bug with this commit.
- Assume you have one remaining user within your limit. You send an invite, this works.
- You cannot "resend" that invite, it will think you're sending a new invite and hit the limit
- You must "revoke" that invite first, and create a new one
- This bug exists because the resend function uses the add endpoint & does a delete+add, but this hits the permission check before the delete
refs https://github.com/TryGhost/Team/issues/510
- When the host config was introduced it was incorrectly introduced as host_settings instead of hostSettings
- All other Ghost config uses camelCase, so changing this now before it becomes a problem
- Note: Also removed some rogue return awaits that don't make sense. It's not possible to hit that case, but cleaning up anyway
refs https://github.com/TryGhost/Team/issues/509
- Allows to update and read 'locale' key along with the deprecated 'lang'
- In Ghost v5 the 'lang' key will be dropped and the migration in settings table will clean up the key name to match the one exposed through the APIs
refs https://github.com/TryGhost/Team/issues/509
- During 3.x we standardised on "lang" instead of "default_locale" for the site setting, which was an assumption based on an earlier change to @site.lang to make <html lang="@site.lang"> read nicer. This was a mistake as the field represents more than "lang" its a "locale". With this changeset we introduce a transition to use "locale" name for the value instead of "lang"
- Adds `@site.locale" value in as well as new 'locale' property in Content API's response
- "lang" will be considered as deprecated starting with API v4 and will be dropped completely with API v5
refs https://github.com/TryGhost/Team/issues/509
- Flipping around key/newKey pair allows to map multiple keys to the same field in the settings
- This becomes handy when there's a need to deprecate a field. For example, we are about to introduce a 'locale' setting which would need to map to 'lang' db key, with current structure it's impossible to have many:1 mapping because it's impossible to have duplicate keys in the JS object ("hash")
refs https://github.com/TryGhost/Team/issues/513
refs https://github.com/TryGhost/Team/issues/477
- We have skipped work to improve the 'defaultTo' value when working on Ghost 4.0 release, so adding this comment while context loaded
- defaultTo should not be set to anything as it leads to more maintenance work during major version bump
- having validation might make sense but could lead to similar maintenance work unless it's linked to some global nosion of "supported API versions" used everywhere
refs https://github.com/TryGhost/Team/issues/513
- Having exports in no particular order was making it hard to spot if there's anything missing
- Having a DESC order on the exported API versions makes it very easy to modify and maintain in the future
refs: https://github.com/TryGhost/Team/issues/510
- added and wired up the new limit service, which is a lazy-loaded service
- this handles the case that there are host limits set in config, and wraps all the logic needed for detecting exceeded limits & throwing limit errors
- expects limits to be set in config under `host_settings.limits`
- supported limits are managed in the limit service, outside of core
refs https://github.com/TryGhost/Ghost/issues/12610
refs https://github.com/mailgun/mailgun-js-boland/blob/v0.22.0/lib/request.js#L285-L333
The mailgun domain is used by the mailgun API to construct the URL for
the API. e.g for a domain of "mg.example.com" the URL for the API
messages would look like:
https://api.mailgun.net/v3/mg.example.com/messages
One weird thing about the mailgun API is that if the path does not map
to an API endpoint, then instead of a 404, we get a 200, with a body of
"Mailgun Magnificent API".
The `mailgun-js` library which we use, expects a JSON response, and will
return a body of undefined if it does not get one.
This all resulted in us trying to read the property `id` of an undefined
`body` variable. The fix here is to reject the containing Promise, if
there is no body. So that the default error handling will kick in.
refs https://github.com/TryGhost/Team/issues/513
- Bumping version as this is now the latest stable API.
- The change might cause this particular side effect (acceptable for major version): if a member requests a login URL when the instance is on 3.x version and site owner upgrades do 4.x before member authenticates through login link, the login will fail and the member will have to request a new login URL.
refs https://github.com/TryGhost/Ghost/pull/12541
refs https://github.com/TryGhost/Ghost/pull/12689
- the analytics job had been switched to create it's own instance of EmailAnalyticsService to avoid requiring logging but the analytics extraction branch was created before this change and wasn't picked up when merging
- pulled `queries` option object into a separate file for re-use
- updated `fetchLatest` job to conform to extracted library interface
closes https://github.com/TryGhost/Team/issues/493
- all functionality except that directly related to Ghost's database and business logic now lives in external packages
- @tryghost/email-analytics-service
- @tryghost/email-analytics-provider-mailgun
refs https://github.com/TryGhost/Ghost/issues/12716
- This change is made to avoid using now deprecated 'v2' API anywhere in the codebase.
- Switching to 'v4' should not cause sideeffects as this parameter is always present within the URL when Admin API is used
refs https://github.com/TryGhost/Ghost/issues/12716
- The code in serializePostModel was broken and always defaulted to 'v3'! It refered to non-existent `model.get('api_version')` there's no such field in posts model! Changed the implementation so that the API version is passed in as a parameter to the method instead
- The style of providing "defaults" everywhere creates a need for future maintenance when we bump the version e.g in Ghost v5. Maybe reworking these methods to require a passed version and throwing an error instead would be more maintainable long-term?
refs https://github.com/TryGhost/Ghost/issues/12716
- The default API in Ghost v4 will be `'v4'`. When a new webhook is created or any data passes through serialization it should assume `v4` as a fallback if not specified.
refs https://github.com/TryGhost/Team/issues/498
Requests to fetch the data of the logged in member made without a
session cookie were responsing with 400 Bad Request. This was incorrect
and always should have been a 401 Unauthorized.
refs https://github.com/TryGhost/Team/issues/498
This was filling up logs with less than useful information - for every
single request made by a non-member to the frontend. Be gone!
no-issue
The slack setting in 1.x did not have a `username` property. When
updating from 1.x, the migration to move the slack setting from a JSON
blob to two individual settings, would assume that the setting in the
database would have a value for the `username` property. This resulted
in errors on SQLite "sqlite does not support inserting default values."
The fix here is to add defaults when reading from the database, meaning
that we will _never_ attempt to insert `undefined`
refs https://github.com/TryGhost/Team/issues/476
* Moved paid subscription events population migration
This migration relies on the members_stripe_customers and
members_stripe_customer_subscriptions tables having no orphaned records
in order for it to correcly generate its data.
The migration to clean up orphaned records in those tables has not been
implemented yet, moving this migration free's up the "14" slot
* Removed orphaned stripe data from SQLite3
SQLite databases do not handle removing orphaned stripe records after a
member has been deleted. Our migration to populate the paid subscription
events relies on each customer and subscription being associated with a
member.
refs https://github.com/TryGhost/Team/issues/494
The migrations in 3.1.0 which added email permissions did not add those
permissions to the roles. This means that whilst we have the permissions
in the database, only the Owner role could use any of them.
This migration ensures that the email related permissions are added to
the correct roles.
refs https://github.com/TryGhost/Ghost/issues/12602
As part of the member events, we added a third status of 'comped'.
Members with a status of 'comped' should still be considered paid, so
this fixes the definition of the paid flag to take that into account.
refs 17feb14e4a
- The original commit adding this intended to add transactions, following the pattern of always forcing a transaction when we use bookshelf-relations
- (We use bookshelf-relations here because integrations have api-keys and webhooks associated wtih them and we upsert as one)
- These add and edit methods were inadvertently added to the wrong argument object/section of bookshelf (really fucking easily done, one day we will fix bookshelf so its easier to work with)
- Bottom line: these methods have never been called
- I tried moving them to the right section, but this created test failures throughout our acceptance tests:
- Error: Transaction query already complete, run with DEBUG=knex:tx for more info
- This is likely because we need to account for integrations being used as part of the auth step in the before part of tests
- In terms of yak-shaving, fixing these tests is one step too far right now. I think not having this code here at all is a better state than having it look like it works when it doesn't
no issue
- the analytice-node v4 update contains a breaking change that throws an
exception if the message is over 32kb
- I'm pretty sure we won't hit this, but it's good to track these errors
anyway and stop Ghost from bombing out if there's an issue
- this commit wraps the tracking call and adds Sentry + logging to the
error
refs https://github.com/TryGhost/Team/issues/467
refs https://github.com/TryGhost/Team/issues/221
- we've introduced backwards-compatible changes to rendering in 3.0 such as srcset and sizes which will only have taken effect on posts created or edited since the changes were made
- 4.0 brings additional changes such as image card width/height
- re-generating the `html` field of all posts from the `mobiledoc` brings all content up to latest rendering output
refs https://github.com/TryGhost/Team/issues/469
The initial implementation was just to get us going with the dashboard
but the requirements have changed now, rather than updating the code we
allow to pass the `limit` options so the Admin can choose how many
events to display.
- There is now one true way to start Ghost - you create a server, and then call start with an express app
- We may well expand this again to improve testing pathways in future, but it will be done with a bit more clarity about expectations
refs https://github.com/TryGhost/Team/issues/221#issuecomment-759105424
- Mailgun responds to an email send with a provider id in the format `<x@y.com>` but everywhere else it's used in their API it uses the format `x@y.com`
- updates email batch save to strip the brackets, and migration removes brackets from existing records so we no longer have to add special handling for the stored id any time we use it
refs https://github.com/TryGhost/Ghost/issues/12496
- having the logging require here means that workers wanting to use the db are unable to do so without requiring logging as a side-effect
- `connection.loggingHook` does not appear to be widely used for anything outside of specific debugging scenarios when using MySQL so it should be safe to disable until a proper fix is found for workers+logging leaking file descriptors
refs https://github.com/TryGhost/Ghost/issues/12496
By requiring the models layer the shared logging util was being required as a side-effect causing the open file descriptors problem to continue. Removing logging from the models layer isn't feasible due to deep require chains spreading across the codebase, it's much quicker to remove the need for models in the analytics job.
- models layer was only needed because it's used by the session service
- updated analytics job to create it's own instance of `EmailAnalyticsService` rather than the default instance in order to pass in custom dependencies
- pass in custom `logging` object that uses `parentPort.postMessage` as a way of writing log output
- pass in custom `settings` object that returns settings that have been manually fetched and cached during job instantiation
refs https://github.com/TryGhost/Ghost/issues/12633
Adds new `browse` endpoint for emails that allows Admin to check performance of newsletters over time and show stats on dashboard as primary usecase
refs https://github.com/TryGhost/Ghost/issues/12496
- Handling logging in the main thread avoids file handle leaks which happen due to leaky implementation of bunyan logger (see referenced issue for more context)
- Bumped job-manager version to allow for `workerMessageHandler` callback funciton
refs https://github.com/TryGhost/Ghost/issues/12496
- Using ghost-ignition logging caused file handle leaks. As there is no straight-forward way to handle write streams with bunyan (ghost-ignition's underlying logging library) this method of logging was chosen as an alternative to keep the amount of open file handles to minimum
- The follow up changes will include custom formatter for jobs service which should make logging match the same format as has been used inside the jobs
refs https://github.com/TryGhost/Ghost/pull/12598
- This changeset adds idepmotence to situations where unique contraint has to be dropped or added to the table
- Note '4.0/07-alter-unique-constraint-for-posts-slug.js` was migration that was effected by lack of idempotence
refs https://github.com/TryGhost/Ghost/issues/10318
- Storing JSON object in settings has caused multiple bugs in the past and was considered an antipattern
- This is a last bit in the long process of getting rid of "object" settings in the database. At this point there should be no settings with this type. Yey!
- In the old boot the server wasn't started til we were ready
- In new boot, we start the server immediately and send the old started event
- Then, when we are ready to accept some traffic, we send a ready event
- At the moment, ready isn't quite sent at the right time:
- It _should_ be when we're ready to serve real traffic, not just send 503s
- This is after the URL generation has finished
- But this requires more refactoring work :(
- So for now we send when everything else is ready
- This really needs some tests
- Notify is a more familiar name e.g. systemd has the sd_notify system which this is similar to
- We're actually announcing the server started, it's not actually ready for traffic (will serve 503s)
- use theme.ready for loading themes instead of server.start and properly clean this up
- remove server.start and server.stop as they are no longer used (only server.start was used, and only for themes)
- we're moving away from the pattern of using global events like this as they are hard to reason about
- This commit removes the old boot process and any files that are no longer needed as a result
- Remove the duplicate event for triggering inactive themes to load
- Tidied up a few other bits
no issue
- all migrations should contain a logging message
- info for successful things, or warn if we deviated from what was
expected for some reason
- also added some spacing to make them easier to read
closes https://github.com/TryGhost/Team/issues/468
- updated post-gating
- clears excerpt if there's no access
- rebuilds excerpt from free preview if paywall card is used and there's no custom excerpt
closes https://github.com/TryGhost/Team/issues/466
- upgraded kg-default-cards to include paywall card
- extracted `htmlToPlaintext` from post model to shared util for re-use
- updated post-gating to set html+plaintext to the free preview if a paywall card has been used
- re-generates plaintext from the truncated html using `htmlToPlaintext` util
- display free content in the `{{content}}` helper via the default CTA template
refs https://github.com/TryGhost/Ghost/issues/10318
- API changes introduced:
canary/v4 Admin API
GET /settings/ (browse)
+ "unsplash" present in response as boolean value
GET /settings/:settingName (read)
+ "unsplash" present in response as boolean value
PUT /settings/ (edit)
+ "unsplash" updates setting, accepts ONLY boolean format
v3 Admin API
GET /settings/ (browse)
+ "unsplash" present in response with object value
GET /settings/:settingName (read)
+ "unsplash" present in response with object value
PUT /settings/ (edit)
+ "unsplash" updates setting, accepts either boolean or object formats
v2 Admin API
GET /settings/ (browse)
+ "unsplash" present in response with object value
GET /settings/:settingName (read)
+ "unsplash" present in response with object value
PUT /settings/ (edit)
+ "unsplash" updates setting, accepts object format
refs https://github.com/TryGhost/Ghost/pull/12631
- Initialy writen migration would break if executed twice
- Also accounted for situation if only part of the migration executed leaving only one of the values migrated
- Allow the frontend to accept post messages to generate previews of the frontend
- Created a new endpoint in admin we can use to render these previews, which is possibly not necessary
- Supports a limited group of settings, which can easily be expanded, but care should be taken if expanding to use user-provided strings
refs https://github.com/TryGhost/Ghost/issues/10318
- Because members is effectively "enabled" by default starting Ghost 4.0 have hardcoded labs setting to be such. The alternative of removing this key from labs would be equivalent to `labs.members === false` which is undesireable and would mean additional work on theme developer's side.
refs https://github.com/TryGhost/Ghost/issues/10318
- `labs` setting is dropped from setting values as the use of JSON objec
to sore settings has been deprecated
- `labs` setting is no longer accepted as a paramter in the Settings API nor the
impoprter. The value is ignored if present in the POST/PUT requests and
returns 404 in case it is requested by key at `GET /settings/:key`
refs https://github.com/TryGhost/Team/issues/332
- The last value that has been used in the code was "members"
- By default members will be always "on" starting Ghost 4.0, so there's no need for this flag anymore
- Therefore there's no real need to keep "labs" around
refs https://github.com/TryGhost/Team/issues/467
refs https://github.com/TryGhost/Team/issues/221
- we added robust url transformation in Ghost 3.0 which ensures newly created or edited content is stored with relative URLs in the database
- this migration loops over all posts (and posts_meta) in the database performing the absolute->relative transformation to normalise data for old posts
closes https://github.com/TryGhost/Ghost#12562
- From NodeJS v13 `error.errno` returns error code instead of a string. Because of that use friendly "port is already in use" message did not work anymore.
- Changed to use `error.code` which acts the same way as `error.errno` in older NodeJS versions.
refs https://github.com/TryGhost/Ghost/issues/12646
- adds `ghostVersion` property to mobiledoc of all posts so that we can pin the rendering output of existing content to protect against future breaking changes
- cleans any deprecated markdown card names from old Ghost 1.0 content
- cleans up HR cards that had an [accidental payload added](610d801bcf)
refs https://github.com/TryGhost/Ghost/issues/12646
- if the version is missing then the content was created before Ghost 4.0
- setting the version to `'3.0'` means it will continue to use the same rendering output so there are no unexpected breaking changes when migrating content
refs https://github.com/TryGhost/Ghost/issues/12602
* Updated members_status_events table
By replacing the `status` column with a `from_status` and `to_status`
column, we are able to track the changes between multiple statuses
easier, and accumulate the data. e.g. the delta of paid members in a
given time range is the sum of the `to_status` columns set to 'paid'
minus the sum of the `from_status` columns set to 'paid' within that
time range
* Updated MEGA to handle addition of 'comped' status
With the addition of the 'comped' status, we need to ensure that MEGA
will still send emails to the correct recipients. I've opted to use an
"inverse" filter, as that is the intention of the free/paid split in
MEGA - as far as MEGA is concerned, "free" is the opposite of "paid"
* Updated customQuery for MemberStatusEvent
With the `status` column replaced with `from_status` and `to_status`
this allows us to fix and update the customQuery to correctly accumulate
the data into deltas over time, broken down by day.
* Populated members_status_events table
As the table will be used to generate deltas, we need to backfill the
data so that existing sites will be able to sum up the deltas and
calculate correct data.
The assumptions used in backfilling is that a Member's current status,
is their only status.
refs https://github.com/TryGhost/Ghost/issues/12602
As part of collecting Member event data, we have added a third status
for members "comped" - this fixes the population of the column to handle
this
refs https://github.com/TryGhost/Ghost/issues/12602
The gross volume aggregate query was incorrectly naming the selected `SUM(amount)` value as `gross_volume` instead of expected `volume_delta`
refs https://github.com/TryGhost/Ghost/issues/12633
Adds new stats endpoints for showing member dashboard, which fetches aggregate date over member events tables to form relevant datasets -
- Adds new endpoint for tracking MRR of site over time - `/members/stats/mrr`
- Adds new endpoint for tracking subscribers on site over time - `/members/stats/subscribers`
- Adds new endpoint for tracking gross volume of site over time - `/members/stats/gross_volume`
- Adds new endpoint for tracking member count on site over time - `/members/stats/count`
