closes#8037🔥 Remove API-level default settings population
- This is a relic!
- We ALWAYS populate defaults on server start therefore this code could never run.
- This was a lot of complicated code that wasn't even needed!!
🎨 Move settings cache
- Move settings cache to be its own thing
- Update all references
- Adds TODOs for further cleanup
🎨 Create settings initialisation step
- Create new settings library, which will eventually house more code
- Unify the interface for initialising settings (will be more useful later)
- Reduce number of calls to updateSettingsCache
no issue
* ✨ Add new server start & stop events
* 🔥 Get rid of unused availableApps concept
- when we need an API endpoint for a list of apps, we'll build one 😝
* ✨ Move theme loading into a module
- move loading from API method to a module method and use as needed
- wire up read one vs read all as per LTS
- read one (the active theme) on boot, and read the rest after
- fudge validation - this isn't all that helpful
* Settings API tests need to preload themes
- this used to automatically happen as part of loading settings
- now we need to trigger this to happen specifically for this test
refs #7707
- be able to add a custom redirect file into the content folder
- define redirects as JSON format
The redirects feature is already present in the LTS branch.
I was not able to cherry-pick over, too many changes or conflicts.
Creating a PR to ensure 1. tests pass and 2. overview of code changes.
I had to add an example active theme to our test fixture utils, because otherwise Ghost will complain when forking Ghost.
* 🔥 kill apiUrl helper, use urlFor helper instead
More consistency of creating urls.
Creates an easier ability to add config changes.
Attention: urlFor function is getting a little nesty, BUT that is for now wanted to make easier and centralised changes to the configs.
The url util need's refactoring anyway.
* 🔥 urlSSL
Remove all urlSSL usages.
Add TODO's for the next commit to re-add logic for deleted logic.
e.g.
- cors helper generated an array of url's to allow requests from the defined config url's -> will be replaced by the admin url if available
- theme handler prefered the urlSSL in case it was defined -> will be replaced by using the urlFor helper to get the blog url (based on the request secure flag)
The changes in this commit doesn't have to be right, but it helped going step by step.
The next commit is the more interesting one.
* 🔥✨ remove forceAdminSSL, add new admin url and adapt logic
I wanted to remove the forceAdminSSL as separate commit, but was hard to realise.
That's why both changes are in one commit:
1. remove forceAdminSSL
2. add admin.url option
- fix TODO's from last commits
- rewrite the ssl middleware!
- create some private helper functions in the url helper to realise the changes
- rename some wordings and functions e.g. base === blog (we have so much different wordings)
- i would like to do more, but this would end in a non readable PR
- this commit contains the most important changes to offer admin.url option
* 🤖 adapt tests
IMPORTANT
- all changes in the routing tests were needed, because each routing test did not start the ghost server
- they just required the ghost application, which resulted in a random server port
- having a random server port results in a redirect, caused by the ssl/redirect middleware
* 😎 rename check-ssl middleware
* 🎨 fix theme-handler because of master rebase
* 🎨🔥 do not store settings in config and make settings cache easier available
- remove remembering settings value in theme config
- if we need a cache value, we are asking the settings cache directly
- instead of settings.getSettingSync we use settings.cache.get
- added TODO:
- think about moving the settings cache out of api/settings
- we could create a folder named cache cache/settings
- this settings cache listens on model changes for settings
- decoupling
* 🔥 remove timezone from config
- no need to store in overrides config and in defaults settings
* 🎨 context object helper
- replace config.get('theme') by settings cache
* 🎨 replace config.get('theme') by settings.cache.get
* 🎨 adapt tests
* fixes from comments
refs #7688
Adds an `uploads/icon/` endpoint to the api route to get a seperate entry point for blog icon validations. The blog icon validation will specifically check for images which have icon extensions (`.ico` & `.png`) and throw errors if:
- the icon file size is too big (>100kb)
- the icon is not a squaer
- the icon size is smaller than 32px
- the icon size is larger than 1000px
- the icon is not `.ico` or `.png` extension
TODOs for this PR:
- [X] get image dimensions
- [X] validate for image
- [X] size
- [X] form (must be square)
- [X] type
- [X] dimenstion (min 32px and max 1,000px)
- [X] return appropriate error messages
- [X] write tests
--------------------
TODOs for #7688:
- [X] Figure out, which favicon should be used (uploaded or default) -> #7713
- [ ] Serve and redirect the favicon for any browser requests, incl. redirects -> #7700 [WIP]
- [X] Upload favicon via `general/settings` and implement basic admin validations -> TryGhost/Ghost-Admin#397
- [X] Build server side validations -> this PR
closes#7766, refs #7579
- ensure we are using the correct brute keys
- ensure we are using req.ip as Ghost is configured with trust proxy option
- tidy up a little
closes#7769
Because Google AMP is bitching around and shows errors in Googles' webmaster tools for missing post images and blog icons, we decided to make AMP optional. It will be enabled by default, but can be disabled in general settings. Once disabled, the `amp` route doesn't work anymore.
This PR contains the back end changes for Ghost-alpha:
- Adds `amp` to settings table incl default setting `true`
- Adds `amp` value to our settings cache
- Changes the route handling of AMP app to check for the `amp` setting first.
- Adds tests to check the route handling and ghost_head output
- Includes changes to `post-lookup.js` as done by @kirrg001 in #7842
* 🛠 bookshelf tarball, bson-objectid
* 🎨 schema changes
- change increment type to string
- add a default fallback for string length 191 (to avoid adding this logic to every single column which uses an ID)
- remove uuid, because ID now represents a global resource identifier
- keep uuid for post, because we are using this as preview id
- keep uuid for clients for now - we are using this param for Ghost-Auth
* ✨ base model: generate ObjectId on creating event
- each new resource get's a auto generate ObjectId
- this logic won't work for attached models, this commit comes later
* 🎨 centralised attach method
When attaching models there are two things important two know
1. To be able to attach an ObjectId, we need to register the `onCreating` event the fetched model!This is caused by the Bookshelf design in general. On this target model we are attaching the new model.
2. We need to manually fetch the target model, because Bookshelf has a weird behaviour (which is known as a bug, see see https://github.com/tgriesser/bookshelf/issues/629). The most important property when attaching a model is `parentFk`, which is the foreign key. This can be null when fetching the model with the option `withRelated`. To ensure quality and consistency, the custom attach wrapper always fetches the target model manual. By fetching the target model (again) is a little performance decrease, but it also has advantages: we can register the event, and directly unregister the event again. So very clean code.
Important: please only use the custom attach wrapper in the future.
* 🎨 token model had overriden the onCreating function because of the created_at field
- we need to ensure that the base onCreating hook get's triggered for ALL models
- if not, they don't get an ObjectId assigned
- in this case: be smart and check if the target model has a created_at field
* 🎨 we don't have a uuid field anymore, remove the usages
- no default uuid creation in models
- i am pretty sure we have some more definitions in our tests (for example in the export json files), but that is too much work to delete them all
* 🎨 do not parse ID to Number
- we had various occurances of parsing all ID's to numbers
- we don't need this behaviour anymore
- ID is string
- i will adapt the ID validation in the next commit
* 🎨 change ID regex for validation
- we only allow: ID as ObjectId, ID as 1 and ID as me
- we need to keep ID 1, because our whole software relies on ID 1 (permissions etc)
* 🎨 owner fixture
- roles: [4] does not work anymore
- 4 means -> static id 4
- this worked in an auto increment system (not even in a system with distributed writes)
- with ObjectId we generate each ID automatically (for static and dynamic resources)
- it is possible to define all id's for static resources still, but that means we need to know which ID is already used and for consistency we have to define ObjectId's for these static resources
- so no static id's anymore, except of: id 1 for owner and id 0 for external usage (because this is required from our permission system)
- NOTE: please read through the comment in the user model
* 🎨 tests: DataGenerator and test utils
First of all: we need to ensure using ObjectId's in the tests. When don't, we can't ensure that ObjectId's work properly.
This commit brings lot's of dynamic into all the static defined id's.
In one of the next commits, i will adapt all the tests.
* 🚨 remove counter in Notification API
- no need to add a counter
- we simply generate ObjectId's (they are auto incremental as well)
- our id validator does only allow ObjectId as id,1 and me
* 🎨 extend contextUser in Base Model
- remove isNumber check, because id's are no longer numbers, except of id 0/1
- use existing isExternalUser
- support id 0/1 as string or number
* ✨ Ghost Owner has id 1
- ensure we define this id in the fixtures.json
- doesn't matter if number or string
* 🎨 functional tests adaptions
- use dynamic id's
* 🎨 fix unit tests
* 🎨 integration tests adaptions
* 🎨 change importer utils
- all our export examples (test/fixtures/exports) contain id's as numbers
- fact: but we ignore them anyway when inserting into the database, see https://github.com/TryGhost/Ghost/blob/master/core/server/data/import/utils.js#L249
- in 0e6ed957cd (diff-70f514a06347c048648be464819503c4L67) i removed parsing id's to integers
- i realised that this ^ check just existed, because the userIdToMap was an object key and object keys are always strings!
- i think this logic is a little bit complicated, but i don't want to refactor this now
- this commit ensures when trying to find the user, the id comparison works again
- i've added more documentation to understand this logic ;)
- plus i renamed an attribute to improve readability
* 🎨 Data-Generator: add more defaults to createUser
- if i use the function DataGenerator.forKnex.createUser i would like to get a full set of defaults
* 🎨 test utils: change/extend function set for functional tests
- functional tests work a bit different
- they boot Ghost and seed the database
- some functional tests have mis-used the test setup
- the test setup needs two sections: integration/unit and functional tests
- any functional test is allowed to either add more data or change data in the existing Ghost db
- but what it should not do is: add test fixtures like roles or users from our DataGenerator and cross fingers it will work
- this commit adds a clean method for functional tests to add extra users
* 🎨 functional tests adaptions
- use last commit to insert users for functional tests clean
- tidy up usage of testUtils.setup or testUtils.doAuth
* 🐛 test utils: reset database before init
- ensure we don't have any left data from other tests in the database when starting ghost
* 🐛 fix test (unrelated to this PR)
- fixes a random failure
- return statement was missing
* 🎨 make changes for invites
refs #7494, refs #7495
I saw tests adding permissions and roles twice. (see screenshots)
That happened because the setup in the test was mis-used and there is no restriction for static resources to create duplicates.
With this PR i suggest to make name unique.
no issue
- removes count from user checks model
- uses brute express brute with brute-knex adaptor to store persisted data on spam prevention
- implement brute force protection for password/token exchange, password resets and private blogging
closes#7628
With this PR we expose a public configuration endpoint.
When /ghost is requested, we don't load and render the configurations into the template anymore. Instead, Ghost-Admin can request the public configuration endpoint.
* 🎨 make configuration endpoint public
* 🔥 remove loading configurations in admin app
- do not render them into the default html page
* ✨ load client credentials in configuration endpoint
- this is not a security issue, because we have exposed this information anyway before (by rendering them into the requested html page)
* 🎨 extend existing configuration integration test
* ✨ tests: add ghost-auth to data generator
* ✨ add functional test
* 🔥 remove type/value pattern
* 🎨 do not return stringified JSON objects
closes#4172, closes#6948, refs #7491, refs #7488, refs #7542, refs #7484
* 🎨 Co-locate all admin-related code in /admin
- move all the admin related code from controllers, routes and helpers into a single location
- add error handling middleware explicitly to adminApp
- re-order blogApp middleware to ensure the shared middleware is mounted after the adminApp
- TODO: rethink the structure of /admin, this should probably be an internal app
* 💄 Group global middleware together
- There are only a few pieces of middleware which are "global"
- These are needed for the admin, blog and api
- Everything else is only needed in one or two places
* ✨ Introduce a separate blogApp
- create a brand-new blogApp
- mount all blog/theme only middleware etc onto blogApp
- mount error handling on blogApp only
* 🎨 Separate error handling for HTML & API JSON
- split JSON and HTML error handling into separate functions
- re-introduce a way to not output the stack for certain errors
- add more tests around errors & an assertion framework for checking JSON Errors
- TODO: better 404 handling for static assets
Rationale:
The API is very different to the blog/admin panel:
- It is intended to only ever serve JSON, never HTML responses
- It is intended to always serve JSON
Meanwhile the blog and admin panel have no need for JSON errors,
when an error happens on those pages, we should serve HTML pages
which are nicely formatted with the error & using the correct template
* 🐛 Fix checkSSL to work for subapps
- in order to make this work on a sub app we need to use the pattern `req.originalUrl || req.url`
* 🔥 Get rid of decide-is-admin (part 1/2)
- delete decide-is-admin & tests
- add two small functions to apiApp and adminApp to set res.isAdmin
- mount checkSSL on all the apps
- TODO: deduplicate the calls to checkSSL by making blogApp a subApp :D
- PART 2/2: finish cleaning this up by removing it from where it's not needed and giving it a more specific name
Rationale:
Now that we have both an adminApp and an apiApp,
we can temporarily replace this weird path-matching middleware
with middleware that sets res.isAdmin for api & admin
* 🎨 Wire up prettyURLs on all Apps
- prettyURLs is needed for all requests
- it cannot be global because it has to live after asset middleware, and before routing
- this does not result in duplicate redirects, but does result in duplicate checks
- TODO: resolve extra middleware in stack by making blogApp a sub app
* ⏱ Add debug to API setup
* 🎨 Rename blogApp -> parentApp in middleware
* 🎨 Co-locate all blog-related code in /blog
- Move all of the blogApp code from middleware/index.js to blog/app.js
- Move routes/frontend.js to blog/routes.js
- Remove the routes/index.js and routes folder, this is empty now!
- @TODO is blog the best name for this? 🤔
- @TODO sort out the big hunk of asset-related mess
- @TODO also separate out the concept of theme from blog
* 🎉 Replace middleware index with server/app.js
- The final piece of the puzzle! 🎉🎈🎂
- We no longer have our horrendous middleware/index.js
- Instead, we have a set of app.js files, which all use a familiar pattern
* 💄 Error handling fixups
refs #7489
* 🎨 protect error when creating owner
* 🎨 reset migration table
- temporary solution, see TODO's
* 🎨 use sephiroth in bootUp script
- do not populate the database
- ask sephiroth for database state
- do seeding manually (this will be removed in next seeding PR)
* 🎨 rewrite createTableIfNotExists because it causes error when running twice
- see knex issue
- hasTable and createTable
- indexes can cause trouble when calling them twice
* 🎨 tests: populate db in test env
- when forking db
- when starting ghost()
- this basically affects only the functional tests
* 🎨 server spec test adaption
- we now throw an error when database is not populated, instead of populating the database
* 🎨 migration spec adaption
- reset database now deletes migration table
- we will move the reset script into sephiroth and then we make it pretty
* 🎨 error creation adaption in bootUp
* 🎨 fixes
- sephiroth error handling
- fix tests
- 🛠 add bunyan and prettyjson, remove morgan
- ✨ add logging module
- GhostLogger class that handles setup of bunyan
- PrettyStream for stdout
- ✨ config for logging
- @TODO: testing level fatal?
- ✨ log each request via GhostLogger (express middleware)
- @TODO: add errors to output
- 🔥 remove errors.updateActiveTheme
- we can read the value from config
- 🔥 remove 15 helper functions in core/server/errors/index.js
- all these functions get replaced by modules:
1. logging
2. error middleware handling for html/json
3. error creation (which will be part of PR #7477)
- ✨ add express error handler for html/json
- one true error handler for express responses
- contains still some TODO's, but they are not high priority for first implementation/integration
- this middleware only takes responsibility of either rendering html responses or return json error responses
- 🎨 use new express error handler in middleware/index
- 404 and 500 handling
- 🎨 return error instead of error message in permissions/index.js
- the rule for error handling should be: if you call a unit, this unit should return a custom Ghost error
- 🎨 wrap serve static module
- rule: if you call a module/unit, you should always wrap this error
- it's always the same rule
- so the caller never has to worry about what comes back
- it's always a clear error instance
- in this case: we return our notfounderror if serve static does not find the resource
- this avoid having checks everywhere
- 🎨 replace usages of errors/index.js functions and adapt tests
- use logging.error, logging.warn
- make tests green
- remove some usages of logging and throwing api errors -> because when a request is involved, logging happens automatically
- 🐛 return errorDetails to Ghost-Admin
- errorDetails is used for Theme error handling
- 🎨 use 500er error for theme is missing error in theme-handler
- 🎨 extend file rotation to 1w
refs #6982
- create config util fn: getContentPath
- we can later let the user change the folder names in contentPath
- get rid of custom/default storage paths
[ci skip]
refs #7204
- Adds a new ThemeValidationError class
- This error has a top level message, but will also contain all the individual errors within the `errorDetails` property
- Updated the API error handling to return `errorDetails` if it is present
no issue
Double slashes are treated as a HTTP calls as specified in [RFC1801](http://www.ietf.org/rfc/rfc1808.txt). Because of this behaviour the uncapitalise created an open redirect. By removing double slashes in the path we ensure open redirects cannot be created.
As an example, please click the following URL: https://dev.ghost.org///Google.com/.
This issue has been reported by pentesters of our product [LearningSpaces.io](http://learningspaces.io).
closes#6588, #7095
* `ImageObject` with image dimensions (#7152, #7151, #7153)
- Returns meta data as promise
- returns a new Promise from meta data
- uses `Promise.props()` to resolve `getClient()` and `getMetaData()`
- Adds 'image-size' util
The util returns an object like this
```
{
height: 50,
url: 'http://myblog.com/images/cat.jpg',
width: 50
};
```
if the dimensions can be fetched and rejects with error, if not.
In case we get a locally stored image or a not complete url (like `//www.gravatar.com/andsoon`), we add the protocol to the incomplete one and use `urlFor()` to get the absolute URL. If the request fails or `image-size` is not able to read the file, we reject with error.
- adds 'image-size' module to dependencies
- adds `getImageSizeFromUrl` function that returns image dimensions
- In preparation of AMP support and to improve our schema.org JSON-LD and structured data, I made the following changes:
- Changes the following properties to be `Objects`, which have a `url` property by default and a `dimensions` property, if `width` and `height` are available:
- `metaData.coverImage`
- `metaData.authorImage`
- `metaData.blog.logo`
- Checks cache by calling `getCachedImageSizeFromUrl`. If image dimensions were fetched already, returns them from cache instead of fetching them again.
- If we have image dimensions on hand, the output in our JSON-LD changes from normal urls to be full `ImageObjects`. Applies to all images and logos.
- Special case for `publisher.logo` as it has size restrictions: if the image doesn't fulfil the restrictions (<=600 width and <=60 height), we simply output the url instead, so like before.
- Adds new property for schema.org JSON-LD: `mainEntityOfPage` as an Object.
- Adds additional Open Graph data (if we have the image size): `og:image:width` and `og:image:height`
- Adds/updates tests
* AMP router and controller (#7171, #7157)
Implements AMP in `/apps/`:
- renders `amp.hbs` if route is `/:slug/amp/`
- updates `setResponseContext` to set context to `['amp', 'post']` for a amp post and `['amp', 'page']` for a page, but will not render amp template for a page
- updates `context_spec`
- registers 'amp' as new internal app
- adds the `amp.hbs` template to `core/server/apps/amp` which will be the default template for AMP posts.
- adds `isAmpURL` to `post-lookup`
* 🎨 Use `context` in meta as array (#7205)
Instead of reading the first value of the context array, we're checking if it includes certain context values.
This is a preparation change for AMP, where the context will be delivered as `['amp', 'post']`.
* ✨ AMP helpers (#7174, #7216, #7215, #7223)
- Adds AMP helpers `{{amp_content}}`, `{{amp_component}}` and `{{amp_ghost_head}}` to support AMP:
- `{{amp_content}}`:
- Adds `Amperize` as dependency
- AMP app uses new helper `{{amp_content}}` to render AMP HTML
- `Amperize` transforms regular HTML into AMP HTML
- Adds test for `{{amp_content}}` helper
- Adds 'Sanitize-HTML` as dependendy
- After the HTML get 'amperized' we still might have some HTML tags, which are prohibited in AMP HTML, so we use `sanitize-html` to remove those. With every update, `Amperize` gets and it is able to transform more HTML tags, they valid AMP HTML tags (e. g. `video` and `amp-video`) and will therefore not be removed.
- `{{amp_ghost_head}}`:
- registers `{{amp_ghost_head}}` helper, but uses `{{ghost_head}}` code
- uses `{{amp_ghost_head}}` in `amp.hbs` instead of `{{ghost_head}}`
- `{{ghost_head}}`:
- Render `amphtml` link in metadata for post, which links to the amp post (`getAmpUrl`)
- Updates all test in metadata to support `amp` context
- Changes context conditionals to work with full array instead of first array value
- Adds conditionals, so no additional javascript gets rendered in `{{ghost_head}}`
- Removes trailing `/amp/` in URLs, so only `amphtml` link on regular post renders it
- Adds a conditional, so no code injection will be included, for an `amp` context.
- `{{amp_components}}`:
- AMP app uses new helper `{{amp_components}}` to render necessary script tags for AMP extended components as `amp-iframe`, `amp-anime` and `amp-form`
- Adds test for `{{amp_components}}`
no issue
- Source out validation logic into a upload validation middleware for all upload types (csv, image, subscribers). This unit can be later used for Ghost 1.0 as a pre validation core unit.
- More usage of route tests than controller tests. These are use case tests, a use case only changes if the product changes
refs #6413
- PUT endpoint to publish a post/page for the scheduler
- fn endpoint to get all scheduled posts (with from/to query params) for the scheduler
- hardcoded permission handling for scheduler client
- fix event bug: unscheduled
- basic structure for scheduling
- post scheduling basics
- offer easy option to change adapter
- integrate the default scheduler adapter
- update scheduled posts when blog TZ changes
- safety check before scheduler can publish a post (not allowed to publish in the future or past)
- add force flag to allow publishing in the past
- invalidate cache header for /schedules/posts/:id
refs #6949
- Adds a new VersionMismatchError with status 400 (bad request)
- Adds middleware that checks the X-Ghost-Version header if it is provided
- If it is not provided, the middleware does nothing
- If it is provided, and the versions match, the middleware does nothing
- If it is provided, and the versions don't match, the middleware returns a VersionMismatchError
- Includes both unit and a functional test to prove the middleware works alone and as part of the whole system
issues #6406#6399
- all dates are stored as UTC with this commit
- use moment.tz.setDefault('UTC')
- add migration file to recalculate local datetimes to UTC
- store all dates in same format into our three supported databases
- add option to remeber migrations inside settings (core)
- support DST offset for migration
- ensure we force UTC in test env
- run whole migration as transaction
- extend: Settings.findOne function
Refs #6642
- Do not send CORS headers on an invalid "origin"
header, but otherwise allow the response to
proceed normally. This enforces CORS for the browser
but does not blow up non-CORS requests.
refs #6354, #6495 & #6599
- don't allow config.database to be merged, instead, override it always
- make sure that route tests call done even when they error
closes#5887
- make use of the new 'fallthrough' option which landed in express-static 1.10.0
- change local-file-store and middleware serving `/public/` and `/shared/` files to use the new `fallthrough: false` option
- 404s are now served directly, without slashes or uncapitalise getting triggered
