refs #9601
- implementation of resource listener updated
- if you define two collections: `featured:true` (1) and `page:false` (2) you can run into the following bug:
- you create a published post (owned by (2))
- you change the status to featured
- still owned by (2), because the filter still matches (it's still not a static page)
- this adaption fixes the behaviour
- less smart logic, but less error prone
closes https://github.com/TryGhost/Ghost/issues/9674
- with dynamic routing the first collection get's the "index" context attached
- the index context signalises the main post listening route (first collection)
- this behaviour was present < 1.24 - we have to keep this behaviour
refs #9601
- if you call `express.Router()`, the router's name is always "router"
- that is caused by the closure behaviour in express:
- https://github.com/expressjs/express/blob/4.16.3/lib/router/index.js#L46
- Ghost creates a couple of express routers for dynamic routing
- it depends how much you configure in your routes.yaml file
- but every router is called "router"
- this is hard to work with
- with this router wrapping logic, we are able to give each router an exact name
If you enable `DEBUG=ghost:services:routing:*`, you have seen this before
> ghost:services:routing:ParentRouter site: mountRouter: router +0ms
With the wrapper logic, you will see:
> ghost:services:routing:ParentRouter site: mountRouter: StaticPagesRouter +0ms
- furthermore, if you have to access the router stack (`app.router.stack`), you can easily identify and find router instances by name
no issue
- if you define no collections, but a static route, it can happen that the target template to render
makes use of the {{ghost_head}} helper
- the {{ghost_head}} helper tries to create the primary rss feed url
- at the moment: no collections, no primary rss feed url
- if we offer the option to define custom rss rules, this function might need an extension
no issue
- this mock eat already too much of my/our time
- the idea of adding a knex mock was definitely a failed approach/try
- it's too much to maintaince and have not found a module which does this already
- we have to support any query format
- this is too crazy
- the idea was to use the knex mock for model unit tests, because if we want to unit test models we have to
run through bookshelf, because the whole model layer depends on bookshelf e.g. events
- for now we simply use the real database
- we could use the sqlite3 memory mode, but that would mean every unit test runs on sqlite3
- something to consider for later e.g. run unit tests on one matrix
- run the rest on another matrix for sqlite + mysql
closes#9675
- with dynamic routing we have introduced a breaking change, which we have overseen
- Ghost does not return absolute urls, that's why the clients need to concat the blog url and the resource url
- with 1.24.0 Ghost returned resource urls including the subdirectory
- this caused trouble for e.g. zapier or the preview feature in the admin client
- revert breaking change and ensure we only expose resource urls without subdirectory
refs https://github.com/TryGhost/Ghost/issues/9623
- add `oembed-parser` module for checking provider availability for a url and fetching data from the provider
- require it in the `overrides.js` file before the general Promise override so that the `promise-wrt` sub-dependency doesn't attempt to extend the Bluebird promise implementation
- add `/oembed` authenticated endpoint
- takes `?url=` query parameter to match against known providers
- adds safeguard against oembed-parser's providers list not recognising http+https and www+non-www
- responds with `ValidationError` if no provider is found
- responds with oembed response from matched provider's oembed endpoint if match is found
no issue
- was introduced with dynamic routing beta: https://github.com/TryGhost/Ghost/releases/tag/1.24.0
- the slug param wasn't forwarded correctly
- you were not able to render a custom tag or author template e.g. `tag-news.hbs`
refs #9681
- we already had a protection against these situations when serving the site (theme)
- it can happen that we have to initialise the express engine in the error handler in case the first request to /ghost produces an error (e.g. 503)
- otherwise the underlying error message is hidden and Ghost doesn't render the error html template correctly
closes#9674
- the collection router had a hardcoded default context "home"
- this is wrong
- the context array get's automatically filled for the collection
- if you are serving a page e.g. /page/2/ -> it's "paged"
- if you are serving / -> it's "home"
- same for {{body_class}}, it outputs "home-template" on "/"
- this is the same behaviour as in 1.23.x
no issue
- reverse must happen once in the constructor
- otherwise we reverse the array on each request
- Ghost would randomly pick the first and then the second template
no issue
- from now on: you have to manually reconfigure your slack hook after importing your data
- we were running into trouble that Ghost had import slack hooks, because it can happen very fast
that you are importing someone's slack hook
no issue
- there was a timing bug in Ghost
- we do operations in parallel on bootstrap
- 1) we fetch the resources as early as possible
- 2) we do all the rest (express bootstrapping, theme loading, router registration) etc.
- it can happen that (2) happens too slow and ends in the situation that the queue, which is responsible
to handle both parallel actions, does not wait for the routers and closes the event
- this is a short term fix
- i need to reconsider if there is a better long term fix
refs #9601
### Dynamic Routing
This is the beta version of dynamic routing.
- we had a initial implementation of "channels" available in the codebase
- we have removed and moved this implementation
- there is now a centralised place for dynamic routing - server/services/routing
- each routing component is represented by a router type e.g. collections, routes, static pages, taxonomies, rss, preview of posts
- keep as much as possible logic of routing helpers, middlewares and controllers
- ensure test coverage
- connect all the things together
- yaml file + validation
- routing + routers
- url service
- sitemaps
- url access
- deeper implementation of yaml validations
- e.g. hard require slashes
- ensure routing hierarchy/order
- e.g. you enable the subscriber app
- you have a custom static page, which lives under the same slug /subscribe
- static pages are stronger than apps
- e.g. the first collection owns the post it has filtered
- a post cannot live in two collections
- ensure apps are still working and hook into the routers layer (or better said: and register in the routing service)
- put as much as possible comments to the code base for better understanding
- ensure a clean debug log
- ensure we can unmount routes
- e.g. you have a collection permalink of /:slug/ represented by {globals.permalink}
- and you change the permalink in the admin to dated permalink
- the express route get's refreshed from /:slug/ to /:year/:month/:day/:slug/
- unmount without server restart, yey
- ensure we are backwards compatible
- e.g. render home.hbs for collection index if collection route is /
- ensure you can access your configured permalink from the settings table with {globals.permalink}
### Render 503 if url service did not finish
- return 503 if the url service has not finished generating the resource urls
### Rewrite sitemaps
- we have rewritten the sitemaps "service", because the url generator does no longer happen on runtime
- we generate all urls on bootstrap
- the sitemaps service will consume created resource and router urls
- these urls will be shown on the xml pages
- we listen on url events
- we listen on router events
- we no longer have to fetch the resources, which is nice
- the urlservice pre-fetches resources and emits their urls
- the urlservice is the only component who knows which urls are valid
- i made some ES6 adaptions
- we keep the caching logic -> only regenerate xml if there is a change
- updated tests
- checked test coverage (100%)
### Re-work usage of Url utility
- replace all usages of `urlService.utils.urlFor` by `urlService.getByResourceId`
- only for resources e.g. post, author, tag
- this is important, because with dynamic routing we no longer create static urls based on the settings permalink on runtime
- adapt url utility
- adapt tests
no issue
- the dot notation only works if you install a single lodash dependency e.g. `yarn install lodash.get`
- otherwise we have to use `lodash/get`
no issue
- discovered while coding
- the value was always false, because we've tried to read the value from the config object
- the value lives in the database and is accessible via the labs service
no issue
- we sanitise any incoming slug on the model layer e.g uppercase -> lowercase
- and when importing e.g. an uppercase slug, the importer was trying to compare the uppercase slug with the sanitised slug
closes https://github.com/TryGhost/Ghost/issues/9620
- adjust the `deduplicateSubDir` function's regex to only match duplicate subdirectories when the `url` is only a path rather than full url or the duplicate match starts with a `/`
closesTryGhost/Support#426
refs TryGhost/gscan#106
needs TryGhost/gscan#107
GScan can return errors, which was not handled in our theme validator and caused Ghost to crash completely. GScan will now return an Ignition error when its not able to read the `.zip` file.
e. g.: `{"errors":[{"message":"Failed to read zip file","context":"tife.zip","errorType":"ValidationError","errorDetails":"invalid relative path: ../tife/"}]}`
refs https://github.com/TryGhost/Ghost/issues/9505
- remove requirement for the `enableDeveloperExperiments` flag to be able to use Koenig
- it's now possible to enable as a standard Labs beta feature
refs https://github.com/TryGhost/Ghost/issues/9505
- updates mobiledoc converter's `render` method to accept a `version` argument
- `1` === Ghost 1.0's markdown-only renderer output
- `2` === Koenig's full mobiledoc renderer output
- switch between mobiledoc renderer versions in Post model's `onSaving` hook
- version 1 by default
- version 2 if Koenig is enabled (currently behind dev experiments config + labs flag)
- version 2 if the post's mobiledoc is not compatible with the markdown-only renderer
- "version 2" full-Koenig mobiledoc renderer output
- wraps content in a `.kg-post` div
- removes wrapper around markdown and html card output
- adds classes to image card output including selected image size/style
- standardises es6 usage across mobiledoc related files
refs #9601
- replace raw knex queries by Bookshelf queries
- optimise lot's of test setups, so we don't experience a massive slow down in the test run
- this has troubled in the past e.g. with normalisation, any custom model logic - the test env always had to simulate things
- there are for sure thousands things which can be optimised now, but because of time, we do them step by step
- this is especially important for the url service (https://github.com/TryGhost/Ghost/issues/9601), because we have to ensure that inserting/updating/removing resources will trigger model events
`grunt test-all` with SQLite finishes in 2,5-3min. (on master: 1-2min)
`grunt test-all` with MySQL finishes in 4min. (on master: 3min)
**NOTE: We want to move as much as possible routing and integration tests to unit tests. This will speed up the test run again.** See #9342. But we need to find time for that. Any help is welcome!
refs #9601
- while i was testing different collections and different filters, i somehow thought that the default
collection does not contain featured posts 😀🙊
- this is wrong (!!!!)
- the url service is not yet connected
- so: this is not a bug
refs https://github.com/TryGhost/Ghost/pull/9592
- we add bookshelf-relations step by step if we need it
- with https://github.com/TryGhost/Ghost/pull/9592 we have rewritten the test env to use Bookshelf
- this is important for our new url service
- because the service is listening on model updates and updates the urls based on the model events
- so with moving to Bookshelf, we need any easy way to add relations
- the test env inserts test fixtures
- it adds permissions and each permission get's roles attached
- `models.Permission.add({roles: [...]})
no issue
- replaced token creation by `lib.common.security`
- added unit tests for adding invites
- allow a different invite status for internal access
refs https://github.com/TryGhost/Team/issues/65
- it's easier for the architecture if we read the setting files synchronously,
because the dynamic routing component is part of the express bootstrap and
the whole routing bootstrap is synchronously
- for now: we only read one file anyway
- it's for now easier to read the file synchronously, then i don't have to change
any existing express bootstrap architecture
refs https://github.com/TryGhost/Team/issues/65
- this is just the first optimisation regarding relative/absolute urls
- the full strike will happen when i start with the url utility re-write
- for now: there will be only one subscriber of url events -> the sitemaps service
- the sitemaps service outputs absolute urls
- we don't want to receive an url event and ask the url service again to get an absolute version of the url
closes#9569
- Removed the `<1 min read` time clause, effectively making `1 min read` the minimum reading time
- Removed the `seconds` option for i18n strings, which contained the less than one minute display string
- Kept the other i18n string options the same
- Amended and improved tests for new functionality
refs #9584
- object by reference 🎡
- we modify the object later on
- this has resulted in a wrong context output in the admin client
- e.g. we've output an updated user reference
refs https://github.com/TryGhost/Team/issues/65
- currently we generate a relative resource url
- if you configure a subdirectory, the urls have to respect that
- e.g. you configure `localhost:2368/blog`, your url results in e.g. `/blog/my-post/`
- this is not yet a critical bug, because the url service is not connected yet
- @TODO: consider absolute vs. relative urls in the url service
no issue
- removed the `routeKeywords` property from the config and used hard coded keywords.
- removed `routeKeywords` from public configuration API endpoint, as it's no longer used in the Admin.
refs https://github.com/TryGhost/Team/issues/65
We are currently work on dynamic routing (aka channels).
An important piece of this feature is the url service, which always knows the url of a resource at any time.
Resources can belong to collections or taxonomies, which can be defined in a [routing yaml file](https://github.com/TryGhost/Ghost/issues/9528). We are currently shipping portions, which will at end form the full dynamic routing feature.
### Key Notes
- each routing type (collections, taxonomies, static pages) is registered in order - depending on the yaml routes file configuration
- static pages are an internal concept - they sit at the end of the subscriber queue
- we make use of a temporary [`Channels2`](https://github.com/TryGhost/Ghost/pull/9550/files#diff-9e7251409844521470c9829013cd1563) file, which simulates the current static routing in Ghost (this file will be modified, removed or whatever - this is one of the next steps)
- two way binding: you can ask for a resource url based on the resource id, you can ask for the resource based on the url
- in theory it's possible that multiple resources generate the same url: we don't handle this with collision (because this is error prone), we handle this with the order of serving content. if you ask the service for a resource, which lives behind e.g. /test/, you will get the resource which is served
- loose error handling -> log errors and handle instead of throw error and do nothing (we log the errors with a specific code, so we can react in case there is a bug)
- the url services fetches all resources on bootstrap. we only fetch and keep a reduced set of attributes (basically the main body of a resource)
- the bootstrap time will decrease a very little (depending on the amount of resources you have in your database)
- we still offer the option to disable url preloading (in your config `disableUrlPreload: true`) - this option will be removed as soon as the url service is connected. You can disable the service in case you encounter a problem
- **the url service is not yet connected, we will connect the service step by step. The first version should be released to pre-catch bugs. The next version will add 503 handling if the url service is not ready and it will consume urls for resources.**
----
- the url service generates urls based on resources (posts, pages, users, tags)
- the url service keeps track of resource changes
- the url service keeps track of resource removal/insert
- the architecture:
- each routing type is represented by a url generator
- a routing type is a collection, a taxonomiy or static pages
- a queue which ensures that urls are unique and can be owned by one url generator
- the hierarchy of registration defines that
- we query knex, because bookshelf is too slow
- removed old url service files + logic
- added temp channels alternative (Channels2) -> this file will look different soon, it's for now the temporary connector to the url service. Also the name of the file is not optimal, but that is not really important right now.
no issue
- support more cases
- e.g. multiple where matches
- @TODO
- take time to look for NPM module, which does this already
- test sqlite3 :memory: mode again
closes#9528
These code changes introduce a YAML parser which will load and parse YAML files from the `/content/settings` directory. There are three major parts involved:
1. `ensure-settings.js`: this fn takes care that on bootstrap, the supported files are present in the `/content/settings` directory. If the files are not present, they get copied back from our default files. The default files to copy from are located in `core/server/services/settings`.
2. `loader.js`: the settings loader reads the requested `yaml` file from the disk and passes it to the yaml parser, which returns a `json` object of the file. The settings loader throws an error, if the file is not accessible, e. g. because of permission errors.
3. `yaml-parser`: gets passed a `yaml` file and returns a `json` object. If the file is not parseable, it returns a clear error that contains the information, what and where the parsing error occurred (e. g. line number and reason).
- added a `get()` fn to settings services, that returns the settings object that's asked for. e. g. `settings.get('routes').then(()...` will return the `routes` settings.
- added a `getAll()` fn to settings services, that returns all available settings in an object. The object looks like: `{routes: {routes: {}, collections: {}, resources: {}}, globals: {value: {}}`, assuming that we have to supported settings `routes` and `globals`.
Further additions:
- config `contentPath` for `settings`
- config overrides for default `yaml` files location in `/core/server/services/settings`
**Important**: These code changes are in preparation for Dynamic Routing and not yet used. The process of copying the supported `yaml` files (in this first step, the `routes.yaml` file) is not yet activated.
no issue
- required for model events
- otherwise you won't receive a full data set
- in worst case you have to re-fetch the post
- required for the url service
- the url service always needs relations (authors,tags) to be able to generate the url properly
@IMPORTANT
- no API change, we still return what you are asking for
- we first edit/add the resource
- then we fetch the data with the API options
- @TODO: this can be optimised and will improve performance
picking/selecting it from the insert/update response
- this is an internal change
closes#9547
- you setup a blog with the following owner:
- email: test@ghost.org
- name: test
- slug: test
- now you import a JSON db file, which holds the exact same owner
- this owner won't be imported, because it's a duplicate
- but the slug is different (!)
- the importer tries to find a matching existing user, but won't find anything
- the importer then send an empty authors array `post.authors=[]` into the model layer
- this is not allowed -> this would mean, you are actively trying to unset all authors
no issue
- if multiple queries run in a transaction, the model events are triggered before the txn finished
- if the txn rolls back, the events are anyway emitted
- the events are triggered too early
- solution:
- `emitChange` needs to detect that a transaction is happening
- it listens on a txn event to determine if events should be triggered
no issue
- if you delete all content, we expect two events
- `post.deleted` and `post.unpublished`
- `post.unpublished` was never triggered, because the api implementation made use of `collection.invoke(`destroy`)`
- what happened?
- you fetch all posts (columns:id)
- you destroy the post (only id column is available)
- the model events are triggered
- but you have no access to a default set of data
- the result is that the event handler can't even tell if this is a post or a page
- added a proper test to ensure which events are triggered
