Rewrite url service (#9550)
refs https://github.com/TryGhost/Team/issues/65
We are currently work on dynamic routing (aka channels).
An important piece of this feature is the url service, which always knows the url of a resource at any time.
Resources can belong to collections or taxonomies, which can be defined in a [routing yaml file](https://github.com/TryGhost/Ghost/issues/9528). We are currently shipping portions, which will at end form the full dynamic routing feature.
### Key Notes
- each routing type (collections, taxonomies, static pages) is registered in order - depending on the yaml routes file configuration
- static pages are an internal concept - they sit at the end of the subscriber queue
- we make use of a temporary [`Channels2`](https://github.com/TryGhost/Ghost/pull/9550/files#diff-9e7251409844521470c9829013cd1563) file, which simulates the current static routing in Ghost (this file will be modified, removed or whatever - this is one of the next steps)
- two way binding: you can ask for a resource url based on the resource id, you can ask for the resource based on the url
- in theory it's possible that multiple resources generate the same url: we don't handle this with collision (because this is error prone), we handle this with the order of serving content. if you ask the service for a resource, which lives behind e.g. /test/, you will get the resource which is served
- loose error handling -> log errors and handle instead of throw error and do nothing (we log the errors with a specific code, so we can react in case there is a bug)
- the url services fetches all resources on bootstrap. we only fetch and keep a reduced set of attributes (basically the main body of a resource)
- the bootstrap time will decrease a very little (depending on the amount of resources you have in your database)
- we still offer the option to disable url preloading (in your config `disableUrlPreload: true`) - this option will be removed as soon as the url service is connected. You can disable the service in case you encounter a problem
- **the url service is not yet connected, we will connect the service step by step. The first version should be released to pre-catch bugs. The next version will add 503 handling if the url service is not ready and it will consume urls for resources.**
----
- the url service generates urls based on resources (posts, pages, users, tags)
- the url service keeps track of resource changes
- the url service keeps track of resource removal/insert
- the architecture:
- each routing type is represented by a url generator
- a routing type is a collection, a taxonomiy or static pages
- a queue which ensures that urls are unique and can be owned by one url generator
- the hierarchy of registration defines that
- we query knex, because bookshelf is too slow
- removed old url service files + logic
- added temp channels alternative (Channels2) -> this file will look different soon, it's for now the temporary connector to the url service. Also the name of the file is not optimal, but that is not really important right now.
2018-04-17 12:29:04 +03:00
|
|
|
'use strict';
|
|
|
|
|
|
|
|
const _ = require('lodash'),
|
|
|
|
Promise = require('bluebird'),
|
|
|
|
moment = require('moment-timezone'),
|
|
|
|
jsonpath = require('jsonpath'),
|
|
|
|
debug = require('ghost-ignition').debug('services:url:generator'),
|
2018-04-17 16:36:51 +03:00
|
|
|
localUtils = require('./utils'),
|
Rewrite url service (#9550)
refs https://github.com/TryGhost/Team/issues/65
We are currently work on dynamic routing (aka channels).
An important piece of this feature is the url service, which always knows the url of a resource at any time.
Resources can belong to collections or taxonomies, which can be defined in a [routing yaml file](https://github.com/TryGhost/Ghost/issues/9528). We are currently shipping portions, which will at end form the full dynamic routing feature.
### Key Notes
- each routing type (collections, taxonomies, static pages) is registered in order - depending on the yaml routes file configuration
- static pages are an internal concept - they sit at the end of the subscriber queue
- we make use of a temporary [`Channels2`](https://github.com/TryGhost/Ghost/pull/9550/files#diff-9e7251409844521470c9829013cd1563) file, which simulates the current static routing in Ghost (this file will be modified, removed or whatever - this is one of the next steps)
- two way binding: you can ask for a resource url based on the resource id, you can ask for the resource based on the url
- in theory it's possible that multiple resources generate the same url: we don't handle this with collision (because this is error prone), we handle this with the order of serving content. if you ask the service for a resource, which lives behind e.g. /test/, you will get the resource which is served
- loose error handling -> log errors and handle instead of throw error and do nothing (we log the errors with a specific code, so we can react in case there is a bug)
- the url services fetches all resources on bootstrap. we only fetch and keep a reduced set of attributes (basically the main body of a resource)
- the bootstrap time will decrease a very little (depending on the amount of resources you have in your database)
- we still offer the option to disable url preloading (in your config `disableUrlPreload: true`) - this option will be removed as soon as the url service is connected. You can disable the service in case you encounter a problem
- **the url service is not yet connected, we will connect the service step by step. The first version should be released to pre-catch bugs. The next version will add 503 handling if the url service is not ready and it will consume urls for resources.**
----
- the url service generates urls based on resources (posts, pages, users, tags)
- the url service keeps track of resource changes
- the url service keeps track of resource removal/insert
- the architecture:
- each routing type is represented by a url generator
- a routing type is a collection, a taxonomiy or static pages
- a queue which ensures that urls are unique and can be owned by one url generator
- the hierarchy of registration defines that
- we query knex, because bookshelf is too slow
- removed old url service files + logic
- added temp channels alternative (Channels2) -> this file will look different soon, it's for now the temporary connector to the url service. Also the name of the file is not optimal, but that is not really important right now.
2018-04-17 12:29:04 +03:00
|
|
|
settingsCache = require('../settings/cache'),
|
|
|
|
/**
|
|
|
|
* @TODO: This is a fake version of the upcoming GQL tool.
|
|
|
|
* GQL will offer a tool to match a JSON against a filter.
|
|
|
|
*/
|
|
|
|
transformFilter = (filter) => {
|
|
|
|
filter = '$[?(' + filter + ')]';
|
|
|
|
filter = filter.replace(/(\w+):(\w+)/g, '@.$1 == "$2"');
|
|
|
|
filter = filter.replace(/"true"/g, 'true');
|
|
|
|
filter = filter.replace(/"false"/g, 'false');
|
|
|
|
filter = filter.replace(/"0"/g, '0');
|
|
|
|
filter = filter.replace(/"1"/g, '1');
|
|
|
|
filter = filter.replace(/\+/g, ' && ');
|
|
|
|
return filter;
|
|
|
|
};
|
|
|
|
|
|
|
|
class UrlGenerator {
|
|
|
|
constructor(routingType, queue, resources, urls, position) {
|
|
|
|
this.routingType = routingType;
|
|
|
|
this.queue = queue;
|
|
|
|
this.urls = urls;
|
|
|
|
this.resources = resources;
|
|
|
|
this.uid = position;
|
|
|
|
|
|
|
|
debug('constructor', this.toString());
|
|
|
|
|
|
|
|
// CASE: channels can define custom filters, but not required.
|
|
|
|
if (this.routingType.getFilter()) {
|
|
|
|
this.filter = transformFilter(this.routingType.getFilter());
|
|
|
|
debug('filter', this.filter);
|
|
|
|
}
|
|
|
|
|
|
|
|
this._listeners();
|
|
|
|
}
|
|
|
|
|
|
|
|
_listeners() {
|
|
|
|
/**
|
|
|
|
* @NOTE: currently only used if the permalink setting changes and it's used for this url generator.
|
|
|
|
* @TODO: remove in Ghost 2.0
|
|
|
|
*/
|
|
|
|
this.routingType.addListener('updated', () => {
|
|
|
|
const myResources = this.urls.getByGeneratorId(this.uid);
|
|
|
|
|
|
|
|
myResources.forEach((object) => {
|
|
|
|
this.urls.removeResourceId(object.resource.data.id);
|
|
|
|
object.resource.release();
|
|
|
|
this._try(object.resource);
|
|
|
|
});
|
|
|
|
});
|
|
|
|
|
|
|
|
/**
|
|
|
|
* Listen on two events:
|
|
|
|
*
|
|
|
|
* - init: bootstrap or url reset
|
|
|
|
* - added: resource was added
|
|
|
|
*/
|
|
|
|
this.queue.register({
|
|
|
|
event: 'init',
|
|
|
|
tolerance: 100
|
|
|
|
}, this._onInit.bind(this));
|
|
|
|
|
|
|
|
// @TODO: listen on added event per type (post optimisation)
|
|
|
|
this.queue.register({
|
|
|
|
event: 'added'
|
|
|
|
}, this._onAdded.bind(this));
|
|
|
|
}
|
|
|
|
|
|
|
|
_onInit() {
|
|
|
|
debug('_onInit', this.toString());
|
|
|
|
|
|
|
|
// @NOTE: get the resources of my type e.g. posts.
|
|
|
|
const resources = this.resources.getAllByType(this.routingType.getType());
|
|
|
|
|
|
|
|
_.each(resources, (resource) => {
|
|
|
|
this._try(resource);
|
|
|
|
});
|
|
|
|
}
|
|
|
|
|
|
|
|
_onAdded(event) {
|
|
|
|
debug('onAdded', this.toString());
|
|
|
|
|
|
|
|
// CASE: you are type "pages", but the incoming type is "users"
|
|
|
|
if (event.type !== this.routingType.getType()) {
|
|
|
|
return Promise.resolve();
|
|
|
|
}
|
|
|
|
|
|
|
|
const resource = this.resources.getByIdAndType(event.type, event.id);
|
|
|
|
|
|
|
|
this._try(resource);
|
|
|
|
}
|
|
|
|
|
|
|
|
_try(resource) {
|
|
|
|
/**
|
|
|
|
* CASE: another url generator has taken this resource already.
|
|
|
|
*
|
|
|
|
* We have to remember that, because each url generator can generate a different url
|
|
|
|
* for a resource. So we can't directly check `this.urls.getUrl(url)`.
|
|
|
|
*/
|
|
|
|
if (resource.isReserved()) {
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
|
|
|
const url = this._generateUrl(resource);
|
|
|
|
|
|
|
|
// CASE 1: route has no custom filter, it will own the resource for sure
|
|
|
|
// CASE 2: find out if my filter matches the resource
|
|
|
|
if (!this.filter) {
|
|
|
|
this.urls.add({
|
|
|
|
url: url,
|
|
|
|
generatorId: this.uid,
|
|
|
|
resource: resource
|
|
|
|
});
|
|
|
|
|
|
|
|
resource.reserve();
|
|
|
|
this._resourceListeners(resource);
|
|
|
|
return true;
|
|
|
|
} else if (jsonpath.query(resource, this.filter).length) {
|
|
|
|
this.urls.add({
|
|
|
|
url: url,
|
|
|
|
generatorId: this.uid,
|
|
|
|
resource: resource
|
|
|
|
});
|
|
|
|
|
|
|
|
resource.reserve();
|
|
|
|
this._resourceListeners(resource);
|
|
|
|
return true;
|
|
|
|
} else {
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-04-17 16:36:51 +03:00
|
|
|
/**
|
|
|
|
* We currently generate relative urls.
|
|
|
|
*
|
|
|
|
* @TODO: reconsider? e.g. sitemaps would receive a relative url, but we show absolute urls
|
|
|
|
*/
|
Rewrite url service (#9550)
refs https://github.com/TryGhost/Team/issues/65
We are currently work on dynamic routing (aka channels).
An important piece of this feature is the url service, which always knows the url of a resource at any time.
Resources can belong to collections or taxonomies, which can be defined in a [routing yaml file](https://github.com/TryGhost/Ghost/issues/9528). We are currently shipping portions, which will at end form the full dynamic routing feature.
### Key Notes
- each routing type (collections, taxonomies, static pages) is registered in order - depending on the yaml routes file configuration
- static pages are an internal concept - they sit at the end of the subscriber queue
- we make use of a temporary [`Channels2`](https://github.com/TryGhost/Ghost/pull/9550/files#diff-9e7251409844521470c9829013cd1563) file, which simulates the current static routing in Ghost (this file will be modified, removed or whatever - this is one of the next steps)
- two way binding: you can ask for a resource url based on the resource id, you can ask for the resource based on the url
- in theory it's possible that multiple resources generate the same url: we don't handle this with collision (because this is error prone), we handle this with the order of serving content. if you ask the service for a resource, which lives behind e.g. /test/, you will get the resource which is served
- loose error handling -> log errors and handle instead of throw error and do nothing (we log the errors with a specific code, so we can react in case there is a bug)
- the url services fetches all resources on bootstrap. we only fetch and keep a reduced set of attributes (basically the main body of a resource)
- the bootstrap time will decrease a very little (depending on the amount of resources you have in your database)
- we still offer the option to disable url preloading (in your config `disableUrlPreload: true`) - this option will be removed as soon as the url service is connected. You can disable the service in case you encounter a problem
- **the url service is not yet connected, we will connect the service step by step. The first version should be released to pre-catch bugs. The next version will add 503 handling if the url service is not ready and it will consume urls for resources.**
----
- the url service generates urls based on resources (posts, pages, users, tags)
- the url service keeps track of resource changes
- the url service keeps track of resource removal/insert
- the architecture:
- each routing type is represented by a url generator
- a routing type is a collection, a taxonomiy or static pages
- a queue which ensures that urls are unique and can be owned by one url generator
- the hierarchy of registration defines that
- we query knex, because bookshelf is too slow
- removed old url service files + logic
- added temp channels alternative (Channels2) -> this file will look different soon, it's for now the temporary connector to the url service. Also the name of the file is not optimal, but that is not really important right now.
2018-04-17 12:29:04 +03:00
|
|
|
_generateUrl(resource) {
|
2018-04-17 16:36:51 +03:00
|
|
|
let url = this.routingType.getPermalinks().getValue();
|
|
|
|
url = this._replacePermalink(url, resource);
|
|
|
|
|
|
|
|
return localUtils.createUrl(url, false, false);
|
Rewrite url service (#9550)
refs https://github.com/TryGhost/Team/issues/65
We are currently work on dynamic routing (aka channels).
An important piece of this feature is the url service, which always knows the url of a resource at any time.
Resources can belong to collections or taxonomies, which can be defined in a [routing yaml file](https://github.com/TryGhost/Ghost/issues/9528). We are currently shipping portions, which will at end form the full dynamic routing feature.
### Key Notes
- each routing type (collections, taxonomies, static pages) is registered in order - depending on the yaml routes file configuration
- static pages are an internal concept - they sit at the end of the subscriber queue
- we make use of a temporary [`Channels2`](https://github.com/TryGhost/Ghost/pull/9550/files#diff-9e7251409844521470c9829013cd1563) file, which simulates the current static routing in Ghost (this file will be modified, removed or whatever - this is one of the next steps)
- two way binding: you can ask for a resource url based on the resource id, you can ask for the resource based on the url
- in theory it's possible that multiple resources generate the same url: we don't handle this with collision (because this is error prone), we handle this with the order of serving content. if you ask the service for a resource, which lives behind e.g. /test/, you will get the resource which is served
- loose error handling -> log errors and handle instead of throw error and do nothing (we log the errors with a specific code, so we can react in case there is a bug)
- the url services fetches all resources on bootstrap. we only fetch and keep a reduced set of attributes (basically the main body of a resource)
- the bootstrap time will decrease a very little (depending on the amount of resources you have in your database)
- we still offer the option to disable url preloading (in your config `disableUrlPreload: true`) - this option will be removed as soon as the url service is connected. You can disable the service in case you encounter a problem
- **the url service is not yet connected, we will connect the service step by step. The first version should be released to pre-catch bugs. The next version will add 503 handling if the url service is not ready and it will consume urls for resources.**
----
- the url service generates urls based on resources (posts, pages, users, tags)
- the url service keeps track of resource changes
- the url service keeps track of resource removal/insert
- the architecture:
- each routing type is represented by a url generator
- a routing type is a collection, a taxonomiy or static pages
- a queue which ensures that urls are unique and can be owned by one url generator
- the hierarchy of registration defines that
- we query knex, because bookshelf is too slow
- removed old url service files + logic
- added temp channels alternative (Channels2) -> this file will look different soon, it's for now the temporary connector to the url service. Also the name of the file is not optimal, but that is not really important right now.
2018-04-17 12:29:04 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* @TODO:
|
|
|
|
* This is a copy of `replacePermalink` of our url utility, see ./utils.
|
|
|
|
* But it has modifications, because the whole url utility doesn't work anymore.
|
|
|
|
* We will rewrite some of the functions in the utility.
|
|
|
|
*/
|
|
|
|
_replacePermalink(url, resource) {
|
|
|
|
var output = url,
|
2018-04-17 13:05:27 +03:00
|
|
|
primaryTagFallback = 'all',
|
Rewrite url service (#9550)
refs https://github.com/TryGhost/Team/issues/65
We are currently work on dynamic routing (aka channels).
An important piece of this feature is the url service, which always knows the url of a resource at any time.
Resources can belong to collections or taxonomies, which can be defined in a [routing yaml file](https://github.com/TryGhost/Ghost/issues/9528). We are currently shipping portions, which will at end form the full dynamic routing feature.
### Key Notes
- each routing type (collections, taxonomies, static pages) is registered in order - depending on the yaml routes file configuration
- static pages are an internal concept - they sit at the end of the subscriber queue
- we make use of a temporary [`Channels2`](https://github.com/TryGhost/Ghost/pull/9550/files#diff-9e7251409844521470c9829013cd1563) file, which simulates the current static routing in Ghost (this file will be modified, removed or whatever - this is one of the next steps)
- two way binding: you can ask for a resource url based on the resource id, you can ask for the resource based on the url
- in theory it's possible that multiple resources generate the same url: we don't handle this with collision (because this is error prone), we handle this with the order of serving content. if you ask the service for a resource, which lives behind e.g. /test/, you will get the resource which is served
- loose error handling -> log errors and handle instead of throw error and do nothing (we log the errors with a specific code, so we can react in case there is a bug)
- the url services fetches all resources on bootstrap. we only fetch and keep a reduced set of attributes (basically the main body of a resource)
- the bootstrap time will decrease a very little (depending on the amount of resources you have in your database)
- we still offer the option to disable url preloading (in your config `disableUrlPreload: true`) - this option will be removed as soon as the url service is connected. You can disable the service in case you encounter a problem
- **the url service is not yet connected, we will connect the service step by step. The first version should be released to pre-catch bugs. The next version will add 503 handling if the url service is not ready and it will consume urls for resources.**
----
- the url service generates urls based on resources (posts, pages, users, tags)
- the url service keeps track of resource changes
- the url service keeps track of resource removal/insert
- the architecture:
- each routing type is represented by a url generator
- a routing type is a collection, a taxonomiy or static pages
- a queue which ensures that urls are unique and can be owned by one url generator
- the hierarchy of registration defines that
- we query knex, because bookshelf is too slow
- removed old url service files + logic
- added temp channels alternative (Channels2) -> this file will look different soon, it's for now the temporary connector to the url service. Also the name of the file is not optimal, but that is not really important right now.
2018-04-17 12:29:04 +03:00
|
|
|
publishedAtMoment = moment.tz(resource.data.published_at || Date.now(), settingsCache.get('active_timezone')),
|
|
|
|
permalink = {
|
|
|
|
year: function () {
|
|
|
|
return publishedAtMoment.format('YYYY');
|
|
|
|
},
|
|
|
|
month: function () {
|
|
|
|
return publishedAtMoment.format('MM');
|
|
|
|
},
|
|
|
|
day: function () {
|
|
|
|
return publishedAtMoment.format('DD');
|
|
|
|
},
|
|
|
|
author: function () {
|
|
|
|
return resource.data.primary_author.slug;
|
|
|
|
},
|
|
|
|
primary_author: function () {
|
|
|
|
return resource.data.primary_author ? resource.data.primary_author.slug : primaryTagFallback;
|
|
|
|
},
|
|
|
|
primary_tag: function () {
|
|
|
|
return resource.data.primary_tag ? resource.data.primary_tag.slug : primaryTagFallback;
|
|
|
|
},
|
|
|
|
slug: function () {
|
|
|
|
return resource.data.slug;
|
|
|
|
},
|
|
|
|
id: function () {
|
|
|
|
return resource.data.id;
|
|
|
|
}
|
|
|
|
};
|
|
|
|
|
|
|
|
// replace tags like :slug or :year with actual values
|
|
|
|
output = output.replace(/(:[a-z_]+)/g, function (match) {
|
|
|
|
if (_.has(permalink, match.substr(1))) {
|
|
|
|
return permalink[match.substr(1)]();
|
|
|
|
}
|
|
|
|
});
|
|
|
|
|
|
|
|
return output;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* I want to know if my resources changes.
|
|
|
|
* Register events of resource.
|
|
|
|
*/
|
|
|
|
_resourceListeners(resource) {
|
|
|
|
const onUpdate = (updatedResource) => {
|
|
|
|
// 1. remove old resource
|
|
|
|
this.urls.removeResourceId(updatedResource.data.id);
|
|
|
|
|
|
|
|
// 2. free resource, the url <-> resource connection no longer exists
|
|
|
|
updatedResource.release();
|
|
|
|
|
|
|
|
// 3. try to own the resource again
|
|
|
|
// Imagine you change `featured` to true and your filter excludes featured posts.
|
|
|
|
const isMine = this._try(updatedResource);
|
|
|
|
|
|
|
|
// 4. if the resource is no longer mine, tell the others
|
|
|
|
// e.g. post -> page
|
|
|
|
// e.g. post is featured now
|
|
|
|
if (!isMine) {
|
|
|
|
debug('free, this is not mine anymore', updatedResource.data.id);
|
|
|
|
|
|
|
|
this.queue.start({
|
|
|
|
event: 'added',
|
|
|
|
action: 'added:' + resource.data.id,
|
|
|
|
eventData: {
|
|
|
|
id: resource.data.id,
|
|
|
|
type: this.routingType.getType()
|
|
|
|
}
|
|
|
|
});
|
|
|
|
}
|
|
|
|
};
|
|
|
|
|
|
|
|
const onRemoved = (removedResource) => {
|
|
|
|
this.urls.removeResourceId(removedResource.data.id);
|
|
|
|
removedResource.release();
|
|
|
|
};
|
|
|
|
|
|
|
|
resource.removeAllListeners();
|
|
|
|
resource.addListener('updated', onUpdate.bind(this));
|
|
|
|
resource.addListener('removed', onRemoved.bind(this));
|
|
|
|
}
|
|
|
|
|
|
|
|
getUrls() {
|
|
|
|
return this.urls.getByGeneratorId(this.uid);
|
|
|
|
}
|
|
|
|
|
|
|
|
toString() {
|
|
|
|
return this.routingType.toString();
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
module.exports = UrlGenerator;
|