closes#8479
- removes `markdown` field from schema
- removes `legacyMarkdown` converter
- updates tests to work with `mobiledoc` field instead of `markdown` and adapt for mobiledoc HTML output where necessary
refs #5422
- we can support null titles after this PR if we want
- user model: fix getAuthorRole
- user model: support adding roles by name
- we support this for roles as well, this makes it easier when importing related user roles (because usually roles already exists in the database and the related id's are wrong e.g. roles_users)
- base model: support for null created_at or updated_at values
- post or tag slugs are always safe strings
- enable an import of a null slug, no need to crash or to cover this on import layer
- add new DataImporter logic
- uses a class inheritance mechanism to achieve an easier readability and maintenance
- schema validation (happens on model layer) was ignored
- allow to import unknown user id's (see https://github.com/TryGhost/Ghost/issues/8365)
- most of the duplication handling happens on model layer (we can use the power of unique fields and errors from the database)
- the import is splitted into three steps:
- beforeImport
--> prepares the data to import, sorts out relations (roles, tags), detects fields (for LTS)
- doImport
--> does the actual import
- afterImport
--> updates the data after successful import e.g. update all user reference fields e.g. published_by (compares the imported data with the current state of the database)
- import images: markdown can be null
- show error message when json handler can't parse file
- do not request gravatar if email is null
- return problems/warnings after successful import
- optimise warnings in importer
- do not return warnings for role duplications, no helpful information
- error handler: return context information of error
- we show the affected json entries as one line in the UI
- show warning for: detected duplicated tag
- schema validation: fix valueMustBeBoolean translation
- remove context property from json parse error
closes#8354
- i thought about transforming scheduled posts into drafts on export, but this has two disadvantages:
1. existing exports with scheduled posts won't import
2. if you schedule a post for next week and you export/import earlier, the post is back to draft
- by this we ensure that we can simply import the post back to a scheduled post
- if the published_at is already in the past, the scheduler will care and instantly publish the post
* 🙀 change database schema for images
- rename user/post/tag images
- contains all the required changes from the schema change
* Refactor helper/meta data
- rename cover to cover_image
- also rename default settings to match the pattern
- rename image to profile_image for user
- rename image to feature_image for tags/posts
* {{image}} >>> {{img_url}}
- rename
- change the functionality
- attr is required
- e.g. {{img_url feature_image}}
* gscan 1.0.0
- update yarn.lock
* Update casper reference: 1.0-changes
- see 5487b4da8d
closes#5599
If two users edit the same post, it can happen that they override each others content or post settings. With this change this won't happen anymore.
✨ Update collision for posts
- add a new bookshelf plugin to detect these changes
- use the `changed` object of bookshelf -> we don't have to create our own diff
- compare client and server updated_at field
- run editing posts in a transaction (see comments in code base)
🙀 update collision for tags
- `updateTags` for adding posts on `onCreated` - happens after the post was inserted
--> it's "okay" to attach the tags afterwards on insert
--> there is no need to add collision for inserting data
--> it's very hard to move the updateTags call to `onCreating`, because the `updateTags` function queries the database to look up the affected post
- `updateTags` while editing posts on `onSaving` - all operations run in a transactions and are rolled back if something get's rejected
- Post model edit: if we push a transaction from outside, take this one
✨ introduce options.forUpdate
- if two queries happening in a transaction we have to signalise knex/mysql that we select for an update
- otherwise the following case happens:
>> you fetch posts for an update
>> a user requests comes in and updates the post (e.g. sets title to "X")
>> you update the fetched posts, title would get overriden to the old one
use options.forUpdate and protect internal post updates: model listeners
- use a transaction for listener updates
- signalise forUpdate
- write a complex test
use options.forUpdate and protect internal post updates: scheduling
- publish endpoint runs in a transaction
- add complex test
- @TODO: right now scheduling api uses posts api, therefor we had to extend the options for api's
>> allowed to pass transactions through it
>> but these are only allowed if defined from outside {opts: [...]}
>> so i think this is fine and not dirty
>> will wait for opinions
>> alternatively we have to re-write the scheduling endpoint to use the models directly
no issue
- the UTC offset diff of the current and previous timezone must switch
- i have added more tests and more example case descriptions to understand why
refs #8275
- If the HTML field has changed, update the plaintext field
- Use html-to-text to generate a plaintext version of the HTML which retains some structure
- Add a couple of tests - although there's much to do here!
no issue
- i don't know if this never worked or has worked and something changed in bookshelf
- but this fixes: saving the content (no change for published_at) of a scheduled post within the 2minutes window
- add `beforeWrite` option to hasDateChanged helper, see comment
- use previous for `beforeWrite` operations
- add a test and fix some other small issues in the scheduler tests
refs #8258
* 🎨 change last_login to last_seen
- rename the column
- a change in Ghost-Admin is required as well
* test utils: revert export examples
* revert line breaks
no issue
- change out should.equal for // jshint ignore:line
- ensure should is the first require in every test, and ALWAYS require
- make sinon the second require, and sandbox the last thing
- ALWAYS use sandbox, futureproofs tests against contributors who don't know it
- change require formatting
refs #8032
- this was used to disable the upload image functionality in Ghost-Admin
- we no longer need this boolean, because people can add their own storage adapter
refs #8111
- Ghost returns now all (active+none active) users by default
- protect login with suspended status
- test permissions and add extra protection for suspending myself
- if a user is suspended and tries to activate himself, he won't be able to proceed the login to get a new token
refs #8093✨ Add activate theme permission
- add permission to activate themes
- update tests
- also: update tests for invites
TODO: change how the active theme setting is updated to reduce extra permissions
✨ Move theme validation to gscan
- add a new gscan validation method and use it for upload
- update activate endpoint to do validation also using gscan
- change to using SettingsModel instead of API so that we don't call validation or permissions on the settings API
- remove validation from the settings model
- remove the old validation function
- add new invalid theme message to translations & remove a bunch of theme validation related unused keys
📖 Planned changes
🚨 Tests for theme activation API endpoint
🐛 Don't allow deleting the active theme
🚫 Prevent activeTheme being set via settings API
- We want to control how this happens in future.
- We still want to store the information in settings, via the model.
- We just don't want to be able to change this info via the settings edit endpoint
🐛✨ Fix warnings for uploads & add for activations
- warnings for uploads were broken in f8b498d
- fix the response + adds tests to cover that warnings are correctly returned
- add the same response to activations + more tests
- activations now return a single theme object - the theme that was activated + any warnings
🎨 Improve how we generate theme API responses
- remove the requirement to pass in the active theme!
- move this to a specialist function, away from the list
🎨 Do not load gscan on boot
no issue
🔥 Remove unnecessary cache update
🎨 simplify updateSettingsCache()
🎨 Simplify readSettingsResult
- although this is more code, it's now much clearer what happens in the two cases
🎨 Don't use readSettingResult for edit
🎨 Simplify updateSettingsCache further
🔥 Remove now unused readSettingsResult
🎨 Change populateDefault to return all
🎨 Move the findAll call out of updateSettingsCache
🔥 Remove updateSettingsCache!!
🎨 Restructure init & finish up settingsCache
- move initialisation into settingsCache.init AT LAST
- change settingCache to use cloneDeep, so that the object can't be modified outside of the functions
- add lots of docs to settings cache
🎨 Cleanup db api endpoints
🔥 Don't populate settings in migrations
closes#8037🔥 Remove API-level default settings population
- This is a relic!
- We ALWAYS populate defaults on server start therefore this code could never run.
- This was a lot of complicated code that wasn't even needed!!
🎨 Move settings cache
- Move settings cache to be its own thing
- Update all references
- Adds TODOs for further cleanup
🎨 Create settings initialisation step
- Create new settings library, which will eventually house more code
- Unify the interface for initialising settings (will be more useful later)
- Reduce number of calls to updateSettingsCache
no issue
* ✨ Add new server start & stop events
* 🔥 Get rid of unused availableApps concept
- when we need an API endpoint for a list of apps, we'll build one 😝
* ✨ Move theme loading into a module
- move loading from API method to a module method and use as needed
- wire up read one vs read all as per LTS
- read one (the active theme) on boot, and read the rest after
- fudge validation - this isn't all that helpful
* Settings API tests need to preload themes
- this used to automatically happen as part of loading settings
- now we need to trigger this to happen specifically for this test
refs #7432🚨 database: change hard limits and field types
- we went over all schema fields and decided to decrease/increase the hard limits
- the core goal is to have more flexibility in the future
- we reconsidered string vs. text
There are 5 groups:
- small strings: 50 characters
- static strings
- status, visibility, language, role name, permission name, client name etc.
- medium strings: 191 characters
- all unique fields or fields which can be unique in the future
- slug, tokens, user name, password, tag name, email
- large strings: 1000-2000 characters
- these fields need to be very flexible
- these fields get a soft limit attached (in a different PR)
- post title, meta title, meta description, urls
- medium text: 64kb characters
- bio, settings, location, tour
- long text: 1000000000 chars
- html, amp, mobiledoc, markdown
🙄 sort_order for tests
- sort order was not set for the tests, so it was always 0
- mysql could return a different result
in my case:
- field length 156 returned the following related tags ["bacon", "kitchen"]
- field length 157 returned the following related tags ["kitchen", "kitchen"]
Change client.secret to 191
Tweak field lengths
- Add 24 char limit for ids
- Limited fields are the exact length they need
- Unified 1000 and 2000 char string classes to all be 2000
- Changed descriptions to be either 2000, except user & tag which is text 65535 as these may be used to store HTML later?!
- Updated tests
🛠 Update importer tests
- The old 001-003 tests are kind of less relevant now.
- Rather than worrying about past versions of the data structure, we should check that the importer only imports what we consider to be valid data
- I've changed the tests to treat the title-length check as a length-validation check, rather than a test for each of the old versions
🔥 Remove foreign key from subscribers.post_id
- There's no real need to have an index on this column, it just makes deleting posts hard.
- Same as created_by type columns, we can reference ids without needing keys/indexes
closes#7256
- original code changes made by @golya in https://github.com/TryGhost/Ghost/pull/7304
- refactored edit method in user model to validate an existing email address
- added test coverage for existing email update in user model spec
no issue
- sometimes the integration test for api notifications failed
- this was caused because multiple tests are using the notifications API, which has a local notification store
- we need to ensure that tests, who add notifications to this store, reset the store after it's test iteration
* 🎨🔥 do not store settings in config and make settings cache easier available
- remove remembering settings value in theme config
- if we need a cache value, we are asking the settings cache directly
- instead of settings.getSettingSync we use settings.cache.get
- added TODO:
- think about moving the settings cache out of api/settings
- we could create a folder named cache cache/settings
- this settings cache listens on model changes for settings
- decoupling
* 🔥 remove timezone from config
- no need to store in overrides config and in defaults settings
* 🎨 context object helper
- replace config.get('theme') by settings cache
* 🎨 replace config.get('theme') by settings.cache.get
* 🎨 adapt tests
* fixes from comments
refs #7489
- as we are now using a different migration approach (knex-migrator), we don't need to remember the database version anymore
- it was once used to check the state of a database and based on it we decided to migrate or not
- with knex-migrator everything depends on the migration table entries and the current ghost version you are on
- on current master the leftover usage is to add the db version when exporting the database, which can be replaced by reading the ghost version
- removing this solves also an interesting migration case with knex-migrator:
- you are on 1.0
- you update to 1.1, but 1.1 has no migrations
- the db version would remain in 1.0
- because the db version was only updated when knex migrator executed a migration
refs #7724
- we already fixed the permissions for the editor
- see 3d3101ad0e
- but as we are inside of a refactoring process, we had two fixtures.json files
- we fixed the fixtures.json in the wrong place
- now that the permissions are used, we can see failing tests
- i have added the correct permissions handling
closes#7766, refs #7579
- ensure we are using the correct brute keys
- ensure we are using req.ip as Ghost is configured with trust proxy option
- tidy up a little
no issue
- we started implementing logging and error handling in Ghost
- later we outsourced both into a module
- use the module now in Ghost
- this commit basically just removes the logging and error implementation and uses Ignition
refs #7666
Using `urlFor('home')` instead `config.get('url')` in Ghost.
When `urlFor('home', true)` returns the absolute adress of the blog as defined in the config.
Will always return a trailing `/`.
* 🛠 bookshelf tarball, bson-objectid
* 🎨 schema changes
- change increment type to string
- add a default fallback for string length 191 (to avoid adding this logic to every single column which uses an ID)
- remove uuid, because ID now represents a global resource identifier
- keep uuid for post, because we are using this as preview id
- keep uuid for clients for now - we are using this param for Ghost-Auth
* ✨ base model: generate ObjectId on creating event
- each new resource get's a auto generate ObjectId
- this logic won't work for attached models, this commit comes later
* 🎨 centralised attach method
When attaching models there are two things important two know
1. To be able to attach an ObjectId, we need to register the `onCreating` event the fetched model!This is caused by the Bookshelf design in general. On this target model we are attaching the new model.
2. We need to manually fetch the target model, because Bookshelf has a weird behaviour (which is known as a bug, see see https://github.com/tgriesser/bookshelf/issues/629). The most important property when attaching a model is `parentFk`, which is the foreign key. This can be null when fetching the model with the option `withRelated`. To ensure quality and consistency, the custom attach wrapper always fetches the target model manual. By fetching the target model (again) is a little performance decrease, but it also has advantages: we can register the event, and directly unregister the event again. So very clean code.
Important: please only use the custom attach wrapper in the future.
* 🎨 token model had overriden the onCreating function because of the created_at field
- we need to ensure that the base onCreating hook get's triggered for ALL models
- if not, they don't get an ObjectId assigned
- in this case: be smart and check if the target model has a created_at field
* 🎨 we don't have a uuid field anymore, remove the usages
- no default uuid creation in models
- i am pretty sure we have some more definitions in our tests (for example in the export json files), but that is too much work to delete them all
* 🎨 do not parse ID to Number
- we had various occurances of parsing all ID's to numbers
- we don't need this behaviour anymore
- ID is string
- i will adapt the ID validation in the next commit
* 🎨 change ID regex for validation
- we only allow: ID as ObjectId, ID as 1 and ID as me
- we need to keep ID 1, because our whole software relies on ID 1 (permissions etc)
* 🎨 owner fixture
- roles: [4] does not work anymore
- 4 means -> static id 4
- this worked in an auto increment system (not even in a system with distributed writes)
- with ObjectId we generate each ID automatically (for static and dynamic resources)
- it is possible to define all id's for static resources still, but that means we need to know which ID is already used and for consistency we have to define ObjectId's for these static resources
- so no static id's anymore, except of: id 1 for owner and id 0 for external usage (because this is required from our permission system)
- NOTE: please read through the comment in the user model
* 🎨 tests: DataGenerator and test utils
First of all: we need to ensure using ObjectId's in the tests. When don't, we can't ensure that ObjectId's work properly.
This commit brings lot's of dynamic into all the static defined id's.
In one of the next commits, i will adapt all the tests.
* 🚨 remove counter in Notification API
- no need to add a counter
- we simply generate ObjectId's (they are auto incremental as well)
- our id validator does only allow ObjectId as id,1 and me
* 🎨 extend contextUser in Base Model
- remove isNumber check, because id's are no longer numbers, except of id 0/1
- use existing isExternalUser
- support id 0/1 as string or number
* ✨ Ghost Owner has id 1
- ensure we define this id in the fixtures.json
- doesn't matter if number or string
* 🎨 functional tests adaptions
- use dynamic id's
* 🎨 fix unit tests
* 🎨 integration tests adaptions
* 🎨 change importer utils
- all our export examples (test/fixtures/exports) contain id's as numbers
- fact: but we ignore them anyway when inserting into the database, see https://github.com/TryGhost/Ghost/blob/master/core/server/data/import/utils.js#L249
- in 0e6ed957cd (diff-70f514a06347c048648be464819503c4L67) i removed parsing id's to integers
- i realised that this ^ check just existed, because the userIdToMap was an object key and object keys are always strings!
- i think this logic is a little bit complicated, but i don't want to refactor this now
- this commit ensures when trying to find the user, the id comparison works again
- i've added more documentation to understand this logic ;)
- plus i renamed an attribute to improve readability
* 🎨 Data-Generator: add more defaults to createUser
- if i use the function DataGenerator.forKnex.createUser i would like to get a full set of defaults
* 🎨 test utils: change/extend function set for functional tests
- functional tests work a bit different
- they boot Ghost and seed the database
- some functional tests have mis-used the test setup
- the test setup needs two sections: integration/unit and functional tests
- any functional test is allowed to either add more data or change data in the existing Ghost db
- but what it should not do is: add test fixtures like roles or users from our DataGenerator and cross fingers it will work
- this commit adds a clean method for functional tests to add extra users
* 🎨 functional tests adaptions
- use last commit to insert users for functional tests clean
- tidy up usage of testUtils.setup or testUtils.doAuth
* 🐛 test utils: reset database before init
- ensure we don't have any left data from other tests in the database when starting ghost
* 🐛 fix test (unrelated to this PR)
- fixes a random failure
- return statement was missing
* 🎨 make changes for invites
* 🎨 schema change
- simply role_id attribute
* 🎨 update invite model
- remove all methods we don't need
- ensure we remove the relation from the model
- ensure we do not allow to call withRelated
* 🎨 adapt api changes
* 🎨 adapt auth module
* 🎨 adapt tests
* 🎨 better error handling
* schema update
refs #7494, refs #7495
I saw tests adding permissions and roles twice. (see screenshots)
That happened because the setup in the test was mis-used and there is no restriction for static resources to create duplicates.
With this PR i suggest to make name unique.
* 🎨 settingsCache is available
- do not destroy the object reference
- added TODO to reconsider the config values for theme
- get one or all cached settings
* 🚨 remove api.init
- this functiion has just wrapped a function to update the settings cache
- if we have multiple tasks todo later, we can re-add
- but for now: this is way easier to read
- adapt test
* 🎨 tests
no issue
- removes count from user checks model
- uses brute express brute with brute-knex adaptor to store persisted data on spam prevention
- implement brute force protection for password/token exchange, password resets and private blogging
* 🎨 knex-migrator reset
[ci skip]
* ✨ add migration example
- hooks
- 1.0
[ci skip]
* 🛠 knex-migrator tarball
- remove when released
[ci skip]
* 🎨 jscs/jshint
* 🕵🏻 do not drop the database connection when running tests
- please read the comments in the commit
* 🔥 remove example migration
* 🛠 knex-migrator 0.1.0
* 🛠 knex-migrator 0.1.1
- fix a single test to ensure we catch the error
* 🛠 knex-migrator 0.1.2
* 🎨 make tests green
- added my keyword: kate-migrations
- i will go over all TODO's when removing the old migrations code
* 🛠 knex-migrator update
* 🛠 knex-migrator 0.2.0
* 🔥 remove User model functions
- validateToken
- generateToken
- resetPassword
- all this logic will re-appear in a different way
Token logic:
- was already extracted as separate PR, see https://github.com/TryGhost/Ghost/pull/7554
- we will use this logic in the controller, you will see in the next commits
Reset Password:
Was just a wrapper for calling the token logic and change the password.
We can reconsider keeping the function to call: changePassword and activate the status of the user - but i think it's fine to trigger these two actions from the controlling unit.
* 🔥 remove password reset tests from User model
- we already have unit tests for change password and the token logic
- i will re-check at the end if any test case is missing - but for now i will just burn the tests
* ✨ add token logic to controlling unit
generateResetToken endpoint
- the only change here is instead of calling the User model to generate a token, we generate the token via utils
- we fetch the user by email, and generate a hash and return
resetPassword endpoint
- here we have changed a little bit more
- first of all: we have added the validation check if the new passwords match
- a new helper method to extract the token informations
- the brute force security check, which can be handled later from the new bruteforce middleware (see TODO)
- the actual reset function is doing the steps: load me the user, compare the token, change the password and activate the user
- we can think of wrapping these steps into a User model function
- i was not sure about it, because it is actually part of the controlling unit
[ci skip]
* 🎨 tidy up
- jscs
- jshint
- naming functions
- fixes
* ✨ add a test for resetting the password
- there was none
- added a test to reset the password
* 🎨 add more token tests
- ensure quality
- ensure logic we had
* 🔥 remove compare new password check from User Model
- this part of controlling unit
* ✨ compare new passwords for user endpoint
- we deleted the logic in User Model
- we are adding the logic to controlling unit
* 🐛 spam prevention forgotten can crash
- no validation happend before this middleware
- it just assumes that the root key is present
- when we work on our API, we need to ensure that
1. pre validation happens
2. we call middlewares
3. ...
* 🎨 token translation key
closes#7628
With this PR we expose a public configuration endpoint.
When /ghost is requested, we don't load and render the configurations into the template anymore. Instead, Ghost-Admin can request the public configuration endpoint.
* 🎨 make configuration endpoint public
* 🔥 remove loading configurations in admin app
- do not render them into the default html page
* ✨ load client credentials in configuration endpoint
- this is not a security issue, because we have exposed this information anyway before (by rendering them into the requested html page)
* 🎨 extend existing configuration integration test
* ✨ tests: add ghost-auth to data generator
* ✨ add functional test
* 🔥 remove type/value pattern
* 🎨 do not return stringified JSON objects
* 🎨 do not call generateSlug twice for User.setup
* 🎨 call generatePasswordHash onSaving only
- now we can add defaults to User Model
- it was not possible before because add User model did the following:
1. validate password length
2. hash password manually
3. call ghostBookshelf.Model.add and THEN bookshelf defaults fn gets triggered
- call generatePasswordHash in onSaving hook for all use case
- add more tests to user model, juhu
