refs #7432🚨 database: change hard limits and field types
- we went over all schema fields and decided to decrease/increase the hard limits
- the core goal is to have more flexibility in the future
- we reconsidered string vs. text
There are 5 groups:
- small strings: 50 characters
- static strings
- status, visibility, language, role name, permission name, client name etc.
- medium strings: 191 characters
- all unique fields or fields which can be unique in the future
- slug, tokens, user name, password, tag name, email
- large strings: 1000-2000 characters
- these fields need to be very flexible
- these fields get a soft limit attached (in a different PR)
- post title, meta title, meta description, urls
- medium text: 64kb characters
- bio, settings, location, tour
- long text: 1000000000 chars
- html, amp, mobiledoc, markdown
🙄 sort_order for tests
- sort order was not set for the tests, so it was always 0
- mysql could return a different result
in my case:
- field length 156 returned the following related tags ["bacon", "kitchen"]
- field length 157 returned the following related tags ["kitchen", "kitchen"]
Change client.secret to 191
Tweak field lengths
- Add 24 char limit for ids
- Limited fields are the exact length they need
- Unified 1000 and 2000 char string classes to all be 2000
- Changed descriptions to be either 2000, except user & tag which is text 65535 as these may be used to store HTML later?!
- Updated tests
🛠 Update importer tests
- The old 001-003 tests are kind of less relevant now.
- Rather than worrying about past versions of the data structure, we should check that the importer only imports what we consider to be valid data
- I've changed the tests to treat the title-length check as a length-validation check, rather than a test for each of the old versions
🔥 Remove foreign key from subscribers.post_id
- There's no real need to have an index on this column, it just makes deleting posts hard.
- Same as created_by type columns, we can reference ids without needing keys/indexes
no issue
- we have to remember the auth url in Ghost
- if Ghost starts for the first time, it registers a public client in the defined auth service
- if you change the auth service, Ghost won't recognize
- if Ghost doesn't recognize, you will see a client does not exist error in Ghost Admin
refs #7489
- as we are now using a different migration approach (knex-migrator), we don't need to remember the database version anymore
- it was once used to check the state of a database and based on it we decided to migrate or not
- with knex-migrator everything depends on the migration table entries and the current ghost version you are on
- on current master the leftover usage is to add the db version when exporting the database, which can be replaced by reading the ghost version
- removing this solves also an interesting migration case with knex-migrator:
- you are on 1.0
- you update to 1.1, but 1.1 has no migrations
- the db version would remain in 1.0
- because the db version was only updated when knex migrator executed a migration
no issue
- if re-running the init scripts (for example: you lost the init rows in the migrations table), then it was throwing errors
- 1. the owner slug and email can change -> no match and it tried to reinsert the user with id 1, which failed
- 2. querying an inactive user is not allowed, because the user model protects against it
refs #7688
Adds logic in theme settings api to either serve an uploaded favicon and give it the type `upload` or use the default settings `default`, which will serve the favicon from our shared directory.
TODOs for #7688:
- [X] Figure out, which favicon should be used (uploaded or default) -> this PR
- [ ] Serve and redirect the favicon for any browser requests, incl. redirects
- [ ] Upload favicon via `general/settings` and implement basic admin validations -> [WIP] TryGhost/Ghost-Admin#397
- [ ] Built server side validations
closes#7769
Because Google AMP is bitching around and shows errors in Googles' webmaster tools for missing post images and blog icons, we decided to make AMP optional. It will be enabled by default, but can be disabled in general settings. Once disabled, the `amp` route doesn't work anymore.
This PR contains the back end changes for Ghost-alpha:
- Adds `amp` to settings table incl default setting `true`
- Adds `amp` value to our settings cache
- Changes the route handling of AMP app to check for the `amp` setting first.
- Adds tests to check the route handling and ghost_head output
- Includes changes to `post-lookup.js` as done by @kirrg001 in #7842
* 🛠 bookshelf tarball, bson-objectid
* 🎨 schema changes
- change increment type to string
- add a default fallback for string length 191 (to avoid adding this logic to every single column which uses an ID)
- remove uuid, because ID now represents a global resource identifier
- keep uuid for post, because we are using this as preview id
- keep uuid for clients for now - we are using this param for Ghost-Auth
* ✨ base model: generate ObjectId on creating event
- each new resource get's a auto generate ObjectId
- this logic won't work for attached models, this commit comes later
* 🎨 centralised attach method
When attaching models there are two things important two know
1. To be able to attach an ObjectId, we need to register the `onCreating` event the fetched model!This is caused by the Bookshelf design in general. On this target model we are attaching the new model.
2. We need to manually fetch the target model, because Bookshelf has a weird behaviour (which is known as a bug, see see https://github.com/tgriesser/bookshelf/issues/629). The most important property when attaching a model is `parentFk`, which is the foreign key. This can be null when fetching the model with the option `withRelated`. To ensure quality and consistency, the custom attach wrapper always fetches the target model manual. By fetching the target model (again) is a little performance decrease, but it also has advantages: we can register the event, and directly unregister the event again. So very clean code.
Important: please only use the custom attach wrapper in the future.
* 🎨 token model had overriden the onCreating function because of the created_at field
- we need to ensure that the base onCreating hook get's triggered for ALL models
- if not, they don't get an ObjectId assigned
- in this case: be smart and check if the target model has a created_at field
* 🎨 we don't have a uuid field anymore, remove the usages
- no default uuid creation in models
- i am pretty sure we have some more definitions in our tests (for example in the export json files), but that is too much work to delete them all
* 🎨 do not parse ID to Number
- we had various occurances of parsing all ID's to numbers
- we don't need this behaviour anymore
- ID is string
- i will adapt the ID validation in the next commit
* 🎨 change ID regex for validation
- we only allow: ID as ObjectId, ID as 1 and ID as me
- we need to keep ID 1, because our whole software relies on ID 1 (permissions etc)
* 🎨 owner fixture
- roles: [4] does not work anymore
- 4 means -> static id 4
- this worked in an auto increment system (not even in a system with distributed writes)
- with ObjectId we generate each ID automatically (for static and dynamic resources)
- it is possible to define all id's for static resources still, but that means we need to know which ID is already used and for consistency we have to define ObjectId's for these static resources
- so no static id's anymore, except of: id 1 for owner and id 0 for external usage (because this is required from our permission system)
- NOTE: please read through the comment in the user model
* 🎨 tests: DataGenerator and test utils
First of all: we need to ensure using ObjectId's in the tests. When don't, we can't ensure that ObjectId's work properly.
This commit brings lot's of dynamic into all the static defined id's.
In one of the next commits, i will adapt all the tests.
* 🚨 remove counter in Notification API
- no need to add a counter
- we simply generate ObjectId's (they are auto incremental as well)
- our id validator does only allow ObjectId as id,1 and me
* 🎨 extend contextUser in Base Model
- remove isNumber check, because id's are no longer numbers, except of id 0/1
- use existing isExternalUser
- support id 0/1 as string or number
* ✨ Ghost Owner has id 1
- ensure we define this id in the fixtures.json
- doesn't matter if number or string
* 🎨 functional tests adaptions
- use dynamic id's
* 🎨 fix unit tests
* 🎨 integration tests adaptions
* 🎨 change importer utils
- all our export examples (test/fixtures/exports) contain id's as numbers
- fact: but we ignore them anyway when inserting into the database, see https://github.com/TryGhost/Ghost/blob/master/core/server/data/import/utils.js#L249
- in 0e6ed957cd (diff-70f514a06347c048648be464819503c4L67) i removed parsing id's to integers
- i realised that this ^ check just existed, because the userIdToMap was an object key and object keys are always strings!
- i think this logic is a little bit complicated, but i don't want to refactor this now
- this commit ensures when trying to find the user, the id comparison works again
- i've added more documentation to understand this logic ;)
- plus i renamed an attribute to improve readability
* 🎨 Data-Generator: add more defaults to createUser
- if i use the function DataGenerator.forKnex.createUser i would like to get a full set of defaults
* 🎨 test utils: change/extend function set for functional tests
- functional tests work a bit different
- they boot Ghost and seed the database
- some functional tests have mis-used the test setup
- the test setup needs two sections: integration/unit and functional tests
- any functional test is allowed to either add more data or change data in the existing Ghost db
- but what it should not do is: add test fixtures like roles or users from our DataGenerator and cross fingers it will work
- this commit adds a clean method for functional tests to add extra users
* 🎨 functional tests adaptions
- use last commit to insert users for functional tests clean
- tidy up usage of testUtils.setup or testUtils.doAuth
* 🐛 test utils: reset database before init
- ensure we don't have any left data from other tests in the database when starting ghost
* 🐛 fix test (unrelated to this PR)
- fixes a random failure
- return statement was missing
* 🎨 make changes for invites
* 🎨 schema change
- simply role_id attribute
* 🎨 update invite model
- remove all methods we don't need
- ensure we remove the relation from the model
- ensure we do not allow to call withRelated
* 🎨 adapt api changes
* 🎨 adapt auth module
* 🎨 adapt tests
* 🎨 better error handling
* schema update
- i thought of keeping our schema, because it might be less confusing
- it's basically the same config brute-knex uses as default
- see last commit why we are using this type definition
refs #7494, refs #7495
I saw tests adding permissions and roles twice. (see screenshots)
That happened because the setup in the test was mis-used and there is no restriction for static resources to create duplicates.
With this PR i suggest to make name unique.
* 🎨 use updateClient function to update redirectUri
refs #7654
* 🎨 name instead of clientName
* 🎨 config.get('theme:title') for client name
- initial read can happen from config
* ✨ register public client: client name and description
- no update yet
- for initial client creation
- we forward title/description to Ghost Auth
- TODO: use settings-cache when merged
* ✨ store blog_uri in db
* 🎨 passport logic changes
- use updateClient instead of changeCallbackURL
- be able to update: blog title, blog description, redirectUri and blogUri
- remove retries, they get implemented in passport-ghost soon
- reorder logic a bit
* 🛠 passport-ghost 1.2.0
* 🎨 tests: extend DataGenerator createClient
- set some defaults
* 🎨 tests
- extend tests
- 👻
* ✨ run auth.init in background
- no need to block the bootstrap process
- if client can't be registered, you will see an error
- ensure Ghost-Admin renders correctly
* 🛠 passport-ghost 1.3.0
- retries
* 🎨 use client_uri in Client Schema
- adapt changes
- use blog_uri only when calling the passport-ghost instance
- Ghost uses the client_uri notation to improve readability
* ✨ read blog title/description from settings cache
* 🚨 Ghost Auth returns email instead of email_address
- adapt Ghost
no issue
- removes count from user checks model
- uses brute express brute with brute-knex adaptor to store persisted data on spam prevention
- implement brute force protection for password/token exchange, password resets and private blogging
* 🎨 knex-migrator reset
[ci skip]
* ✨ add migration example
- hooks
- 1.0
[ci skip]
* 🛠 knex-migrator tarball
- remove when released
[ci skip]
* 🎨 jscs/jshint
* 🕵🏻 do not drop the database connection when running tests
- please read the comments in the commit
* 🔥 remove example migration
* 🛠 knex-migrator 0.1.0
* 🛠 knex-migrator 0.1.1
- fix a single test to ensure we catch the error
* 🛠 knex-migrator 0.1.2
* 🎨 make tests green
- added my keyword: kate-migrations
- i will go over all TODO's when removing the old migrations code
* 🛠 knex-migrator update
* 🛠 knex-migrator 0.2.0
* 🎨 move heart of fixtures to schema folder and change user model
- add fixtures.json to schema folder
- add fixture utils to schema folder
- keep all the logic!
--> FIXTURE.JSON
- add owner user with roles
--> USER MODEL
- add password as default
- findAll: allow querying inactive users when internal context (defaultFilters)
- findOne: do not remove values from original object!
- add: do not remove values from original object!
* 🔥 remove migrations key from default_settings.json
- this was a temporary invention for an older migration script
- sephiroth keep alls needed information in a migration collection
* 🔥 add code property to errors
- add code property to errors
- IMPORTANT: please share your opinion about that
- this is a copy paste behaviour of how node is doing that (errno, code etc.)
- so code specifies a GhostError
* 🎨 change error handling in versioning
- no need to throw specific database errors anymore (this was just a temporary solution)
- now: we are throwing real DatabaseVersionErrors
- specified by a code
- background: the versioning unit has not idea about seeding and population of the database
- it just throws what it knows --> database version does not exist or settings table does not exist
* 🎨 sephiroth optimisations
- added getPath function to get the path to init scripts and migration scripts
- migrationPath is still hardcoded (see TODO)
- tidy up database naming to transacting
* ✨ migration init scripts are now complete
- 1. add tables
- 2. add fixtures
- 3. add default settings
* 🎨 important: make bootup script smaller!
- remove all TODO'S except of one
- no seeding logic in bootup script anymore 🕵🏻
* ✨ sephiroth: allow params for init command
- param: skip (do not run this script)
- param: only (only run this script)
- very simple way
* 🎨 adapt tests and test env
- do not use migrate.populate anymore
- use sephiroth instead
- jscs/jshint
* 🎨 fix User model status checks
refs #7489
* 🎨 protect error when creating owner
* 🎨 reset migration table
- temporary solution, see TODO's
* 🎨 use sephiroth in bootUp script
- do not populate the database
- ask sephiroth for database state
- do seeding manually (this will be removed in next seeding PR)
* 🎨 rewrite createTableIfNotExists because it causes error when running twice
- see knex issue
- hasTable and createTable
- indexes can cause trouble when calling them twice
* 🎨 tests: populate db in test env
- when forking db
- when starting ghost()
- this basically affects only the functional tests
* 🎨 server spec test adaption
- we now throw an error when database is not populated, instead of populating the database
* 🎨 migration spec adaption
- reset database now deletes migration table
- we will move the reset script into sephiroth and then we make it pretty
* 🎨 error creation adaption in bootUp
* 🎨 fixes
- sephiroth error handling
- fix tests
* 💄 Combine slashes & uncapitalise middleware
- these bits of middleware belong together
- ideally they should be optimised
* 🎨 Move ghostLocals out of themeHandler
GhostLocals sets several important values which are needed for every part of the application,
admin, api and theme. Therefore, it doesn't make sense for it to be bundled in the themeHandler.
* 🐛 Fix the uncapitalise middleware
- Updated to make correct use of req.baseUrl, req.path, req.url & req.originalUrl
- Updated the tests to actually cover our weird cases
* 🎨 Move ghostVersion logic out of config
* 💄 Group static / asset-related middleware together
* 🔥 Remove /shared/ asset handling
- The 5 files which are located in `/shared/` are all handled by individual calls to `serveSharedFile`
- Therefore this code is redundant
refs #7489
- new database versioning scheme which is based upon the Ghost version, and so easier to reason about
- massive refactor of all the version related code
Summary of changes:
* ✨ new error: DatabaseNotSeeded
* 🎨 change versioning module
- versioning is based on Ghost Version
* 🎨 change bootUp file
- add big picture description
- version error get's trigger from versioning module
* 🎨 default setting for database version is null
- very important change: this is caused by the big picture
- see bootUp description
- the database version get's set by the seed script later
- db version is by default null
- 1. population happens (we ensure that this has finished, by checking if each table exists)
- 2. seeds happening (we ensure that seeds happend if database version is set to X.X)
* 🎨 temporary change for population logic
- set database version after population happens
- ensure population of default settings happend before
- both: get's removed in next iteration
* 🎨 adapt tests && mark TODO's
* 🎨 err instance checking
refs #7116, refs #2001
- Changes the way Ghost errors are implemented to benefit from proper inheritance
- Moves all error definitions into a single file
- Changes the error constructor to take an options object, rather than needing the arguments to be passed in the correct order.
- Provides a wrapper so that any errors that haven't already been converted to GhostErrors get converted before they are displayed.
Summary of changes:
* 🐛 set NODE_ENV in config handler
* ✨ add GhostError implementation (core/server/errors.js)
- register all errors in one file
- inheritance from GhostError
- option pattern
* 🔥 remove all error files
* ✨ wrap all errors into GhostError in case of HTTP
* 🎨 adaptions
- option pattern for errors
- use GhostError when needed
* 🎨 revert debug deletion and add TODO for error id's
refs #7452
- remove references to 'patronus' in favour of GhostAuth, Note: this will require databases to be deleted ;)
- remove email addresses from test data
issue #7452
Remote oauth2 authentication with Ghost.org.
This PR supports:
- oauth2 login or local login
- authentication on blog setup
- authentication on invite
- normal authentication
- does not contain many, many tests, but we'll improve in the next alpha weeks
- Don't let people start Ghost Alpha with non-alpha databases.
- Provide a new welcome message for development mode (a little bit of positive reinforcment)
- Provide a RED WARNING when in production mode (will still be used for developing, but we can ignore)
- Change package.json to 1.0.0-alpha.0, we won't relelase this, will bump to .1 for release
closes#5071
- Send application/json requests to UpdateCheck service. New UpdateCheck service accepts JSON request
- If UpdateCheck service respponse has messages[] array, iterate over the array and create custom notifications intended for current version
- Save custom notification if its not already in the store AND its uuid is not in seenNotifications array
- When a custom notification is dismissed, store its uuid in seenNotifications array
- setup test fixtures to trigger tests properly
- api_notification_spec test to ensure custom notification can be added to store and added to seenNotifications when dismissed
- update_check_spec test to ensure custom notification can be displayed for a specific Ghost version
- added test to ensure messages meant for other versions don't create notifications
issues #6406#6399
- all dates are stored as UTC with this commit
- use moment.tz.setDefault('UTC')
- add migration file to recalculate local datetimes to UTC
- store all dates in same format into our three supported databases
- add option to remeber migrations inside settings (core)
- support DST offset for migration
- ensure we force UTC in test env
- run whole migration as transaction
- extend: Settings.findOne function
closes#6406
- adding timeZone Service to get the offset (=timezone reg. moment-timezone) overall available
- new publishedAtOffset date as CP using timeZone service and moment-timezone to calculate offset incl. DST
- removing timezone-obj transform as it became obsolete with moment-timezone
- reading timezones from configuration/timezones api endpoint
- adding a moment-utc transform to only work with utc times in backend
- when switching the timezone in the select box, the user will be shown the local time of the selected timezone
- added clock service to show actual time ticking below select box
- default timezone is '(GMT) Greenwich Mean Time : Dublin, Edinburgh, London'
- if no timezone is saved in the settings yet, the default value will be used
- showing local time in 'Publish Date' when it's a draft and no actual publishedAt value exists
- Removed the format 'DD MMM YY @ HH:mm (UTC Z)' which resolves to '01 Jan 16 @ 14:00 (UTC +02:00)'
- Changing the date.js helper in core/server for moment-timezone
- Fix timezone select: updates `selectedTimezone` to return the matching object from `availableTimezones`
- Including timezones in test for date-helper
- update to moment-timezone 0.5.1
- moving form-group of 'selectTimezone' further up so
- Tests:
- Set except for clock service in test env
- adding fixtures to mirage
- adding 'service.ajax' to navigation-test.js
- adding 'service:ghostPaths' to navigation-test.js
- Code improvements
- Changing clockservice to ES6
closes#6826
- refactors the validation of facebook and twitter input field in `general.js` and `user.js` controller
- Example validations for facebook:
- `facebook.com/username` will be corrected to the full URL
- `user` will show error `Your Page name is not a valid Facebook Page name' for `general.js` and `Your Username is not a valid Facebook Username` for `user.js` as the username in facebook has to be at least 5 characters long
- `twitter.com/username` will be autocorrected to the valid facebook URL incl. the `username`
- Example validations for twitter:
- `twitter.com/user_` will be corrected to the full URL
- `user:99` will show error `Your Username is not a valid Twitter Username`
- `facebook.com/username` will be autocorrected to the valid twitter URL incl. the `username`
- updates both acceptance tests
- adds further validation for facebook pages in general settings and user. Submitting a url which incl. `/page/` or `/pages/` will now accept any username followed incl. further `/`.
- adds a custom transform `facebook-url-user` which will extract the username (if it's a facebook page, incl. `pages/`) to store only this in the backend
- uses the `twitter-url-user` transform now also for user
closes#6534
- new input fields in general settings incl. validation
- facebook and twitter as new models in settings.js
- adds values for facebook and twitter to default-settings.js
- adds blog helpers for facebook and twittter
- rather than saving the whole URL, the Twitter username incl. '@' will be extracted from URL and saved in the settings. The User will still input the full URL. After saving the blog setting, the stored Twitter username will be parsed again as the full URL and available in the input field. A custom transform is used for this.
- adding meta fields to be rendered in {{ghost_head}}:
- '<meta property="article:publisher" content="https://www.facebook.com/page" />' and
- '<meta name="twitter:site" content="@user"/>'
- adds facebook and twitter to unit test for structured data
- adds unit test for general settings
- adds acceptance test for new input fields in general settings
- adds a custom transform for twitter model to save only the username to the server
- adds unit test for transform
closes#6584
- Frontend Changes:
- adds 'Apps' to Navigation Menu
- adds 'Slack' as nested page to Apps
- adds `apps.css`
- adds `slack-integration` model and uses `slack-settings` custom transform to parse JSON file
- adds validation for `slack` model
- adds fixtures and `slack/test` API endpoint to Mirage
- adds acceptance tests for `apps-test` and `slack-test`
- adds unit tests for `slack-settings` and `slack-integration`
- Backend Changes:
- adds API endpoint `slack/test` to send Test Notification
- adds default-values for slack model
- sends payload to slack:
- text: the url of the blogpost / test message
- icon_url: url to ghost logo
- username: Ghost
- adds `slack/index.js` to send webhook to slack if
- a new post is published (if slack webhook url is saved in settings)
- user clicks on 'Send Test Notification' in UI
- adds `slack.init()` to `server.index.js` to add event listener
- adds unit test for `slack/index`
closes#6462
- monkey-patch validator.extends() since it was dropped by validator @5.0.0
- coerce input to string prior to validation (custom toString func)
- need to handle boolean validation based on column type not isIn()
- use `lodash.tostring` to convert input values to strings
refs #6301, #6399
- new scheduler client will be used by any web app that handles time and calls back to the scheduling API at the right time
- new scheduler client will need to be confidential, rather than public, hence the 'web' type instead of 'ua'
- adds validation to client type that it must have a type of 'ua', 'web', or 'native'
refs #6301
- bump the default version & update corresponding test
- add empty task folders for 005 data & fixture migrations
- update tests to cover the new 005 upgrades
refs #6301
- Replace builder & automated database upgrade with a set of explicit tasks
- Ensure the tasks can only happen if they need to
- Remove some duplicate code between fixture & db upgrades (more to do)
- Add tests
refs #6301
- changes createTable to use createTableIfNotExists, this is consistent with deletion
- splits out backup, reset, update and populate functions from migration/index into their own files
- moves the wrapped function for populatingDefaultSettings to fixtures.ensureDefaultSettings
- moves `modelOptions` down to the fixture files that actually use it
- adds test coverage for backup, reset and populate, but not for update as that needs refactoring
refs #6301
- `currentVersion` was leftover from before the first public release of Ghost!
- simplified the code for `getDatabaseVersion`
- improved & made consistent how errors are handled in `getDatabaseVersion`
- migration error handling updated to reflect the changes in `getDatabaseVersion`
- added tests for both `getDatabaseVersion` and `setDatabaseVersion`
refs #6301
Fixtures:
- Removed all the old (and now broken) 003 fixture upgrades
- Split the 004 fixture upgrade tasks out, each into their own file
- Improved the versioning code that figured out which upgrades we should do
- Added lots of test coverage to make sure all the fixtures are still being run correctly.
Permissions fixtures:
- Changed the code that was populating permissions fixtures to be more automated & based only on what is in the permissions.json file.
- Added lots of test coverage to make sure all the permissions are still being created correctly.
Merging the two things:
- Merged the content of permissions.json and fixtures.json into fixtures.json, but using the new structure from permissions.json.
- Changed to use the new automated model & relation creation for all fixtures, not just permissions.
- The only thing that can't be auto-populated just now is the owner creation, I think that's important enough to do separately.
- There were (so far) never any permission updates, so code for updating fixtures stays the same through the merge.
- This results in two clear code paths: populate (fill out a brand new DB) and update (make any changes since the last version)
Test coverage is now 100% across both updates and populations.
refs #6301
- In the migration folder, commands.js changed to builder.js to resolve conflict with the 'commands' inside data/utils/clients/.
- a new data/schema/ folder has been added to hold all the code related to the database schema
- data/utils/clients have been moved to data/schema/clients
- data/utils/index.js has become data/schema/commands.js
- data/schema.js has been split, the definition of the DB schema stays put, the additional checks have moved to data/schema/checks.js
- data/validation/index.js has become data/schema/versioning.js
- data/fixtures has moved to data/migration/fixtures
- data/default-settings.json has moved to data/schema/default-settings.json
