refs https://github.com/TryGhost/Ghost/security/advisories/GHSA-9gh8-wp53-ccc6
refs https://github.com/TryGhost/Toolbox/issues/465
- Bookshelf relations allows us to edit relational records by default, which was used liberally in the codebase.
- Not having a clear track record of editable relations left the model layer prone to triggering unwanted nested saves and created a vulnerability where members were able to edit newsletter settings.
- With explicit editable relations it's easier to keep track of relations having editable access to related records. Makes the relational data modification pattern safer to use too.
- Anyone running 5.x should update to 5.24.1
Credits: Dave McDaniel and other members of [Cisco Talos](https://talosintelligence.com/vulnerability_reports)
refs https://github.com/TryGhost/Ghost/security/advisories/GHSA-9gh8-wp53-ccc6
refs https://github.com/TryGhost/Toolbox/issues/465
- Bookshelf relations allows us to edit relational records by default, which was used liberally in the codebase.
- Not having a clear track record of editable relations left the model layer prone to triggering unwanted nested saves and created a vulnerability where members were able to edit newsletter settings.
- With explicit editable relations it's easier to keep track of relations having editable access to related records. Makes the relational data modification pattern safer to use too.
- Anyone running 5.x should update to 5.24.1
Credits: Dave McDaniel and other members of [Cisco Talos](https://talosintelligence.com/vulnerability_reports)
refs https://github.com/TryGhost/Team/issues/2280
refs 9b0c21e0a2
As part of the email stability work, we added new `source` and `source_type` columns to `email` table, which allows us to store the email source information. The source for all existing emails before the stability work was always `html`, while newer emails will store `mobiledoc` or `lexical` directly.
While the `source` for all existing emails was populated with the `html` as part of above migration, we also need to store the right `source` for all new emails created till the feature is under a flag.
This change updates the current email flow to also store `source` with html data, so it can be used in future with new email service and allows removing old `html` column.
refs: https://github.com/TryGhost/Team/issues/1121
refs: 54574025e0
- The previous change to fall back to a generic error on the server side is resulting in lots of much less useful Sentry reports
- For unexpected errors, change what's sent to Sentry back to context
- This is done by adding a specific code, so we don't have to match on a string that might change
- Also add the error type, id, code & statusCode as tags to the events - these are searchable structured data
- Adding code as a tag also makes it possible to find all errors that showed the generic message
- As demonstrated by my comments in the boot file, I thought sentry was already depending on the version package
- IMO it's undesirable to require package.json directly esp when we have a tool setup and ready for tis
- Added a bunch of tests to show that Sentry does roughly what we think
fixes https://github.com/TryGhost/Team/issues/2284
New batch sending flow (still WIP). Logs the sent emails instead of actually sending them. Unit tests are coming in later commits.
refs https://github.com/TryGhost/Team/issues/2280
We are moving away from storing html and plaintext on email and instead will store the email data in source and source_type columns which allows us to store the email in other formats like mobiledoc and lexical. Storing in those formats allows greater flexibility for later html generation
- adds new `source` column that stores `mobiledoc`/`lexical`/`html` data for a newsletter
- adds new `source_type` column that stores one of `mobiledoc`/`lexical`/`html` to identify type of source
closes https://github.com/TryGhost/Team/issues/2290
Currently, if the whole batch of email fails to send we don’t capture
any errors directly tied to the batch. This makes it hard to debug which
and why a batch failed when debugging email errors. Going forward we'll
store the error information for a failing email batch directly that
allows easier debugging for batch.
- `error_status_code` : Captures statusCode returned by Mailgun,
available in error.status from the example batch error
- `error_message` : Captures short error message from Mailgun and
status, available in context object of batch error
- `error_data` : Captures while whole error json for a batch. As
mentioned in pitch, this will be huge data and we’ll figure out long
term how to best use this.
refs: https://github.com/TryGhost/Toolbox/issues/479
Framework includes:
* command to run tests
* command to record tests
* mechanism for starting and stopping Ghost before and after each suite of tests
* mechanism for loading fixtures into Ghost before starting tests
* sample test for controlling Ghost Admin
fixes https://github.com/TryGhost/Team/issues/2282
Added a new email service package that is used when the email stability
flag is enabled. Currently not yet implemented so will throw an error
for all entry points (if flag enabled).
Removed usage of `labs.isSet.bind` across the code, because that breaks
the stubbing of labs by `mockManager.mockLabsEnabled` and
`mockManager.mockLabsDisabled`. `flag => labs.isSet(flag)` should be
used instead.
All email depending tests now disable the `emailStability` feature flag
to keep the tests passing + make sure we still run all the tests for the
old flow while the email stability package is being built.
refs https://github.com/TryGhost/Team/issues/2268
The approach of using the service to lead email suppression data as
opposed to bookshelf relations allows us to wire things up without
having implemented the database. The getBulkSuppressionData allows us to
do this without much of a DB performance hit.
closes https://github.com/TryGhost/Team/issues/2126
- Cleaned up the following GA flags: `newsletterPaywall`, `freeTrial`, `compExpiring`, `searchHelper`, `emailAlerts`, `fixNewsletterLinks`.
refs https://github.com/TryGhost/Team/issues/2267
This will eventually be replaced by a Mailgun specific implementation,
but for now we're using mock one which responds with fake data for
local development.
no issue
Zip files involved a call to `await`, which caused the microtask queue to be processed. The zip file is deleted during this time because the request has already completed.
refs https://github.com/TryGhost/Team/issues/2246
- This change helps avoid race conditions due to a lack of a transaction
in the email job. It also moves the status check before creating the
email batches (can take a while) to prevent other timing issues in case
the job got scheduled multiple times.
- Sets the patch option to true when changing the status of an email
batch. If we don't do this, the bookshelf-relations plugin might try to
save relations too. This could have caused a 'no rows updated' error.
- Added a test that tests if the email job can only run once
- Added logging to batching logic
closes https://github.com/TryGhost/Team/issues/2274
When clicking on a tier, just before being redirected to the Stripe checkout, a Portal modal incorrectly showed the "Now check your email" message. Instead, this now shows a loading popup as expected while the checkout session is being created and redirected to on latest portal version
refs https://github.com/TryGhost/Toolbox/issues/475
- Having a single coverage config file for unit and e2e type of tests doesn't play well for tracking coverage progress. The unit test ones are run often when developing, so one can observe the changes in coverage. The e2e tests are ran mostly on CI environment and have different purposes/reach comparing to unit tests.
refs https://github.com/TryGhost/Toolbox/issues/475
- Increased statements and lines minimum required numbers as we have increased out baseline and should keep the bar as hight as possible
closes https://github.com/TryGhost/Toolbox/issues/475
- We did not have visibility and history into test coverage statistics in non-unit test suites. This data is useful identifying problematic areas and can be used to keep the code quality under control
- Enabled test coverage for e2e tests, including integration and regression tests
- Decreased the "branches" coverage requirement for the tests to pass (did not want to introduce an additional c8 configuration file just yet)
- updated the cover image to be simpler
- made the change in text fixtures as well, just to keep the fixtures in sync
Co-authored-by: Hannah Wolfe <github.erisds@gmail.com>
refs: ced1fdbdd4
- This is a follow-up to #15826 that I forgot to include in the first commit. Fixes an issue where the placeholder image would offset the youtube iframe embed on initial load.
closes https://github.com/TryGhost/Ghost/issues/14020
- Added CSS for `amp-youtube` to set the dimensions of the iframe instead of passing explicit height and width
attributes.
- Instead we set the video width to the viewport width and then set the height to a standard 16:9 ratio of YouTube videos.
fixes https://github.com/TryGhost/Team/issues/2085
Don't load relations we don't need anymore for the posts table. And
reload the individual post when we open the analytics page with more
relations that we actually need.
refs https://github.com/TryGhost/Team/issues/2216
This change removes old analytics code which was added under `membersActivity` flag as an experimental alpha feature to test the first versions of member analytics, and is no longer active or in use.
This change removes the remaining services and its usage that were created to manage this version of analytics but is no longer active or maintained.
- removes `members-analytics-ingress` service that was used to ingest events from Portal in this experimental feature
- removes `member-analytics-service` service that managed the events from this experimental feature
- removes usages of the 2 services and their dependency in `members-api`
- removes `member-analytic-event` model as the corresponding table for it does not exist anymore and was dropped in 5.0
refs https://github.com/TryGhost/Team/issues/2216
The `membersActivity` flag was an alpha feature to test the first versions of member analytics, and is no longer active or in use. This change removes the remaining pieces of code that are setup behind that flag and are no longer in use or accessible.
refs: https://github.com/TryGhost/Ghost/issues/14882
This commit totally removes Bluebird from the importer. Updated `@tryghost/promise` to use native async/await and refactored importer logic to avoid the need of `reflect()`.
fixes https://github.com/TryGhost/Team/issues/1903
MembersAgent.loginAs sends email, asynchronously via events. Which
conflicts with tests that also test emails. We cannot properly await
these events, so this is currently fixed with a timeout of 200ms. But
this was too random and unreliable.
fixes https://github.com/TryGhost/Team/issues/2025
fixes https://github.com/TryGhost/Team/issues/2023
The `ref` attribute has changed in email links:
- We now use the site name when linking to external sites
- We blacklist facebook.com because it doesn't support ref attributes
- '-newsletter' is not repeated anymore if the newsletter name already ends with 'newsletter'
- We always sluggify the ref
- We no longer overwrite existing ref, utm_source or source parameters
closes https://github.com/TryGhost/Team/issues/2222
Whilst we were checking for Stripe objects being active, we were not
checking for them existing in Stripe. This adds handling to all read
request to Stripe in the payment link flow, so that we can gracefully
handle deleted objects.
We've also included an automated test which fails without this fix.
We've also improved the query to find Stripe Prices which will result
in less request to the Stripe API to check if it is valid.
refs https://ghost.slack.com/archives/C02G9E68C/p1667834794676479
- When enabling tracking, it could be the case that the server is ignoring the attributions because of the cached setting value.
- When disabling tracking, the frontend should take care of not
collecting new tracking information to the server, but still the backend value should be used as a fail-safe.
fixes https://github.com/TryGhost/Team/issues/2220
`Post.findOne` is filtering by status 'published' by default. 'sent' isn't 'published' so we throw a 404 error when trying to send feedback to a 'sent' post.
closes https://github.com/TryGhost/Team/issues/2207
- adds conditional to the post email serializer to switch between
`mobiledocLib` and `lexicalLib` depending on which format the post
contains
closes https://github.com/TryGhost/Team/issues/2211
We were allowing paid Tiers to be imported with non-integer prices which was
causing the Admin to be bricked when attempting to load them. This adds some
validation to the price data of Tiers.
refs https://github.com/TryGhost/Toolbox/issues/461
- The 'vary' header with 'Origin' value should only be set when an OPTIONS header is processed. Otherwise we are prone to leaking the vary header modification to further down in the request pipeline
refs https://github.com/TryGhost/Toolbox/issues/461
- The unit test was never using the "OPTIONS" request method, which did not actually trigger the full logic of the "cors" module used under the hood.
- Using the correct request method triggers all the right pathways and tests the state that's closer to the real world - for example the response does get "ended" instead of calling the "next" middleware.
refs https://github.com/TryGhost/Toolbox/issues/461
- Having a 'Origin' in vary header value present on each `OPTIONS` allows to correctly bucket "allowed CORS" and "disallowed CORS" responses in shared caches
refs https://github.com/TryGhost/Toolbox/issues/461
- Having a 'Origin' in vary header value present on each `OPTIONS` allows to correctly bucket "allowed CORS" and "disallowed CORS" responses in shared caches
refs https://github.com/TryGhost/Toolbox/issues/461
- The codebase has ambiguous behavior with OPTIONS request. Adding tests covering edge cases for all possible variations of OPTIONS responses is the first step to solving cahceability of these requests.
- The obvious question if you look into the changeset itself would also be: "WTF did you do with test suite naming? What are these changes in admin and click tracking suites? You having a bad day Naz?". The answer is "yes" (╯°□°)╯︵ ┻━┻
- On a serious note. I've introduced multiple hacks here that should be fixed:
1. Forced test suite execution order for options request - extreme blasphemy. This was last resort decision. I went deep into trying to fixup the server shutdown in the "admin" test suite, which cascaded into failing "click tracking" suite, which has shortcomings on it's own (see notes left in that suite)
2. Exposed "ghostServer" from the e2e-framework's "getAgentsWithFrontend" method. Exposing ghostServer to be able to shut it down (or do other manipulations) was one of the pitfalls we had in the previous test utils, which ended up plaguing the test codebase. Ideally the framework should only be exposing the agents and the rest would happen behind the scenes.
- To fix the hacks above I've raised a cleanup issue (https://github.com/TryGhost/Toolbox/issues/471). I'm very sorry for this mess. The issue at hand has very little to do with fixing the e2e framework, so leaving things "as is".
refs https://github.com/TryGhost/Toolbox/issues/461
- When testing OPTIONS requests there is a need to get all possible agents available in the system. The "getAgentsWithFrontend" serves exactly this purpose - create all possible agents while starting Ghost instance only once
- This is groundwork for OPTIONS request caching tests and improvements
refs https://github.com/TryGhost/Ghost/commit/1f300fb781f0
The full customer object was not being passed to the StripeAPI service
when it already exists, this was resulting in inconsistent behaviour when
sending the customerEmail param to the API, causing `invalid_email`
errors to be thrown from Stripe and breaking the checkout.
closes https://github.com/TryGhost/Team/issues/2196
We were incorrectly assuming that all requests would have the
`customerEmail` passed in the body. Instead we were incorrectly
passing `undefined` or `''` as the `customerEmail` property to stripe,
which resulted in a validation error.
We've updated the code to pass `null` in the case of a falsy value,
which the Stripe API handles without error.
closes https://github.com/TryGhost/Team/issues/2195
The issue here is two-fold, and specific to using Offers so was not
caught by any automated tests. First, we were incorrectly comparing
the tier.id to the offer.tier.id - this is because the Tier objects id
property is an instance of ObjectID rather than a string.
Secondly we were passing through the cadence parameter from the
request body, but when using Offers this is not including in the
request, so we must pull the data off of the Offer object instead and
pass that to the payments service.
closes https://github.com/TryGhost/Ghost/issues/15740
The validation function for a Tier description was not returning the
validated value, which meant we were unable to set the Tier
description.
refs https://github.com/TryGhost/Toolbox/issues/464
Bceause the import does not use the API, any backwards compat code we put in the
API does not get run for imports, this means we need to update the importer to
map the stripe_prices data onto the products table so that we have valid data in
the database.
refs https://github.com/TryGhost/Toolbox/issues/464
- due to a bug with the content importer, importing a JSON file where
the `products` do not contain price info will store null values in the
table instead of the defaults
- this ends up causing further issues because we're not populating the
table for paid products
- this commit is a copy of the 5.19 migration
`2022-09-02-20-52-backfill-new-product-columns.js`, but adds a check
for a null `t.currency`, which combined with the `t.type === paid`,
should identify the rows we want to update
refs: a8b1676734
- Extended the newly created handlebars test utils with a shouldCompileToError method
- Updated the price helper tests tp use shouldCompileToExpected and shouldCompileToError
- This allows us to test our handlebars helpers in a much more conisstent way
refs: https://github.com/TryGhost/Ghost/issues/14882
- Removing bluebird specific methods in favour of the Ghost sequence method so we can remove the bluebird dependency
Co-authored-by: Carol-Barno <cbarno@innovexsolutions.co.ke>
refs: https://github.com/TryGhost/Ghost/issues/14882
- Removing bluebird specific methods in favour of the Ghost sequence method so we can remove the bluebird dependency
Co-authored-by: Carol-Barno <cbarno@innovexsolutions.co.ke>
no issue
- There are currently two patterns in our handlebars helper unit tests:
1. Treating the helper as a function, and doing a function call
- This is the original way the tests were done, and they're not great as they're approximating how the helpers are really used
2. Using a template string, and rendering the string using a method called shouldCompileToExpected
- These tests are more realistic and powerful and also easier to read
- The new method is only being used in a few places so far, and each place had re-created the `shouldCompileToExpected` method
- Therefore I've moved this method into a util that should make it easier to write unit tests for handlebars helpers
- I also renamed the method in the excerpt tests, because it doesn't do the same thing, it's just a wrapper around a function call rather than compiling a string
The aim is to refactor all of our handlebars helper tests to use `shouldCompileToExpected`
- These tests are very slow, and make the build fail about 2/3 times
- Temporarily skipping until we can fix, as I want to get all our outstanding hacktoberfest PRs merged
refs https://github.com/TryGhost/Team/issues/2158
- the cache invalidation header returned should be specific to the email links pattern, otherwise it blows entire cache on every link edit
fixes https://github.com/TryGhost/Team/issues/2175
- New event type `aggregated_click_event` that is disabled by default in all the existing activity feeds
- This returns click events, but only the first click events for each member/post combination.
- It includes the total count of unique link clicks for that member on that post combination
- Had to resort to some custom knex queries to make this work easily
- Requires `@tryghost/bookshelf-pagination@0.1.31`, included in `@tryghost/bookshelf-plugins@0.6.1` (this fixes an issue with custom selects breaking the total count query of pages)
- Went a bit overboard with the pagination tests to cover as much unknown edge cases as possible
refs https://github.com/TryGhost/Team/issues/2168
- site owners can now disable tracking sources from analytics settings.
- this change removes the loading of attribution script if tracking is
turned off so we don't capture any post/page or external source
attributions
refs https://github.com/TryGhost/Team/issues/2168
- the new setting allows site owners to control if they want to track
the sources for new member signups and subscriptions
- its switched on by default, but can be toggled off from new analytics
settings page
refs: https://github.com/TryGhost/Toolbox/issues/440
This was working locally where the dependency is resolved implicitly, but when deployed there is no @tryghost/data-generator in the node_modules folder.
fixes https://github.com/TryGhost/Team/issues/2129
- This changes how the activity feed API parses the filter.
- We now parse the filter early to a MongoDB filter, and split it in two. One of the filters is applied to the pageActions, and the other one is used individually for every event type. We now allow to use grouping and OR's inside the filters because of this change. As long as we don't combine filters on 'type' with other filters inside grouped filters or OR, then it is allowed.
- We make use of mongoTransformer to manually inject a mongo filter without needing to parse it from a string value again (that would make it a lot harder because we would have to convert the splitted filter back to a string and we currently don't have methods for that).
- Added sorting by id for events with the same timestamp (required for reliable pagination)
- Added id to each event (required for pagination)
- Added more tests for filters
- Added test for pagination
- Removed unsued getSubscriptions and getVolume methods
Used new mongo utility methods introduced here: https://github.com/TryGhost/NQL/pull/49
refs https://github.com/TryGhost/Team/issues/2077
- Members and Posts test suites were using a broad tiers property matcher, which is an anti-pattern for snapshot tests. Without more specific snapshots it would be very hard to track down tier-related breaking changes!
- This change is groundwork for a refactor coming in tier usage at API's output serializers
closesTryGhost/Team#2159
- Added column to email table
- Hide the feedback tab on frontend depending on the column value
Co-authored-by: Daniel Lockyer <daniellockyer@fastmail.com>
refs: https://github.com/TryGhost/Toolbox/issues/440
New command to generate demo data, creates data for over 20 tables in
Ghost, suitable for testing most features of the dashboard, as well as
making guided product tours using newsletters, tiers, many posts and
tags.
Usage: `yarn start generate-data`
Optionally, keep your existing posts / tags with: `yarn start generate-data --use-existing-tags --use-existing-posts`
refs https://github.com/TryGhost/Team/issues/2077
- The "productRepository" methods have been deprecated in favor of "tiers" and "Tiers API".
- The changes migrated usages of "productRepository.getDefaultProduct" to Tiers API's "readDefaultTier"
refs https://github.com/TryGhost/Team/issues/2077
- Passing in the whole "getMembersApi" is just too much state to know about for the importer - it only uses a concept of default tier and members repository, the rest is distracting fluff making it hard to reason about what the importer **has to** know to function
- Passing in two functions breaking up the above state simplifies the constructor API.
- This is also a groundwork before substituting productsRepository for tiersRepository (refed issue objective)
refs: https://github.com/TryGhost/Ghost/issues/15537
- snapshot test created to add confidence to webhook stability and increase overall test coverage.
Co-authored-by: Kritika Sharma <kritikasharma@Kritikas-MacBook-Pro-2.local>
fixes https://github.com/TryGhost/Team/issues/2137
For the analytics page, we need the sent events to show up immediately
after sending an email. Otherwise we need to wait for emails to be
marked as received (which takes too long) before being able to show them
on the analytics page.
This adds the email_sent_event, which is hidden by default everywhere
and used on the analytics page.
fixes https://github.com/TryGhost/Team/issues/2114
fixes https://github.com/TryGhost/Team/issues/2115
When a new newsletter is created, the frontend will send feedback_enabled to true. We'll catch this in the backend and don't allow setting feedback_enabled to true when audience_feedback flag is disabled. This is also handled for editing newsletters.
To fix this in existing sites, I added a migration that disables feedback for all sites (since this is an alpha feature). Once we'll release the feature later, it will be disabled for existing newsletters, just like expected.
refs: https://github.com/TryGhost/Ghost/issues/14882
- Opted to use the in-house `sequence` function when refactoring Bluebird's `Promise.each` to avoid deadlock issues (see 734ef66e6c).
-It's hard to know without tonnes of context if any `Promise.each` are safe to refactor to `Promise.all`.
refs: https://github.com/TryGhost/Ghost/issues/14882
- Removing bluebird specific methods in favour of the Ghost sequence method so we can remove the bluebird dependency
Co-authored-by: Hannah Wolfe <github.erisds@gmail.com>
fixes https://github.com/TryGhost/Team/issues/2112
- Removed a bit of duplicate code across templates and components that was used to handle filters
- Updated filter objects to contain information about the filter
- Added resource filters that are able to select a single resource, which can be used in columns
- Filters can now define columns by themselves. Not all columns already make use of this functionality, but we can move those over later (cleanup: https://github.com/TryGhost/Team/issues/2133)
- The filter definitions became quite long. We should move them to separate files in the future: https://github.com/TryGhost/Team/issues/2134
- Filters can now have custom NQL parsing
- Improved support for parsing recursive or grouped NQL queries
- Added support for filtering members by feedback
refs https://github.com/TryGhost/Team/issues/2116
- allows site owners to edit a link in a post that has already been sent out, fixing any typos or other mistakes
- resets click counter for the edited link back to 0 so site owners can see the clicks on new link, doesn't change the overall click count
