ref https://linear.app/tryghost/issue/CFR-4/
- added request queueing middleware (express-queue) to handle high
request volume
- added new config option `optimization.requestQueue`
- added new config option `optimization.requestConcurrency`
- added logging of request queue depth - `req.queueDepth`
We've done a fair amount of investigation around improving Ghost's
resiliency to high request volume. While we believe this to be partly
due to database connection contention, it also seems Ghost gets
overwhelmed by the requests themselves. Implementing a simple queueing
system allows us a simple lever to change the volume of requests Ghost
is actually ingesting at any given time and gives us options besides
simply increasing database connection pool size.
---------
Co-authored-by: Michael Barrett <mike@ghost.org>
- this version is written in TS, but was published a few months ago and
needs to be bumped here
- also updates a previous deep include into the library, which was
unnecessary anyway
fixes ENG-610
- Previously, when importing an existing member, if the name or note
field is left blank in the CSV file, this would overwrite (re: delete)
the existing name or note in the database.
- This change ensures that the name and note fields are only updated if
they are not blank in the CSV file.
fixes ENG-611
- Previously, if an existing member with newsletter subscriptions was
imported, and `subscribe_to_emails` was blank/empty, the member would be
unsubscribed from all newsletters, which is not the expected behavior.
- This PR changes the behavior so if `subscribe_to_emails` is blank, it
will not unsubscribe existing members.
closes https://github.com/TryGhost/Product/issues/3593
- when importing members via CSV, it's now possible to add the value "auto" for the "stripe_customer_id" field. When this option is passed, the importer will search for a Stripe customer based on the email address provided
- if there are multiple Stripe customers with the same email address, the customer with the most recent subscription is returned
fixes https://github.com/TryGhost/Product/issues/2947
When an import is performed and a member already exists:
- If `subscribe_to_emails` is set to `true` and the existing member
already has newsletter subscriptions, these are preserved
- If `subscribe_to_emails` is set to `true` and the existing member does
not already have newsletter subscriptions, no new subscriptions are
added in case the member has previously unsubscribed
- If `subscribe_to_emails` is set to `false` the existing member will be
unsubscribed from all newsletters regardless of their preference
refs: https://github.com/TryGhost/Toolbox/issues/595
We're rolling out new rules around the node assert library, the first of which is enforcing the use of assert/strict. This means we don't need to use the strict version of methods, as the standard version will work that way by default.
This caught some gotchas in our existing usage of assert where the lack of strict mode had unexpected results:
- Url matching needs to be done on `url.href` see aa58b354a4
- Null and undefined are not the same thing, there were a few cases of this being confused
- Particularly questionable changes in [PostExporter tests](c1a468744b) tracked [here](https://github.com/TryGhost/Team/issues/3505).
- A typo see eaac9c293a
Moving forward, using assert strict should help us to catch unexpected behaviour, particularly around nulls and undefineds during implementation.
refs: https://github.com/TryGhost/Toolbox/issues/188
- some of our older packages used a pattern for linting which missed using test config for linting tests
- we need this to be consistent so that we can add more eslint rules for testing
- two packages also didn't use the lib pattern, which made the lint pattern error - so this was fixed as well
As discussed with the product team we want to enforce kebab-case file names for
all files, with the exception of files which export a single class, in which
case they should be PascalCase and reflect the class which they export.
This will help find classes faster, and should push better naming for them too.
Some files and packages have been excluded from this linting, specifically when
a library or framework depends on the naming of a file for the functionality
e.g. Ember, knex-migrator, adapter-manager
closesTryGhost/Team#2793
- if a member is imported with a created_at in the future, the member
will not appear in the members list in admin
- this commit updates created_at to the current date if it is in the
future upon import
- we previously used `@stdlib/utils` instead of the child package
`@stdlib/copy`, which is a lot smaller and contains our only use of
the parent
- this saves 140+MB of dependencies
- we keep ending up with multiple versions of the depedency in our tree,
and it's causing problems when comparing instances
- the workaround I'm implementing for now is to bump the package
everywhere and set a resolution so we only have 1 shared instance
- hopefully we can come up with a better method down the line
- there's a weird situation when we have mixed versions of the
dependency because different libraries try to compare instances
- this brings the usage up to 1.2.21 so we can fix the build for now
refs https://github.com/TryGhost/Toolbox/issues/501
- this reverts commit 48dda23554
- also includes a resolution for `@elastic/elasticsearch` so we don't
run a version that is potentially problematic - see referenced issue
for context
fixes https://github.com/TryGhost/Team/issues/2366
refs https://ghost.slack.com/archives/C02G9E68C/p1670232405014209
Probem described in issue.
In the old MEGA flow:
- The `email_verification_required` check is now repeated inside the job
In the new email service flow:
- The `email_verification_required` is now checked (didn't happen
before)
- When generating the email batch recipients, we only include members
that were created before the email was created. That way it is
impossible to avoid limit checks by inserting new members between
creating an email and sending an email.
- We don't need to repeat the check inside the job because of the above
changes
Improved handling of large imports:
- When checking `email_verification_required`, we now also check if the
import threshold is reached (a new method is introduced in
vertificationTrigger specifically for this usage). If it is, we start
the verification progress. This is required for long running imports
that only check the verification threshold at the very end.
- This change increases the concurrency of fastq to 3 (refs
https://ghost.slack.com/archives/C02G9E68C/p1670232405014209). So when
running a long import, it is now possible to send emails without having
to wait for the import. Above change makes sure it is not possible to
get around the verification limits.
Refactoring:
- Removed the need to use `updateVerificationTrigger` by making
thresholds getters instead of fixed variables.
- Improved awaiting of members import job in regression test
- this was all getting terribly behind so I've done several things:
- majority of `@tryghost/*` except Lexical packages
- gscan + knex-migrator to remove old `@tryghost/errors` usage
- bumped lockfile
fixes https://github.com/TryGhost/Team/issues/2326
When importing more than 500 members, we didn't testImportThreshold at
the right time. It was called too early because the importing job was
not awaited. This also adds an E2E test for this case.
closes https://github.com/TryGhost/Team/issues/2219
- The CSV importer was failing when a "complimentary_plan" flag was present with a "true" value. The root of the issue was the data model change where the "id" of the Tier object is no longer a String but an ObjectID instance. It's a slight departure from previous bookshelf object behavior where 'id' property is always a string that is a stringified ObjectID.
- In the future we should unify the logic across all data access objects to either keep the convention of using a String under id property or switch to ObjectId instances.
refs https://github.com/TryGhost/Team/issues/1076
- The members CSV import would go into background job (longer running and resulting with an email) when it contained a complimentary_plan column.
- With recent members codebase decoupling Ghost does not make any connection to Stripe as "complimentary plan" data is saved purely in native data structures. Making no need for a background job for complimentary plans
refs https://github.com/TryGhost/Team/issues/1076
- What appeared to be a "boolean" by nature and name, the hasStripeData was holding a result of "find" method - and object or an undefined value
- Fixed the typing, to avoid ambiguity in the future
refs: https://github.com/TryGhost/Ghost/issues/14882
- Removed bluebird from members-csv package-json and update-check-service
- Removing bluebird specific methods in favour of the Ghost sequence method so we can remove the bluebird dependency
refs https://github.com/TryGhost/Team/issues/2077
- The "productRepository" methods have been deprecated in favor of "tiers" and "Tiers API".
- The changes migrated usages of "productRepository.getDefaultProduct" to Tiers API's "readDefaultTier"
refs https://github.com/TryGhost/Team/issues/2077
- Passing in the whole "getMembersApi" is just too much state to know about for the importer - it only uses a concept of default tier and members repository, the rest is distracting fluff making it hard to reason about what the importer **has to** know to function
- Passing in two functions breaking up the above state simplifies the constructor API.
- This is also a groundwork before substituting productsRepository for tiersRepository (refed issue objective)
refs https://github.com/TryGhost/Team/issues/1076
refs 70229e4fd3 (diff-b67ecda91b5bd79c598e5c5a9ec2ccf28dbfab6a924b21352273865e07cd7ceaR57)
- The "products" column has not been doing any logic anything since at least 5.20.0 (see refed commit). The concept of columns in the export file was mostly there for analytical/data filtering reasons - so the user could analyze their exports. CSV was never a good suite for relational data that "products" (or now tiers) represent
- The "tiers" column will still be present in the exported CSV file, but there is not going to be any logic attached to it.
- The only columns that can effect the "tiers" state of the member are: "complimentary_plan" (assign default tier to the member) and "stripe_customer_id" (pulls in subscription/tier data from Stripe)
