fixes https://github.com/TryGhost/Team/issues/2366
refs https://ghost.slack.com/archives/C02G9E68C/p1670232405014209
Probem described in issue.
In the old MEGA flow:
- The `email_verification_required` check is now repeated inside the job
In the new email service flow:
- The `email_verification_required` is now checked (didn't happen
before)
- When generating the email batch recipients, we only include members
that were created before the email was created. That way it is
impossible to avoid limit checks by inserting new members between
creating an email and sending an email.
- We don't need to repeat the check inside the job because of the above
changes
Improved handling of large imports:
- When checking `email_verification_required`, we now also check if the
import threshold is reached (a new method is introduced in
vertificationTrigger specifically for this usage). If it is, we start
the verification progress. This is required for long running imports
that only check the verification threshold at the very end.
- This change increases the concurrency of fastq to 3 (refs
https://ghost.slack.com/archives/C02G9E68C/p1670232405014209). So when
running a long import, it is now possible to send emails without having
to wait for the import. Above change makes sure it is not possible to
get around the verification limits.
Refactoring:
- Removed the need to use `updateVerificationTrigger` by making
thresholds getters instead of fixed variables.
- Improved awaiting of members import job in regression test
- this was all getting terribly behind so I've done several things:
- majority of `@tryghost/*` except Lexical packages
- gscan + knex-migrator to remove old `@tryghost/errors` usage
- bumped lockfile
refs: https://github.com/TryGhost/Team/issues/1121
- This makes several key changes to the way errors are handled in the member importer, to ensure that we only show error messages to users that we wrote.
- Fundamentally, we no longer trust all API errors, and instead only trust a set of very specific API errors. Anything outside of that is replaced with a generic error message.
- Also switches the server-side error generated for email verification (which can throw during member import) to be a HostLimitError, as that is a more appropriate class.
- Note: there are many other parts of Ghost admin that need a similar overhaul, and a similar change we need to introduce server side to fully resolve the underlying issue of bubbling up code errors to the UI.
refs https://github.com/TryGhost/Team/issues/2192
The method signatures of the Event Repository have been updated to
take mongo filter objects, but this call-site was not updated.
Long term we should really be using NQL filter strings for our
filtering API and the mongo filter objects should be an implementation
detail, however we don't have time right now to rectify this.
closes https://github.com/TryGhost/Toolbox/issues/399
- The MemberCreatedEvent event is more accurate representation of the limit nature - counting the number of members created. The previous MemberSubscribeEvent was slightly hacky solution because a member could be subscribed/unsubscribed multiple times and distorting the limit counts.
refs https://github.com/TryGhost/Toolbox/issues/387
- The limit values should be as configurable as possible to adjust verification thresholds dinamically per-usecase. This solves a problem of doing a separate version release when we need to adjust the verification thresholds.
- Before this "importThreshold" was the same concept as "apiThreshold", which makes it hard&confusing to reason about and hard to parameterize each specific case.
refs https://github.com/TryGhost/Toolbox/issues/387
- Similar reasoning as to previous renames - the variables were named with a single trigger source in mind and now would be confusing with multiple verification trigger sources.
refs https://github.com/TryGhost/Toolbox/issues/387
- The "amountImported" was to specific to one verification trigger source. There can be multiple sources that start the verification process.
- Changed `startVerificationProcess` method signature to reflect it's a private method that's only used internally - exposed for testing purposes only.
refs https://github.com/TryGhost/Toolbox/issues/387
- There will three distinct verification limits soon. To keep the naming clear "configThreshold" would be too generic/confusing to use.
- Introduced jsdoc descriptions for the "source" parameter, which will be corelating with each new config parameter ("apiTriggerThreshold", "importTriggerThreshold", "adminTriggerThreshold", etc.). This should give a better visibility into parameters we are dealing in this area.
refs https://github.com/TryGhost/Toolbox/issues/387
- I'm about to add another event source - "admin". Before doing that made the method more parameter dependent, so it can handle limit triggering logic from multiple source and based on multiple configuration parameters.
refs https://github.com/TryGhost/Toolbox/issues/387
- The constructor should be light initialization logic only. Putting business logic into constructor is quite dirty and not really testable!
- cleaned up unused dependencies
- adds missing dependencies that are used in the code
- this should help us be more explicit about the dependencies a package
uses
- because of how the npm scripts were set up, we were running the full
Admin integration tests during the unit tests phase of CI
- this commit renames the majority of `test` to `test:unit` in the
package.json files, and aliases `test` to `test:unit`
- special packages like Admin have no-op'd `test:unit` scripts so we
don't end up running its tests
- if the threshold is Infinity, we shouldn't be loading the newsletter
subscription events because we are saying there is no threshold
- the code has a quick path to avoid comparing the values, but it still
loads the events upfront
- this commit moves the quick path up to return earlier
- this has the nice side-effect of producing 100% coverage on this
package
refs https://github.com/TryGhost/Toolbox/issues/354
- these READMEs were migrated over from when each package was in a
different repo
- they also assume you're going to be publishing the packages because it
mentions install instructions
- only a few of them contain custom content
- this commit deletes the majority of these files because they're now
not useful
- any that contained other instructions have been cut down
refs https://github.com/TryGhost/Toolbox/issues/354
- these repository links made sense when they were in different repos
and published to NPM but we don't publish these packages any more
- this commit deletes those keys from the files
- these were copied over during the monorepo conversion but we're not
going to be publishing these packages so the top-level LICENSE file
covers all packages here
- we're going to be pinning all dependencies within the monorepo
- this shouldn't change anything anyway because we're using the same
version across all packages
- these packages are split apart for local development, but will be
bundled into Ghost when publishing
- therefore, these packages won't be published so we are resetting the
versions to make them cleaner
refs: https://github.com/TryGhost/Toolbox/issues/293
Things needed to create this:
* MemberSubscriptionEvent now has an import source
* Importer now creates events with this type
* Verification trigger logic changed to use 30 day window of imports
