fixes https://github.com/TryGhost/Team/issues/2326
When importing more than 500 members, we didn't testImportThreshold at
the right time. It was called too early because the importing job was
not awaited. This also adds an E2E test for this case.
closes https://github.com/TryGhost/Team/issues/2219
- The CSV importer was failing when a "complimentary_plan" flag was present with a "true" value. The root of the issue was the data model change where the "id" of the Tier object is no longer a String but an ObjectID instance. It's a slight departure from previous bookshelf object behavior where 'id' property is always a string that is a stringified ObjectID.
- In the future we should unify the logic across all data access objects to either keep the convention of using a String under id property or switch to ObjectId instances.
refs https://github.com/TryGhost/Team/issues/1076
- The members CSV import would go into background job (longer running and resulting with an email) when it contained a complimentary_plan column.
- With recent members codebase decoupling Ghost does not make any connection to Stripe as "complimentary plan" data is saved purely in native data structures. Making no need for a background job for complimentary plans
refs https://github.com/TryGhost/Team/issues/1076
- What appeared to be a "boolean" by nature and name, the hasStripeData was holding a result of "find" method - and object or an undefined value
- Fixed the typing, to avoid ambiguity in the future
refs https://github.com/TryGhost/Team/issues/2077
- Passing in the whole "getMembersApi" is just too much state to know about for the importer - it only uses a concept of default tier and members repository, the rest is distracting fluff making it hard to reason about what the importer **has to** know to function
- Passing in two functions breaking up the above state simplifies the constructor API.
- This is also a groundwork before substituting productsRepository for tiersRepository (refed issue objective)
refs https://github.com/TryGhost/Team/issues/1076
refs 70229e4fd3 (diff-b67ecda91b5bd79c598e5c5a9ec2ccf28dbfab6a924b21352273865e07cd7ceaR57)
- The "products" column has not been doing any logic anything since at least 5.20.0 (see refed commit). The concept of columns in the export file was mostly there for analytical/data filtering reasons - so the user could analyze their exports. CSV was never a good suite for relational data that "products" (or now tiers) represent
- The "tiers" column will still be present in the exported CSV file, but there is not going to be any logic attached to it.
- The only columns that can effect the "tiers" state of the member are: "complimentary_plan" (assign default tier to the member) and "stripe_customer_id" (pulls in subscription/tier data from Stripe)
closes https://github.com/TryGhost/Team/issues/1869
- When there were "archived" tiers in the system the importer incorrectly fetched them instead of only taking "active" ones into account. The "getDefaultProduct" on product repository does exactly that.
- Additionally, reusing the "getDefaultProduct" makes testing the importer slightly less complex.
closes https://github.com/TryGhost/Toolbox/issues/430
- The members importer used to import all fields present in the uploaded CSV if the headers match, even if they're not mapped in the UI. This behavior has lead to have misleading consequences and "hidden" features. For example, if the field was present but intentionally left as "Not imported" in the UI the field would still get imported.
- Having a strict list of supported import fields also allows for manageable long-term maintenance of the CSV Import API and detect/communicate changes when they happen.
- The list of the current default field mapping is:
email: 'email',
name: 'name',
note: 'note',
subscribed_to_emails: 'subscribed',
created_at: 'created_at',
complimentary_plan: 'complimentary_plan',
stripe_customer_id: 'stripe_customer_id',
labels: 'labels',
products: 'products'
refs https://github.com/TryGhost/Toolbox/issues/430
- To be able to introduce strict mapping rules (exclude unknown fields) we need to control the CSV header mapping on the importer level. This change moves the configuration up from CSV parser to the importer
- Also adds tests covering correct inserts for specially treated "subscribed_to_emails" field
refs https://github.com/TryGhost/Toolbox/issues/430
- "perform()" is what gets executed by the import job for both immediate import and "inline job" import. Testing it on granular level will allow to change it with more confidence when introducing strict field mapping rules
refs https://github.com/TryGhost/Toolbox/issues/430
- Importer code was filled with an unnecessarily complex "job" object that was passed around. It had an "id" property, which confusingly was a path to a file at all times.
- Simplified the logic significantly by keeping and passing around the path to a "prepared" members CSV.
refs https://github.com/TryGhost/Toolbox/issues/430
refs https://github.com/TryGhost/Ghost/issues/14882
- The MembersCSVImporter constructor is way to complex and needs refactoring. This complexity makes initialization in tests too bulky and makes tests hard to read.
- Having a builder method is a stopgap solution to avoid going into MembersCSVImporter refactoring too deep.
no issue
When importing members, the members-importer isn't aware of the context and therefore falls back to use `member` as a source in members event table. This makes it impossible to determine imported members from others.
- Added an `context` property to the options in the members-importer constructor
- Checked for `importer` context when creating member events and assigned the source `admin` to it
refs: https://github.com/TryGhost/Toolbox/issues/293
Things needed to create this:
* MemberSubscriptionEvent now has an import source
* Importer now creates events with this type
* Verification trigger logic changed to use 30 day window of imports
refs: https://github.com/TryGhost/Toolbox/issues/166
New package handles the email verification workflow to prevent spammers. It currently handles MembersSubscribeEvent to detect potential abuse of the API to add members, and exposes methods for checking the threshold / starting the verification process for use by other areas of the code (at the moment - just member imports).
The import package no longer needs to handle anything related to verification since it can be handled in the wrapper function in Ghost, and the API package doesn't need to do anything other than dispatch the new event.
refs https://github.com/TryGhost/Team/issues/1202
When importing we were transforming the CSV and add missing columns to
it before storing it in preparation to perform the import. This resulted
in the missing columns being updated for existing members with blank
data.
We've updated the Members CSV parsing library to take an options list of
columns to include, which then allows imports to not include all of the
default columns.
refs https://github.com/TryGhost/Team/issues/958
- The import threshold has to become a dynamic number based on external parameters. Because of this it makes most sense to have it as an async function parameter
- There's a package API change for importThreshold constructor paramter becoming a fetchThreshold async funciton parameter
refs https://github.com/TryGhost/Team/issues/912
- We need a way to check if the import threshold has been reached within the members importer. See more deets in refed issue
no issue
- After each test run there were csv files lefover which polluted the workspace.
- Ideally the write file would be some kind of a smart mock, so we'd won't have to do synchronous file remoal after each test but that's for later improvement if it becomes problematic
refs https://github.com/TryGhost/Team/issues/916
- These tests are nowhere close to perfect of even good. They barely pass "good enough" watermark... it's a start for new tests to be added easier in the future once the features are developed around CSV improrter
