Commit Graph

75 Commits

Author SHA1 Message Date
renovate[bot]
2223db5379
Update Test & linting packages 2023-03-13 02:36:20 +00:00
Daniel Lockyer
6b1966ad9b Updated sinon dependency
- this is being done manually instead of merging the Renovate PR because
  the PR bundles another bump which doesn't pass yet
2023-03-02 12:43:42 +01:00
renovate[bot]
9f08732039 Update @tryghost 2023-03-02 09:47:02 +01:00
Daniel Lockyer
2e051fb271
Fixed typo 2023-02-27 15:00:18 +01:00
Simon Backx
89ababb71f
Updated email event fetching to stop when begin and end are the same (#16326)
no issue

Optimization that makes sure we stop fetching when it is no longer
needed.
2023-02-23 15:44:01 +01:00
Simon Backx
923c522778
Implemented email analytics retrying (#16273)
fixes https://github.com/TryGhost/Team/issues/2562

New event fetching loops:
- Reworked the analytics fetching algorithm. Instead of starting again
where we stopped during the last fetching minus 30 minutes, we now just
continue where we stopped. But with ms precision (because no longer
database dependent after first fetch), and we stop at NOW - 1 minute to
reduce chance of missing events.
- Apart from that, a missing fetching loop is introduced. This fetches
events that are older than 30 minutes, and just processes all events a
second time to make sure we didn't skip any because of storage delays in
the Mailgun API.
- A new scheduled fetching loop, that allows us to schedule between a
given start/end date (currently only persisted in memory, so stops after
a reboot)

UI and endpoint changes:
- New UI to show the state of the analytics 'loops'
- New endpoint to request the analytics loop status
- New endpoint to schedule analytics
- New endpoint to cancel scheduled analytics
- Some number formatting improvements, and introduction of 'opened'
count in debug screen
- Live reload of data in the debug screen

Other changes:
- This also improves the support for maxEvents. We can now stop a
fetching loop after x events without worrying about lost events. This is
used to reduce the fetched events in the missing and scheduled event
loop (e.g. when the main one is fetching lots of events, we skip the
other loops).
- Prevents fetching the same events over and over again if no new events
come in (because we always started at the same begin timestamp). The
code increases the begin timestamp with 1 second if it is safe to do so,
to prevent the API from returning the same events over and over again.
- Some optimisations in handing the processing results (less merges to
reduce CPU usage in cases we have lots of events).

Testing:
- You can test with lots of events using the new mailgun mocking server
(Toolbox repo `scripts/mailgun-mock-server`). This can also simulate
events that are only returned after x minutes because of storage delays.
2023-02-20 16:44:13 +01:00
Simon Backx
48f9485f46
🐛 Fixed storing email failures with an empty message (#16260)
no issue

- When we receive an email failure with an empty message, the saving of
the model would fail because of schema validation that requires strings
to be non-empty.
- This adds more logging to the email analytics service to help debug
future issues
- Performance improvement to storing delivered, opened and failed emails
by replacing COALESCE with WHERE X IS NULL (tested and should give a
decent performance boost locally).
2023-02-13 15:25:36 +01:00
Daniel Lockyer
08b786af3b Bumped TryGhost-owned dependencies and lockfile
- this was all getting terribly behind so I've done several things:
  - majority of `@tryghost/*` except Lexical packages
  - gscan + knex-migrator to remove old `@tryghost/errors` usage
  - bumped lockfile
2023-01-02 20:55:22 +01:00
Simon Backx
47cd7a7095
🐛 Handled unknown Mailgun events (#15995)
refs https://ghost.slack.com/archives/C02G9E68C/p1670916538764019

- We receive events that don't have an emailId or providerId.
- We filter those events now and log them as an error
2022-12-14 11:17:45 +01:00
Daniel Lockyer
6f4e663d74
Updated @tryghost dependencies (#16005)
- also includes `knex-migrator` with a simple `sqlite3` bump
2022-12-14 11:18:55 +07:00
renovate[bot]
13abcf6c9d
Update dependency mocha to v10.2.0 2022-12-12 13:20:22 +00:00
Simon Backx
d8187123af
Added storage for email failures (#15901)
fixes https://github.com/TryGhost/Team/issues/2332

Saves events in the database and collects error information.

Do note that we can emit the same events multiple times, and as a result
out of order. That means we should correctly handle that a delivered
event might be fired after a permanent failure. So a delivered event is
ignored if the email is already marked as failed. Also delivered_at is
reset to null when we receive a permanent failure.
2022-12-01 10:00:53 +01:00
Simon Backx
f4fdb4fa6c
Added new email event processor (#15879)
fixes https://github.com/TryGhost/Team/issues/2310

This moves the processing of the events from the event-processor to a
new email-event-processor in the email-service package.

- The `EmailEventProcessor` only translates events from
providerId/emailId to their known emailId, memberId and recipientId, and
dispatches the corresponding events.
- Since `EmailEventProcessor` runs in a separate worker thread, we can't
listen for the dispatched events on the main thread. To accomplish this
communication, the events dispatched from the `EmailEventProcessor`
class are 'posted' via the postMessage method and redispatched on the
main thread.
- A new `EmailEventStorage` class reacts to the email events and stores
it in the database. This code mostly corresponds to the (now deleted)
subclass of the old `EmailEventProcessor`
- Updating a members last_seen_at timestamp has moved to the
lastSeenAtUpdater.
- Email events no longer store `ObjectID` because these are not
encodable across threads via postMessage
- Includes new E2E tests that test the storage of all supported Mailgun
events. Note that in these tests we run the processing on the main
thread instead of on a separate thread (couldn't do this because
stubbing is not possible across threads)

There are some missing pieces that will get added in later PRs (this PR
focuses on porting the existing functionality):
- Handling temporary failures/bounces
- Capturing the error messages of bounce events
2022-11-29 11:15:19 +01:00
renovate[bot]
8fa9f1e7e6
Update Test & linting packages 2022-11-07 20:39:48 +00:00
renovate[bot]
603c78755d
Updated @tryghost dependencies (#15631)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2022-10-24 10:30:40 +07:00
renovate[bot]
e9587e02d0
Update dependency mocha to v10.1.0 2022-10-17 08:41:28 +00:00
renovate[bot]
2c2ee81adb
Update Test & linting packages 2022-10-05 00:36:08 +00:00
Daniel Lockyer
54aa9f016b Fixed full Admin test suite running during unit tests
- because of how the npm scripts were set up, we were running the full
  Admin integration tests during the unit tests phase of CI
- this commit renames the majority of `test` to `test:unit` in the
  package.json files, and aliases `test` to `test:unit`
- special packages like Admin have no-op'd `test:unit` scripts so we
  don't end up running its tests
2022-08-15 15:34:52 +02:00
Simon Backx
82a3133ace
Added replies/likes count and limited replies to comments (#15206)
refs https://github.com/TryGhost/Team/issues/1723

- Added count.replies to comments
- Added replies endpoint
- Limited returned replies to 3.
- Replaced likes_count with count.likes in comments
- Instead of fetching all the likes of a comment to determine the total count, we'll now use count.likes
- Instead of fetching all the likes of a comment to determine whether a member liked a comment, we'll now use count.liked (which returns the amount of likes of the current member, being 0 or 1). This is mapped to `liked` to make it more natural to work with.

The `members.test.snap` file changed because we no longer include `liked: false` if we didn't fetch the liked relation. And in the comments events of the activity feed the liked property is therefore removed.

These changes requires an update to the `bookshelf-include-count` plugin:
- Updated to also work for nested relations
- This moves the count queries from the `bookshelf-include-count` plugin to the `countRelations` method of each model.
- Updated to keep the counts after saving a model (crud.edit didn't return the counts before)
2022-08-10 16:12:35 +02:00
Naz
fa8d94fce2 Fixed the typo
refs e9bfc4ef01

- Did a typo in the find and replace... and now correcting a typo of a typo  -_-
2022-08-04 15:38:32 +01:00
Naz
e9bfc4ef01 Changed the lingo to US of A variation
refs 16728a3ef1

- It's 'merica time!
2022-08-05 02:28:33 +12:00
Daniel Lockyer
308a28d31a
Tidied up package READMEs
refs https://github.com/TryGhost/Toolbox/issues/354

- these READMEs were migrated over from when each package was in a
  different repo
- they also assume you're going to be publishing the packages because it
  mentions install instructions
- only a few of them contain custom content
- this commit deletes the majority of these files because they're now
  not useful
- any that contained other instructions have been cut down
2022-07-25 15:17:12 +02:00
Daniel Lockyer
61125d7605
Removed repository from component package.json files
refs https://github.com/TryGhost/Toolbox/issues/354

- these repository links made sense when they were in different repos
  and published to NPM but we don't publish these packages any more
- this commit deletes those keys from the files
2022-07-25 11:15:16 +02:00
Daniel Lockyer
d73d7da7ef
Deleted non-root LICENSE files
- these were copied over during the monorepo conversion but we're not
  going to be publishing these packages so the top-level LICENSE file
  covers all packages here
2022-07-25 08:35:59 +02:00
renovate[bot]
d228144c27 Pin dependencies 2022-07-21 16:27:57 +02:00
Daniel Lockyer
48fd5ca8cb
Removed posttest linting step
- linting is done as a separate step in CI and it's a git hook upon
  pushing locally, so we don't need to run it after tests
2022-07-21 10:09:36 +02:00
Daniel Lockyer
ed1bdac20c
Removed publicConfig from packages config
- this was leftover from the migration to the monorepo and won't be
  needed because we're not publishing the packages
2022-07-20 17:49:52 +02:00
Daniel Lockyer
9fa789159c
Reset Publishing packages version and visibility
refs https://github.com/TryGhost/Toolbox/issues/354

- these packages are here for development and will be bundled when
  published, so they don't need versioning nor publishing
2022-07-20 17:20:24 +02:00
Matt Hanley
c11b0e822d Published new versions
- @tryghost/custom-theme-settings-service@0.3.3
 - @tryghost/email-analytics-provider-mailgun@1.0.9
 - @tryghost/email-analytics-service@1.0.7
2022-05-12 16:42:53 +01:00
Renovate Bot
a5174ff250 Update dependency sinon to v14 2022-05-09 02:31:27 +00:00
Renovate Bot
7d25bbec89 Update dependency mocha to v10 2022-05-02 00:44:17 +00:00
Renovate Bot
62af5754d0 Update dependency sinon to v13.0.2 2022-04-14 04:48:38 +00:00
Daniel Lockyer
3ecc15e02f Published new versions
- @tryghost/custom-theme-settings-service@0.3.2
 - @tryghost/email-analytics-provider-mailgun@1.0.8
 - @tryghost/email-analytics-service@1.0.6
2022-03-24 10:46:59 +00:00
Renovate Bot
1a1453dc34 Update dependency mocha to v9.2.2 2022-03-11 17:39:51 +00:00
Daniel Lockyer
f4e24a76e2 Added --all to c8 command
refs https://github.com/TryGhost/Toolbox/issues/203

- without `--all`, c8 ignores files that should be included in the
  coverage score but aren't used in tests
- this means we have artificially high scores in places where this
  isn't used
- this commit adds `--all` where previously missing
- where this fails `--check-coverage`, that has been removed for
  now
2022-02-21 13:34:08 +01:00
Renovate Bot
171e15bd13 Update dependency mocha to v9.2.1 2022-02-21 00:48:38 +00:00
Renovate Bot
8b4be67e21 Update dependency sinon to v13.0.1 2022-02-01 15:42:48 +00:00
Renovate Bot
9223d9d1ae Update dependency sinon to v13 2022-01-31 00:29:57 +00:00
Renovate Bot
0da6639d63 Update dependency mocha to v9.2.0 2022-01-24 12:09:30 +00:00
Renovate Bot
1fd73d9429 Update Test & linting packages 2022-01-20 16:21:37 +00:00
Renovate Bot
d8196129cb Update dependency mocha to v9.1.4 2022-01-14 20:50:48 +00:00
Joe Grigg
fc4fcfdcc3 Updated references to master branch to point to new default main branch
refs https://github.com/TryGhost/Toolbox/issues/162
2022-01-10 13:33:00 +00:00
John O'Nolan
7a8b94d1d8 2022 2022-01-06 13:42:05 +00:00
Sam Lord
a2a1144bf7 Published new versions
- @tryghost/email-analytics-provider-mailgun@1.0.6
 - @tryghost/email-analytics-service@1.0.5
2021-12-02 12:27:45 +00:00
Sam Lord
9c9d6c2340 Replaced GhostIgnition with @tryghost packages
refs: https://github.com/TryGhost/Toolbox/issues/146
2021-12-02 12:17:38 +00:00
Kevin Ansfield
8eeefab2e1 Published new versions
- @tryghost/custom-theme-settings-service@0.3.1
 - @tryghost/email-analytics-provider-mailgun@1.0.5
 - @tryghost/email-analytics-service@1.0.4
2021-10-26 10:15:36 +01:00
Daniel Lockyer
73f21a5065 Added codecov.io coverage uploader to CI
refs https://linear.app/tryghost/issue/CORE-74/improve-the-test-situation

- this commit adds the codecov GitHub Action into CI so we can upload
  coverage reports
- the coverage files need to be in XML for them to work with
  codecov, so this commit also adds cobertura (XML) as a reporter
2021-10-20 14:36:35 +02:00
Thibaut Patel
38e9f84d41 Published new versions
- @tryghost/custom-theme-settings-service@0.2.0
 - @tryghost/email-analytics-provider-mailgun@1.0.4
 - @tryghost/email-analytics-service@1.0.3
2021-10-20 11:46:57 +02:00
Renovate Bot
63668b8df3 Update dependency mocha to v9.1.3 2021-10-20 06:31:15 +00:00
Kevin Ansfield
5e40a03336 Published new versions
- @tryghost/custom-theme-settings-service@0.0.5
 - @tryghost/email-analytics-provider-mailgun@1.0.2
 - @tryghost/email-analytics-service@1.0.2
2021-09-28 15:51:13 +01:00