Commit Graph

32749 Commits

Author SHA1 Message Date
Daniel Lockyer
49ea1682d0
Added logging message when websockets client connects
- this helps with debugging because we can see when a client is
  connected
2023-02-20 11:31:33 +01:00
Sam Lord
37f16b1b17 Added roles to users in data generator
no issue
2023-02-20 09:26:28 +00:00
Sam Lord
36ea0cc6d6 Added labels to data-generator
no issue
2023-02-20 09:26:28 +00:00
Sam Lord
db7ac14467 Added offers to data generator
no issue

Can generate a pair of offers.

TODO: Add offers to the base-data pack
2023-02-20 09:26:28 +00:00
Sam Lord
22ac685354 Added pages to data-generator
no issue

Data generator will now add 3 pages by default - can add extra support here, e.g. webmentions later.
2023-02-20 09:26:28 +00:00
Sam Lord
c6e119ecf5 Allow data-generator to make scheduled and draft posts
no issue

Also adds visiblity options, paid, members-only and public
2023-02-20 09:26:28 +00:00
Naz
bb855cd462
Fixed timers in ttl cache tests
refs 95530a6617

- The timers on CI are too sensitive and failing tests. Made wait times relatively longer as a fix
2023-02-20 15:41:10 +08:00
Naz
95530a6617
Added optional in-memory TTL cache
refs https://github.com/TryGhost/Toolbox/issues/515

- We don't have a good way to test TTL caches without setting up Redis in the environment
- Adding in-memory cache adapter with TTL allows to run tests on CI without having to install Redis
- Also, TTL in memory cache can be a great substitution for Redis-based caches on instances that
have a lot of spare RAM and don't need to use Redis necessarily
- MemoryTTL cache accepts two parameters "TTL" and "max"
- TTL - is time in milliseconds to hold the value for in cache
- max - is the maximum amount of items to keep in the cache

- To use MemoryTTL cache specify following config in the cache section:
```
    "adapters": {
        "cache": {
            "imageSizes": {
                "adapter": "MemoryTTL",
                "ttl": 3600
            }
        }
    }
```
- Above config would apply MemoryTTL cache to imageSizes feature with TTL fo 3600 ms
2023-02-20 15:16:33 +08:00
renovate[bot]
77a65fee61
Update dependency sanitize-html to v2.10.0 2023-02-20 00:24:48 +00:00
renovate[bot]
61fd9cc74b
Update dependency c8 to v7.13.0 2023-02-17 21:21:52 +00:00
renovate[bot]
6703a0e91c
Update dependency i18next to v22.4.10 2023-02-17 16:06:34 +00:00
Ghost CI
dc51f4703f v5.35.0 2023-02-17 16:00:47 +00:00
Sam Lord
10a4a8eb12 Fixed tests for demo-data
no issue
2023-02-17 15:08:01 +00:00
Sam Lord
ca395a958c Added post analytics to data generator
no issue
2023-02-17 14:03:59 +00:00
Simon Backx
ffc0cc020a Updated coverage excemption list
Removed the wrapper class for the email service from coverage, because this only wires up a lot of dependencies, which is hard to test in a unit test because we also have to init all the dependencies in a unit test. It is already covered by E2E tests.
2023-02-17 14:27:59 +01:00
Simon Backx
ec3f262ea4 Increased scheduling default test timeout 2023-02-17 13:29:07 +01:00
Fabien "egg" O'Carroll
8908a51547 Improved verification logic for Mentions
refs https://github.com/TryGhost/Team/issues/2550

By using cheerio to parse the HTML we can correctly look for elements
which use the target URL as the href attribute, rather than doing a
plaintext search. This closer to what the spec says.
2023-02-17 19:07:11 +07:00
Fabien "egg" O'Carroll
e14d2e662b Wired up verification to MentionsAPI
refs https://github.com/TryGhost/Team/issues/2550

Whilst this isn't ideal making multiple requests for the same site
it's a first step towards verification properties, and can be
refactored to improve performance later. With the current volume of
incoming Webmentions we've seen so far this shouldn't be a problem.
2023-02-17 19:07:11 +07:00
Fabien "egg" O'Carroll
37cfb96d9f Added support for verified property to BookshelfMentionRepository
refs https://github.com/TryGhost/Team/issues/2549

This ensures we can persist the verification of a Mention to MySQL/SQLite3
2023-02-17 19:07:11 +07:00
Fabien "egg" O'Carroll
277a563138 Added ability to set verified when instantiating Mention
refs https://github.com/TryGhost/Team/issues/2548

We need to be able to do this outside of the verify method so that
repository implementations have the ability to create existing
instances without running expensive verification checks!
2023-02-17 19:07:11 +07:00
Fabien 'egg' O'Carroll
101cabb188
Added separate job service for webmentions (#16285)
[Added initial mentions-jobs
service](3656190114)

This is the result of running `cp -r jobs mentions-jobs` in the services
directory.

[Waited for mentions-jobs queue before
shutdown](2bb1a12a89)

This matches the functionality of the existing jobs service where we
will wait
for jobs to complete before closing the process.

[Used mentions-jobs service in the mentions
service](4e4f9fdd00)

This ensures that any delays in the mentions jobs queue does not effect
other
parts of the application.
2023-02-17 13:05:36 +01:00
Daniel Lockyer
2edfd8b8e0
Increased timeout to allow for delays in browser e2e test
- this test was randomly timing out, and the tests above allow for a
  greater timeout before failing
2023-02-17 12:13:09 +01:00
Aileen Booker
2f57e95a5d
Slack notifications service for Milestones behind flag (#16281)
refs
https://www.notion.so/ghost/Marketing-Milestone-email-campaigns-1d2c9dee3cfa4029863edb16092ad5c4?pvs=4

- Added a `slack-notifications` repository which handles sending Slack
messages to a URL as defined in our Ghost(Pro) config (also includes a
global switch to disable the feature if needed) and listens to
`MilestoneCreatedEvents`.
- Added a `slack-notification` service which listens to the events on
boot.
- In order to have access to further information such as the reason why
a Milestone email hasn't been sent, or the current ARR or Member value
as comparison to the achieved milestone, I added a `meta` object to the
`MilestoneCreatedEvent` which then gets accessible by the event
subscriber. This avoid doing further requests to the DB as we need to
have this information in relation to the event occurred.

---------

Co-authored-by: Fabien "egg" O'Carroll <fabien@allou.is>
2023-02-17 12:59:18 +02:00
Steve Larson
034a230365 Added alpha feature to demonstrate websockets
refs https://github.com/TryGhost/Team/issues/2561

- added simple socket-io implementation to Ghost server
- added alpha flag for websockets
- added route in admin to test websockets using a simple counter stored in server local memory (refreshes on reboot)
2023-02-17 11:55:15 +01:00
Daniel Lockyer
e52efb31b8
Updated coverage exemption list
- like the members jobs, we don't need to run unit tests on these
  because they're not library code
2023-02-17 11:43:13 +01:00
Steve Larson
1e00bd2af9 Handled empty content when sending webmentions
refs https://ghost.slack.com/archives/C02G9E68C/p1676564978732119

- `cheerio` errors when trying to parse `null`
2023-02-17 10:46:48 +01:00
Naz
f27c7ba044
Fixed variable name typo 2023-02-17 17:27:51 +08:00
Naz
3ddfe59e15
Added resource caching to Posts/Pages Content API
refs https://github.com/TryGhost/Toolbox/issues/522

- Browse endpoints for Posts and Pages are creating the most database traffic in the system. These are read-only endpoints that don't have to be fresh 100% of the time. Having optional cache allows to offload some of the database querying to more efficient storage.
- To enable cache for Posts/Pages browse endpoints there are two prerequisites:
  - set 'hostSettings:postsPublicCache:enabled' to 'true' in the configuration file
  - add 'postsPublic' cache adapter in cache configuration

- Example config for adapters with 60s TTL for a cached resource:
```
    "adapters": {
        "cache": {
            "postsPublic": {
                "adapter": "Redis",
                "ttl": 60,
                "keyPrefix": "site_id_here:posts-content-api:"
            }
        }
    },
```
2023-02-17 17:27:51 +08:00
Naz
c3d7104367
Refactored tags repository to be model-agnostic
refs https://github.com/TryGhost/Toolbox/issues/522

- The caching rules for Content API resources like Tags / Posts / Pages / Authors are the same, so it makes sense to make a common repository which would take in a model as a parameter and perform the queries on that level instead of implementing repository-per-model.
- With this refactor getting Posts Content API caching would be much simpler change.
2023-02-17 17:27:51 +08:00
Steve Larson
5d00b64e27
Added timeout to external requests
no refs

- got doesn't have a timeout by default
- we think we might have hanging jobs because of this
2023-02-17 09:16:58 +01:00
Daniel Lockyer
9a2d765c73
Added one-off job name and current queue length into log line
- this makes it easier to know what jobs are being scheduled and what
  the current queue length is
2023-02-17 09:16:40 +01:00
Aileen Nowak
cd3930d08e Promote milestoneEmails to beta feature
no issue

- Promotes `milestoneEmails` to a beta feature so it can be a private beta to be tested
2023-02-17 07:45:51 +02:00
Aileen Nowak
1c0ee675fe Updated milestone values in default config 2023-02-17 07:45:51 +02:00
renovate[bot]
7de35f1ca3 Update dependency node-jose to v2.2.0 [SECURITY] 2023-02-17 05:29:46 +00:00
renovate[bot]
cf7ecb492c
Update dependency c8 to v7.13.0 2023-02-16 22:15:50 +00:00
Sam Lord
b4c90bf07d Reduced number of login events in large-scale data generation
no issue

This should massively increase the speed of importing for the large dataset, which is important as the time to import it on Pro is >10 minutes at the moment
2023-02-16 20:54:14 +00:00
renovate[bot]
9bd2b8c708 Update dependency lint-staged to v13.1.2 2023-02-16 15:01:58 +01:00
renovate[bot]
3e4cb3f85e Update peter-evans/create-or-update-comment digest to 300bfad 2023-02-16 15:01:36 +01:00
renovate[bot]
38642af435 Update dependency cssnano to v5.1.15 2023-02-16 14:59:59 +01:00
Sam Lord
b8f59f07f2 Added --clear-database option to make it easier to use the data generator
no issue
2023-02-16 12:11:08 +00:00
Simon Backx
f8e77ad012 🐛 Fixed uploading images with custom storage adapters
fixes https://github.com/TryGhost/Ghost/issues/16278

`urlToPath` is not a method that is defined on the StorageBase, and thus is not implemented on any custom storage adapter.

We need this method to prevent uploading a file in two different directories. Currently this is an edge case:
- If you upload a file at midnight between a month shift, it is possible that we upload one file at `/2023/02/file.png` and the original at `/2023/03/file_o.png`
- To make sure we upload the second at the same directory, we need the `urlToPath` implementation. The save method sadly only returns the URL, which could be different than the stored file path, so we cannot use that return value.

This fix only uses `urlToPath` if available, to prevent above issue.
2023-02-16 11:49:37 +01:00
Simon Backx
8dfa490638
Added outbound link tagging for web posts (#16201)
fixes https://github.com/TryGhost/Team/issues/2433

- Moved all outbound link tagging code to separate OutboundLinkTagger
- Because a site can easily enable/disable this feature, we don't store
the ?refs in the HTML but add them on the fly for now in the Content
API.
2023-02-16 11:26:35 +01:00
Fabien "egg" O'Carroll
a310ec2e3e Fixed Mention unit test
refs https://github.com/TryGhost/Ghost/commit/c6b90b6ac
2023-02-16 14:37:22 +07:00
Fabien "egg" O'Carroll
c6b90b6ac7 Added verified property to Mentions Admin API
closes https://github.com/TryGhost/Team/issues/2551

Rather than blindly passing all data through the API we explicitly include each
new property. This allows us to make changes to the core entities without
affecting the API. The verified property is being added now to give design the
ability to display these mentions differently.

We also needed to include the verified property in the return value of toJSON,
this was missed as part of the original entity changes
2023-02-16 14:31:05 +07:00
Ghost CI
dd48f65712 Merged v5.34.1 into main 2023-02-16 07:23:52 +00:00
Ghost CI
443085ab87 v5.34.1 2023-02-16 07:23:49 +00:00
Naz
8a0c20b54c
Fixed Redis cluster ttl configuration
refs https://github.com/TryGhost/Toolbox/issues/520

- The cluster config is taking over local adapter ttl configuration - the priority should be reverse adapter config first followed by cluster config
- In addition if we add nested config merging to adapter manager we could achieve the same effect by having per-adapter configuraiton with "clusterConfig.options.ttl" value
2023-02-16 14:13:44 +08:00
John O'Nolan
14c5e48190
Improved micro copy on Portal email tips 2023-02-15 12:30:03 -04:00
Aileen Nowak
5f12c7ad40 Updated Milstone test to expect no event for existing milestone 2023-02-15 14:28:57 +02:00
Aileen Nowak
dc8dc90212 Exposed MilestoneCreatedEvent as module 2023-02-15 14:28:57 +02:00