no ref
We've shipped quite a few updates to our i18n package these past couple
weeks and are due for an update to the consuming packages. Special
thanks to our community contributors for the translations, and in
particular to @cathysarisky for their continued effort!
This PR contains the following updates:
| Package | Type | Update | Change |
|---|---|---|---|
|
[parse-prometheus-text-format](https://redirect.github.com/yunyu/parse-prometheus-text-format)
| devDependencies | pin | [`^1.1.1` ->
`1.1.1`](https://renovatebot.com/diffs/npm/parse-prometheus-text-format/1.1.1/1.1.1)
|
Add the preset `:preserveSemverRanges` to your config if you don't want
to pin your dependencies.
---
### Configuration
📅 **Schedule**: Branch creation - "every weekday" (UTC), Automerge - At
any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR is behind base branch, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/TryGhost/Ghost).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOC45Ny4wIiwidXBkYXRlZEluVmVyIjoiMzguOTcuMCIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOltdfQ==-->
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
ref
https://linear.app/tryghost/issue/ENG-1505/add-prometheus-metrics-server-to-allow-monitoring-ghost-metrics
# Summary
This commit includes two main components: a prometheus client class to
collect metrics from Ghost, and a standalone metrics server that exposes
a /metrics endpoint at a separate port (9416 by default) from the main
Ghost app.
The prometheus client is a very thin wrapper around
[prom-client](https://github.com/siimon/prom-client). We could use
prom-client directly, but this approach should make it easier to switch
to a different prometheus client package (or make our own) if we ever
need to down the line.
The list of default metrics this enables is specified in an e2e test
[here](https://github.com/TryGhost/Ghost/pull/21192/files#diff-ebc52236be2cd14b40be89220ae961f48d3f837693f7d1da76db292348915941R66-R92).
This also gives us the ability to create and collect custom metrics,
although none are included in this commit yet.
# Configuration
The prometheus client and the metrics server are both disabled by
default, but can be enabled by setting the metrics_server:enabled flag
to true.
You can also define a custom host and port using `metrics_server:host`
and `metrics_server:port`.
## Why not expose the /metrics endpoint in one of the existing express
apps?
The standalone express app exists for two main reasons:
1. We don't want these metrics to be public, and the easiest way to
accomplish that is to expose the /metrics endpoint at a different port
that won't be exposed to the internet.
2. Creating a standalone express instance decouples the metrics endpoint
from the Ghost server, so if Ghost is not responding for whatever
reason, we should still be able to scrape metrics to understand what's
going on internally.
## Impact on Boot & Shut down time
The prometheus client is initialized early in the boot process so we can
collect metrics during the boot sequence. Testing locally has shown that
this increases boot time by ~20ms. The metrics server which exposes the
/metrics endpoint is not initialized until after the background
services, and it is not awaited, to avoid impacting boot time. None of
this code, including the requires, will run if the
metrics_server:enabled flag is set to false (or not set).
Shutting down the metrics server is added as a cleanup task for the main
Ghost server instance, and is setup to shut down with 0 grace period to
avoid impacting shut down time.
no issue
- These two tests were relying on the ordering of the posts in the posts
list, which can vary when running the test suite all together (as we do
in CI), since multiple tests can be running and creating/deleting posts
in parallel.
- This commit navigates directly to the post's URL to avoid this
problem.
no ref
Expose (some) Portal error strings for translations
💩This is a somewhat hacky (but test-passing and individual inspection
passing) solution to the way Portal handles errors. Or rather, the
half-dozen ways Portal handles errors.
Passing 't' around with context and state, and occasionally recreating
it from the site object. Yes, I am also somewhat horrified, but a better
implementation will need a major rewrite of Portal.
Addresses errors in both the popover React app and in the
data-attributes.
There are probably more. Since Portal exposes raw API responses in some
places, it's hard to enumerate everything that /might/ end up being
client-facing, but at least I've gotten the ones that I've commonly
seen.
Improvements very welcome.
no issue
Initial pass at refactoring the tests and making each independent so they aren't relying on data from previous tests.
- DRYed up some repeated API requests by extracting `testPostComment()`, `testGetComments()`, and `assertAuthorEmailSent()`
- removed data inter-dependency across tests
- truncated the tables under test before each test is run
- added suite of db fns to populate the database without having to do so via API requests
- updated all tests to include their own data setup calls rather than relying on fixtures and modifications from earlier tests
no ref
We had an instance where this was a ms off and I should've used mock
timers when I first wrote this. This should prevent any rare clock
mishaps.
no ref
Translated/updated all relevant strings in the search & portal feature
to Bahasa Indonesia.
---------
Co-authored-by: Cathy Sarisky <42299862+cathysarisky@users.noreply.github.com>
ref https://linear.app/tryghost/issue/ANAL-96/data-discrepancy-between-charts-when-filtering
- This adds a set of tests to describe what the data should look like when we filter on various values
- We have tests for source and browser which are pulled from different MVs
- The result files are generated using ./scripts/gen_test_results.sh, and then manually verified
- We know they are not yet fully correct
- Added yarn command to update TB CLI, as that needs doing frequently and I can never remember the command
- Improved safety & usability of tinybird test script by ensuring branches are correctly created before running & adding optional delete
- Updated tinybird test to warn only for sanity check as that's not always a valid check (Will prob remove soon)
- Improved output of tinybird test script on failure, so that the diff is readable and closer to what git shows you
- Added tool to convert tinybird ndjson to csv to make it easier to bring the data into google sheets for verifying numbers
no issue
- This test file starts a Ghost server, but doesn't close it, which can
cause other tests to fail when they try to start an instance of Ghost,
with an `EADDRINUSE` error.
- This change closes the server in the `after` hook
no ref
This PR adds the translations for `search.js`. It's a bit opinionated in
that ...
- it uses the Anglicism "Posts" to ... "translate" ... "Posts" 😆;
- it uses an asterisk to make sure we address all genders when
translating "authors".
Happy to discuss. ☺️
no ref
This is a merge & tidy up of [[ayangizzat](https://github.com/ayangizzat)] 's original (https://github.com/TryGhost/Ghost/pull/20698), which was missing the changes to locale, causing folders & files not to generate correctly. All credit goes to @ayangizzat , and thank you for your work!
Kazakh-speaking help still needed for search.json, which is untranslated, and the addition of a few other new strings, and refinements anywhere needed. Please add comments if not yet merged, or open a new PR if already merged.
---------
Co-authored-by: ayangizzt
no ref
We've had several PRs for new locales that forgot to update i18n.js,
which causes them not to get regenerated by yarn translate. This PR
checks for that error, and also the reverse one where they update i18n
but we're missing folders. (Never expected to error, because 'yarn test'
is defined to include 'yarn translate', which generates the folders.)
ref 86d61304b1
ref https://linear.app/tryghost/issue/ONC-323
- added `tracked()` to our proxy model object properties
- fixes default data always showing when opening the modal
- fixed data push after completing modal
- `post.tiers` is set up as an attribute in Admin rather than a relationship
- fixes incorrect tiers list showing when the change access modal is opened again after changing access before the post is re-fetched from the API
- fixed flash of failure button state when saving modal changes
- expanded tests to cover tiers selection
[ANAL-95](https://linear.app/tryghost/issue/ANAL-95/internal-beta-qa)
Various design refinements and fixes for the Stats page:
- Updated scroll area in detail modals so that the Close button and the footer is never outside the viewport
- The detail modal didn't close after clicking on the filter values
- "Show all" button was displayed also when there were no new items in the detail modal
- Dropdown styles needed a visual update: the toggles were way too huge and inconsistent with other dropdowns
- If no audience was selected we still showed stats. Now it's displaying the default empty screen in this case
- Click through filter indicators had low discoverability
- Technical data styles needed some love: changed the alignment and color scheme
- Mobile size viewports were not handled
- The google favicon API returned 404 many times for sources. Swapped the service for another one that returns favicons more reliably
- Default favicon was not handled. Now it comes from static.ghost.org
no ref
This very small helper adds {{content_api_key}} to the collection of handlebars helpers available to theme creators. This will make it easier for themes to access the content API key, without either requiring the user to get it from the integrations page and input it on the theme setting page or resorting to JavaScript to read it from one of the built-in script tag attributes -- both ugly workarounds.
ref https://linear.app/tryghost/issue/ONC-323
After changing a post's access via the posts list context menu, creating new posts or members would not work correctly.
- the issue stemmed from `this.post.set('currentState.parentState.isNew', false);` that was called when changing a post's access level, after that all Ember Data models created from the store would have `isNew: false` causing Ember Data to attempt a PUT request to update the not-yet-created model rather than a POST request to create it
- we were only using a real post model instance in order to run validations against the post access level settings but we can do that just as easily by creating a new object and injecting our validation mixin
no ref
This PR adds and adjusts various German translations in `portal.json`.
I'm a native speaker and have made adjustments to the best of my
knowledge. As in #21163, the gender variations are a bit opinionated and
I'm happy to discuss.
fixes#21104
This PR corrects a single translation. The German translations for "sign
up" and "sign in" _can_ both be "anmelden", but are confusing when used
together. So here we're switch to "register now" and "sign in" to make
the distinction clearer. IMHO even in English "register now" and "sign
in" might be a better (more distinguishable) combination, but it's much
harsher in German.
no ref
This PR restores a few strings that were renamed during the time the
Greek strings were not being correctly processed.
---------
Co-authored-by: Vikas Potluri <vikaspotluri123.github@gmail.com>
ref https://linear.app/tryghost/issue/ONC-323
- we're sometimes seeing our force-refresh failing when Ember Data gets into a bad state but we're not sure why so this log should tell us if it's the browser's native "leave site" modal that is preventing the refresh
- updated the `onbeforeunload` event handler to match modern JS approach
- modern browsers use `event.preventDefault()` to show their dialog
- older browsers use `event.returnValue = true` (this is what our old string return was triggering)
- no browser supports a custom message in the native dialog
- TODO: make these run in CI
- Right now you run them by running `yarn tb` and then `./script/branch_and_test.sh`
- These are snapshot tests that check we get the desired result
Co-authored-by: alejandromav <hi@alejandromav.com>
- Apparently several PRs for new languages were merged without the
corresponding additions to i18n.js, which has caused new strings not to
be generated for those languages, and missing translations in the
frontend, even though we had them in the .json file.
- I've added language codes for all languages that had at least some
strings translated in i18n/locales, and have run yarn translate to
create missing strings and files for these languages.
New language codes for:
* Arabic
* Swiss German
* Greek
* Estonian
* Macedonian
* Serbian (Cyrillic)
* Thai
no issue
- This commit removes all OpenTelemetry related code and dependencies
from Ghost.
- The initial implementation was done as a POC but it raised some
performance concerns with boot time, so we never actually enabled it
widely.
- We can revisit this in the future, but in the meantime it's just
adding unnecessary dependencies and bloating the codebase.
ref https://ghost-foundation.sentry.io/issues/5908152800/
- In the current state, we are maintaining an 'index' key for all
revisions in localStorage. This gives us quick and easy access to all
the revisions in localStorage, but it requires additional "bookkeeping"
to update the index each time we add/remove a key.
- In some obscure edge cases, this results in the `remove()` method
throwing a `QuotaExceededError` (since removing a revision also requires
updating the index with `localStorage.setItem()`). If the `remove()`
call fails, we are sort of stuck — the only way to reduce our storage
usage is to remove items, but if the `remove()` method throws errors, we
can't do that.
- This change removes the whole index concept, and instead loops over
all the keys in localStorage, filtering by the prefix to find all our
revisions. This makes the `keys()` method slightly more complex, as it
has to filter out keys in localStorage that aren't related to revisions,
but it simplifies saving and removing revisions.
- Critically, this also means that `remove()` should never throw a
`QuotaExceededError`, since it no longer needs to call
`localStorage.setItem()` — it now simply calls
`localStorage.removeItem()` for the revision, which should never fail.
ref https://linear.app/tryghost/issue/AP-438
This is going to allow us to load the activitypub package from the jsdelivr
cdn, which means we can release new versions without releasing the admin.
no issue
Either a node or macOS update resulted in our broken image upload tests causing native stack traces:
```
# /Users/kevin/.nvm/versions/node/v20.16.0/bin/node[8841]: static void node::Blob::ToSlice(const FunctionCallbackInfo<v8::Value> &) at ../src/node_blob.cc:248
# Assertion failed: args[1]->IsUint32()
```
- updated our `blob.slice()` calls to ensure the second argument is always an integer rather than possibly a float
This pull request adds Estonian (et) language translations to Ghost.
Translated files:
- comments.json
- ghost.json
- portal.json
- search.json
- signup-form.json
Changes:
- Created new 'et' folder in the locales directory
- Translated all strings from English to Estonian in the above files
I'm a native Estonian speaker, but if there are any questions about
specific translations, please let me know.
Thank you for considering this contribution to make Ghost more
accessible to Estonian-speaking users!
---------
Co-authored-by: Ronald Langeveld <hi@ronaldlangeveld.com>
A needed update of the Bulgarian version. Fixing some errors. Made some
improvements. And some new strings were translated.
---------
Co-authored-by: Ronald Langeveld <hi@ronaldlangeveld.com>
ref ONC-364
- Adds a condition to check whether the record is deleted or if deleting
is in progress before firing the `setFeatureImageCaption`.
- Adds tests. Managed to reproduce the issue using tests.
no issue
- Added Sentry logs to capture how often we are running into
`QuotaExceededErrors` when saving local revisions to localStorage, to
help in deciding if localStorage is sufficient, or if we need to expand
to e.g. IndexedDB.
- Also adds some handling to ignore errors when calling
`localStorage.setItem()` elsewhere in the admin app to avoid crashing if
localStorage isn't supported or the quota is exceeded.
ref https://app.incident.io/ghost/incidents/107
ref cc88757e2a
- added new path in admin `/restore`
- added basic ui for restoring posts from local storage
- added limits for # of revisions for posts with an `id` (5 revisions)
This commit adds a simple UI for restoring posts in case of data loss.
This is a backstop for very rare situations in which it seems Ember gets
into a conflicted state. See ref'd commit for more info. Clicking
'Restore' will create a new post with the saved off content.
ref
https://linear.app/tryghost/issue/ANAL-60/click-through-filtering-for-sources
- In our stats page we use the referrer without a protocol or www, that
is the pure domain as our source that we output
- Meanwhile all the data pipelines had the full url as the referrer
passed through
- When we come to add clickthroughs/filtering, we'll need to use this
value to filter the data. If we have a different value locally in the UI
to what is in the DB, we won't be able to make the filters match
- Also, we pay for everything we store, and this removes all the
https:// and www. data
- Whilst we are in development, we can safely make changes to all
aspects of our pipeline without worrying
- This is because currently, it's safe to delete all data and start over
- This script removes everything excepts the analytics_events
datasource, and then recreates everything fresh, repopulating from the
datasource where possible
- This shouldn't be used after tinybird is in production, we need a
better change process
ref https://github.com/TryGhost/Ghost/issues/16628
This adds translation support to search, which should be the last missing piece of i18n support for Ghost's frontend 🎉
- Translation (t) helper added to sodo-search.
- Ghost head tweaked to include data-locale.
- All (I hope) strings in sodo-search wrapped in the t helper.
- Possibly poor-quality French translation strings added.
---------
Co-authored-by: Vikas Potluri <vikaspotluri123.github@gmail.com>
ref
https://linear.app/tryghost/issue/ANAL-53/10-stats-page-engineering-stuff
- All of these changes are intended to improve developer experience
going forward, to make it easier to implement further changes to the
stats page.
- Moved the modal into the stats components, as it has a lot of shared
code, and it makes it easier to update them all
- Removed various comment blocks that were outdated or didn't really add
value
- Fixed all imports to use the same pattern starting `ghost-admin/`
- Ensured all the components had the correct name
- Dried up the generation of params for Tinybird charts into a utility
function as we'll need to add several more in the near future
- Tried to use a consistent pattern everywhere for the order of
operations
- Dried up the implementation of technical.js which handles the
device/browser charts
REF PLG-226
- Changed title copy from "You want to report this comment?" to "Report
this comment?"
- Changed button copy from "Report this comment" to "Report"
- Updated styles to be more responsive
DES-774
- The buttons in the Admin got very diverse over the last couple of
years. This PR updates the styles to use outline buttons, white
background for better contrast and slightly more rounded corners.
Additionally the right click and dropdown menu typography and spacing
has also been updated.
ref https://linear.app/tryghost/issue/ONC-323
When the store gets into a bad state for new posts that causes saves to fail we can detect that by looking at the `model.isNew` property. Currently our best approach to fix this state is to restart the app.
- added a `didTransition()` hook to our `lexical-edit.new` route
- detects the bad state, logs the error, and triggers a browser refresh
- logs with a `recreatedPostIsGood` property that will let us know if we could instead just try recreating the post and avoiding a full refresh (so far we have no reproduction case so we need to learn what we can)
- added `sinon-chai` dependency for better assertions on spies/stubs
- added `sentry-testkit` dependency so we can test our Sentry integration calls
- we can't use sinon for these calls because of the way Sentry's es6 imports work
- extracted our full Sentry config object generation to a util function so it can be re-used in unit tests
- updated our integrations list to disable the default `dedupe` integration because it can cause very unexpected/difficult to debug test failures when you're asserting using `sentry-testkit`
REF DES-770
- In certain email clients such as Protonmail, the newsletter title
line-height was inherited from the `body` rather than the parent `td`.
This commit adds line-height to the title link explicitly.
ref https://linear.app/tryghost/issue/ONC-323
- sometimes posts can be deleted by another user or in a different tab but then edited in an old tab that had the post loaded in the editor
- in this situation we were displaying our "Editor crashed" error put in place for the rarer situation where the editor is genuinely in a bad state
- added an extra conditional for the bad state and a custom error message for the deleted post state
ref https://app.incident.io/ghost/incidents/107
- We have a rare bug that causes the initial `POST` request to create a
new post from the editor to be skipped or fail. Subsequent `PUT`
requests then fail because there is no post ID, potentially resulting in
data loss. The aim of this commit is to start saving revisions of posts
in the editor to the browser's localStorage, as a last-ditch option to
restore lost work.
- Since we don't know where the bug is yet, and to protect against
future bugs, we've deliberately avoided depending too heavily on the
`lexical-editor` controller or the ember store. We've aimed to create a
direct route to the state in the editor, by hooking into the
`updateScratch` method (effectively the `onChange` handler for the
editor).
- The `scheduleSave` function on the new `local-revisions` service is
called immediately upon any changes to the state of the lexical editor,
which is effectively every keystroke. The service has some logic and
timeouts, so it doesn't actually save a revision on every change to the
editor.
- The "schema" of the datastore is a simple key-value store, where the
key is of the format: `post-revision-${postId}-${timestamp}` if the post
has an ID, or `post-revision-draft-${timestamp}` for an unsaved draft.
There is also an array of all the revisions' keys, which allows us to
clear all the revisions without having to loop over every key in
localStorage (along with some other conveniences, like filtering).
- There is currently no UI for viewing/restoring revisions. In the event
that you need to restore a revision, you can access the service in the
browser console. You can access all the saved revisions using the
`list()` method, which logs all the revisions to the console by title &
timestamp. You can then choose a revision to restore, and call
`restore(revision_key)`, which will `POST` the revision's data to the
server to create a new post.
- Since localStorage data is limited to a 5mb quota in most browsers,
the service has a mechanism for evicting the oldest revisions once it
meets the quota. If a save fails because it would exceed the quota, the
`performSave` method will evict the oldest revision, then recursively
try to save again.
---------
Co-authored-by: Steve Larson <9larsons@gmail.com>
ref https://linear.app/tryghost/issue/ONC-323
- added debug logs to print to console each time the post state changes and include a full list of post state changes within the editor session in the error reports when we hit the 404 error caused by a bad editor state
closes https://linear.app/tryghost/issue/ENG-1533
- the code to switch to "from analytics" state for the editor was applying when clicking the create post button in the nav menu whilst on the analytics screen which was confusing because you not only lost the `< Posts` link in the editor but you couldn't see the post's saving status
closes https://linear.app/tryghost/issue/ANAL-77/na-data-should-be-zero
ref https://www.tinybird.co/blog-posts/tips-9-filling-gaps-in-time-series-on-clickhouse
- Sometimes we have no matching data for a particular date/date range, which makes our charts look super janky
- Clickhouse has a feature to fill these in called WITH FILL, which makes it really easy to fix this!
- WITH FILL works except for on bounce rate. That seems to be due to the column being marked as nullable and so WITH FILL fills missing data with NULL instead of 0
- To fix that, I've updated the code that generates the bounce rate so that it doesn't generate nulls, and that seems to result in a not-nullable column, which then works with WITH FILL
- When I went off, I quickly recreated all our endpoints with some new functionality
- However, I forgot that I was manually managing tokens, this meant the UI for stats broke with a token error
- Adding the tokens to the endpoint definitions should prevent this happening again, by automating the management of the token scopes
ref https://linear.app/tryghost/issue/ONC-323
- the post model state appears to be in an odd situation when this issue occurs, the extra log context should help us determine if the bad state is occurring at the route level or inside the editor controller
Ref: https://linear.app/tryghost/issue/SLO-188/set-a-maximum-limit-for-get-members-api
Endpoint: ghost/api/admin/members/?limit=all
Change Overview: We are updating the GET ghost/api/admin/members/ endpoint to remove support for the limit=all parameter. Previously, a request like GET ghost/api/admin/members/?limit=all would return a list of all members. Going forward, any request with limit=all or a limit greater than 100 will only return up to 100 members per request.
This change aims to improve the performance and scalability of the API.
What changes for users? - They will have to implement pagination to retrieve the list of all members.
refs https://linear.app/tryghost/issue/DEV-23/workaround-for-yarn-caching-issues
- in our build pipeline, we add some more dependencies for our internal
adapters
- recently we've been seeing caching issues with these dependencies, not
sure why
- to workaround that, we'll just include them here and eventually bring
the adapters into the OSS repo
ref https://linear.app/tryghost/issue/ENG-661
Logging the full lexical objects to Sentry for the "showing leave editor modal" event isn't very useful because they almost always truncated due to size or stripped due to potentially sensitive data which makes the reports difficult to debug and action.
- added `code` to the context for each report to make identification easier
- updated `reason` for the lexical diverged reason to better match what's happened
- stripped `lexical`, `scratch`, and `secondaryLexical` from the context that is logged to Sentry because they aren't actionable there and just add noise
- added a diff to the logged context for the lexical change reason to more easily identify the changes that triggered an unexpected modal display
- previously we didn't get full objects in Sentry so couldn't do a comparison and the local workflow was to grab the logged scratch and secondaryLexical values and run a manual diff - this should help in both cases
ref https://linear.app/tryghost/issue/ENG-1466
ref https://linear.app/tryghost/issue/ENG-1484
- Previously, filtering members with multiple "Unsubscribed from
newsletter x" led to no filtering at all, all members were returned
- This was caused by a bug in NQL, that is fixed in version 0.12.5, cf.
[commit](dd18d1d6ca)
- We're also removing the safeguard in the product around bulk deletion
when multiple newsletter filters are in use, as the root problem has
been fixed
ref https://linear.app/tryghost/issue/ONC-323
- our fallback 404 error handler assumed we always had a transition along with the error
- this wasn't a bad assumption, it should be very unlikely that we see a 404 outside of navigating to a non-existent/deleted resource
- unfortunately we weren't handling the error thrown by our error handler which meant the error was silent as far as the user was concerned
- having a silent error meant that in very rare circumstances the editor could get into a state where saving was failing but there was no indication of that
- updated the fallback 404 error handler to only do something when navigation was occurring in which case it transitions to the 404 screen, otherwise let the error continue to our generic API error handling which will stay on the current screen but show an error alert
- adjusted the editor saving to actually trigger autosave-after-change when testing (albeit with 100ms wait compared to 3s) so the tests better reflect actual behaviour
ref
https://linear.app/tryghost/issue/ENG-1543/debounce-the-members-lastseenatupdater
- The `LastSeenAtUpdater.updateLastSeenAt` function is called in
response to a `MemberClickEvent` — when a member clicks a link in an
email with tracking enabled. This function can be called many times for
the same member in a short period of time if e.g. a link checker is
clicking all the links in an email they received.
- This function should only update a member's `last_seen_at` timestamp
once per day. To accomplish this, `updateLastSeenAt` runs a
`select...for update` query to find the member's current `last_seen_at`
timestamp, and only updates the timestamp if the current `last_seen_at`
is before the start of the current day. The `for update` is required to
avoid a race condition, which previously caused this function to update
the `last_seen_at` timestamp more frequently than needed, which results
in many unnecessary database queries. However, we still run the initial
`select...for update` query for each event, which seems to be resulting
in contention for locks on the member's row in the `members` table.
- This commit introduces a simple in-memory cache so that we avoid
calling `updateLastSeenAt` if the member's `last_seen_at` timestamp has
already been updated in the current day, which should avoid running so
many `select...for update` queries and locking the `members` table up.
ref https://linear.app/tryghost/issue/ONC-323
- we're seeing a rare situation where the editor can get into a bad state when creating a new post that means saves silently fail
- added logging to help debug
- Added activity icon for Replies
- Updated Replies design
- Updated hard-coded Profile values to more realistic ones
- Renamed ActivityPub nav item and moved it to the top of the navbar
- Added a check for post attachments
[ANAL-79](https://linear.app/tryghost/issue/ANAL-79/stats-page-v10-design-refinements)
- typography is inconsistent in table headings
- chart section headings are not unified
- spacings are off between section and data headings
- inactive KPI tab colors are too light
- tech data numbers are not formatted
no issue
- These tests were all in a single `describe` block, making it difficult
to navigate and add more tests.
- This commit adds a nested `describe` block for each method on the
`lastSeenAtUpdater`, making it easier to understand the test cases, and
to add more. None of the tests themselves have changed, just the
organization of them.
Ref:
https://linear.app/tryghost/issue/ENG-1526/errors-from-members-api-routercontroller-are-being-lost
The try/catch/re-throw pattern, that hides errors, is used throughout
the RouterController.js file.
I have not changed the try/catch/re-throw pattern because it helps in
sending clean messages to the users. We may not want to return internal
errors as API responses.
I have added logs and Sentry messages that will help us debug without
losing error messages.
[ANAL-43](https://linear.app/tryghost/issue/ANAL-43/implement-all-possible-ui-for-10)
- The BarList component wasn't using the parameters provided in its
latest release
- Number formatting was missing on all numbers
- "See all" links were missing in Content/Sources/Locations
- Empty/default values was showing [blank]
- Flags were missing for country values
Got some translated strings, and hope it's awesome 🎊!
A short description of my changes:
- [x] All lines have been translated
- [x] For lines that already had a translation into Russian - I
double-checked the accuracy of that translation and corrected the
translation if necessary
- [x] I tried to ensure uniformity in the translation of terms and even
compiled a small glossary (local on my computer)
- [x] Frequently referred to and checked the hints from the
`context.json` file
I hope, you will appreciate your contribution!
---------
Co-authored-by: Michael Barrett <mike@ghost.org>
no issue
- Moved business logic from `WebhookController` to dedicated service
classes (`SubscriptionEventService`, `InvoiceEventService`,
`CheckoutSessionEventService`).
- Reduced controller complexity.
- Added unit tests for individual services, increasing overall test
coverage.
- Improved maintainability and scalability by isolating responsibilities
in specific services, making future updates easier and safer.
ref
https://linear.app/tryghost/issue/ONC-217/implement-the-deliverytime-option-in-mailgun-api-calls
Ghost experiences its highest peak load immediately after sending out a
newsletter, as it recieves an influx of traffic from users clicking on
the links in the email, a burst of email analytics events to process
from mailgun, and an increase in organic traffic to the site's frontend
as well as the admin analytics pages. The `BatchSendingService`
currently sends all the batches to Mailgun as quickly as possible, which
may contribute to higher peak loads.
This commit adds a `deliverytime` parameter to our API calls to Mailgun,
which allows us to specify a time in the future when we want the email
to be delivered. This will allow us to moderate the rate at which emails
are delivered, and in turn that should moderate the peak traffic volume
that Ghost receives in the first 2-3 minutes after sending an email.
The `deliverytime` is calculated based on a configurable parameter:
`bulkEmail.targetDeliveryWindow`, which specifies the maximum allowable
time (in milliseconds) after the email is first sent for Ghost to
instruct Mailgun to deliver the emails. Ghost will attempt to space out
all the batches as evenly as possible throughout the specified window.
For example, if the targetDeliveryWindow is set to `300000` (5 minutes)
and there are 100 batches, Ghost will set the `deliveryTime` for each
batch ~3 seconds apart.
no issue
- These retries were added a while ago when these tests were flaking in
CI, but this only serves to cover up any potential problems with these
tests or the code they are testing
- Ran these tests in CI three times in a row without them failing, so I
don't think the retries are necessary
no issue
- One of the tests in this suite added a member and didn't clean it up
when it was finished.
- Because of this, the tests after this one depended on this test
running first, so running an individual test in isolation might fail,
despite passing when run in the whole test suite
- This commit removes the added member, so all the tests in this suite
should pass whether run independently or all together
[ANAL-43](https://linear.app/tryghost/issue/ANAL-50/update-colors-of-barlist)
- Copy is too technical, doesn't follow conventions on Stats page
- Range filter dropdown has to be updated with more meaningful values
- KPI charts need a granularity dropdown to display meaninful charts
depending on the context
- Typography details should be updated
- "Posts/pages" dropdown needs to be added to Content section. This is a
Ghost specific filter that brings high value to customers
- "Campaigns" dropdown needs to be added to Sources section to support
ad tracking and filtering in the future
- BarList colors should be updated to be less purple all over the place
ref https://github.com/TryGhost/Ghost/pull/20835
- reimplemented email analytics changes that prioritized opened events
over other events in order to speed up open analytics
- added db persistence to fetch missing job to ensure we re-fetch every
window of events, especially important if we restart following a large
email batch
We learned a few things with the previous trial run of this. Namely,
that event throughput is not as high as we initially saw in the data for
particularly large databases. This set of changes is more conservative,
while a touch more complicated, in ensuring we capture edge cases for
really large newsletter sends (100k+ members).
In general, we want to make sure we're fetching new open events at least
every 5 mins, and often much faster than that, unless it's a quiet
period (suggesting we haven't had a newsletter send or much outstanding
event data).
Following commit #20858, the following updates were made:
- Inserted the Brazilian standard phone number format for `+1 (123)
456-7890`.
- Translated missing static strings, such as "Enter your email address"
and "Invalid email address."
- Updated some translations to maintain consistency and proper context.
ref https://linear.app/tryghost/issue/ONC-296
Our `stripe_prices.nickname` field had a length of 50 chars which meant we could error out trying to save a donation Stripe price with a generated product nickname containing a long site title.
- updated db schema and added a migration to change column length to 255
- added truncation to nickname generation to enforce a limit of 250 chars to match Stripe's limit
no issue
Give your audience a simple way to support your work with one-time payments, no membership required.
- cleaned up `tipsAndDonations` labs flag
closes https://linear.app/tryghost/issue/PLG-205
The server trims whitespace from the title when saving but our unsaved changes detection was comparing the raw title input field value meaning there would be a mismatch after publishing if the title field contained leading/trailing whitespace.
- updated title comparison to compare trimmed values
- moved and improved unsaved changes modal tests from publish-flow to unsaved-changes acceptance test file
- added util for pasting content into the editor to test for content changes
no issue
- fixed eslint config so tests are correctly linted
- removed `.only` on stats tests preventing other tests from running
- removed unneccessary `return` from async setup functions
[ANAL-1](https://linear.app/tryghost/issue/ANAL-32/add-stats-kpis-charts)
The technical details section in Stats contains only the browser breakdown ATM. This PR adds the rest (devices, operating systems) and fixes a couple of minor UI details on the rest of the charts
ref https://linear.app/tryghost/issue/DEV-20/faster-builds
- `build` should be the overall script to build the package
- `build:ts` should only build TypeScript
- by having them switched around, `build:ts` would call `yarn build`,
and `yarn` adds several hundreds of milliseconds of time to each build
closes https://linear.app/tryghost/issue/ANAL-23/filtering-by-logged-out-logged-in-traffic
- Updated all of our tinybird datasources and pipes to handle member status
- Added member_status as an array query param to the API endpoints
- Added a really dodgy power select multiple to the stats page to demonstrate it works (needs styling)
- Added all of the wiring so each chart updates
- This was done pretty fast, and may not be 100% right yet
closes https://linear.app/tryghost/issue/PLG-190
- often when adding portal links to your own site pages the URLs are added as absolute on the site's homepage due to copy+paste from displayed URLs in Admin
- when clicking absolute portal URLs the homepage is first loaded before the Portal popup is shown resulting in a slower and flashier experience
- added a transform for all local portal URLs on the page when Portal is initialized so links open the Portal popup immediately on the current page
ref https://linear.app/tryghost/issue/DEV-20/faster-builds
- this was previously causing duplicate builds of the TS projects
because Nx was building all projects with `build` targets, and we were
also calling `build:ts`
- this cuts 12 compilation jobs from the archive process, which should
help with container build times
closes
https://linear.app/tryghost/issue/ANAL-9/initial-tracker-in-ghost-head
- Given that all of the correct config is in place, output a tracking
script
- This allows us to send pageview events into tinybird
- All of the details (location of the script, destination etc) are kept
in config so that it's easy to change for different environments
ref DES-706
* After a user publishes or schedules a post, they are directed to the post list
* If a post is sent as an email, they are directed to the Analytics page
* In both cases, a confirmation modal is shown
* If a post is published, they can share it directly from the confirmation modal
* Added a "Share" button and some additional functions (view, edit, and delete post) to
published posts in post analytics
* Added a manual "Refresh" button to post analytics so that there is
no need to reload the whole app to update the data
---------
Co-authored-by: Sag <guptazy@gmail.com>
no ref
This service can get rather noisy when doing local development with our
data generator, as we do not use real urls, and therefore generate a lot
of not found errors in the console.
ref https://linear.app/tryghost/issue/ANAL-27/setup-tinybird-project-and-cicd
- Tinybird has a system for managing it's configuration as code, with full ci/cd support
- The tinybird CLI tool uses python, so we'll run that using docker, via `yarn tb`
- Some of the files tinybird adds should not be in source control, so we've added those to git ignore
- Everything in /ghost/tinybird is tinybird's init config
closes
https://linear.app/tryghost/issue/ANAL-10/stats-page-in-ghost-admin
- Adds all the structure, logic and permissions tests for the Stats page
to check we're loading the right thing at the right time
- Adds @tinybirdco/charts as a dependency, and has a single example of a
chart setup using the right config
closes https://linear.app/tryghost/issue/ANAL-8/flag-and-config
- This checks if tinybird:stats is set, and if so passes through the
config that is set via the config API
- This is used by Ghost admin to configure where to pull charts from
- we added NestJS to Ghost as a way forwards for a new framework within
Ghost but we haven't added much to it
- requiring all the NestJS code adds about 6-9% to our boot time, so if
we're not using it, it's just time we're burning for no benefit
- for now, I've gated this behind an env var to prevent it from loading
- we can't use labs flags in the boot process, so I've gone for an env
var
ref https://linear.app/tryghost/issue/ONC-274
This fix is done at the endpoint layer, which isn't ideal, but has the smallest
surface area for the change. I think we may want to move it up a layer though,
despite the extra complications. If we move the check _into_ the MembersAPI
class however, we start to run into issues with circular dependncies due to the
mess that is our dependency injection approach with the Members service
ref https://linear.app/tryghost/issue/ONC-274
This test checks that a welcome email is not sent when adding a member via the
Admin API if the site is in need of email verification, regardless of whether
or not the flag to send an email is set.
It is currently failing to demonstrate the whole in our logic.
All tests after this one were in a non-sending email state, which luckily
doesn't affect these tests, but it will affect future ones! This just cleans up
the settings so that they're back to a standard default.
ref DES-756, DES-757
- made the email property to make the refresh work properly for published-only posts
- included sentiment in the query to update the feedback percentage when refreshed
- Swap the e2e config API test to use our newer framework, and match against a
snapshot for the default case
- Move the individual test cases to unit tests (new file) - there are more to add
here, but this is parity with what we had before
- We use unit tests for checking through various cases for how config
changes modify the output as this is faster and more explicit
ref
639be25f1d
ref
f705dda314
- These items are never returned from the API as they've been removed
from the serializer
- The tests also check that they are not present
- They were removed by the referenced commits, which changed how these
things were built
ref PLG-160
- Refactored donation handling logic to be processed within the
`checkout.session.completed` webhook event.
- Added support for capturing and storing donation messages from Stripe
sessions.
- Integrated donation messages into the email notifications sent to
staff.
- Added database integration.
- Removed redundant donation logic from the invoice.payment_succeeded
webhook, since custom fields isn't supported.
- Updated and added new tests
---------
Co-authored-by: Sanne de Vries <sannedv@protonmail.com>
no issue
- when redirecting from Stripe back to Ghost after making a donation the URL contained a double slash (`//#/portal/...`) which triggered browser security errors when Portal modified the browser history stack when navigating
- the above could prevent the donation success modal from closing
ref DES-709
- when refresh button is clicked, the numbers in the analytics will be animated if changed
- for the animation to be performant, added a new dependency "animejs"
- to minimize the flash and layout shift, the analytics data is kept as it is while loading
- once finished loading, it will be replaced with the new data
ref https://linear.app/tryghost/issue/ENG-1518
After releasing the analytics job improvements, it appears for large
sites we're awfully close to missing some Mailgun events because of an
unexpected behavior of the aggregateStats call for just the opened
events job. This is taking 2-5x(+) the amount of time that the aggregate
queries take for the other jobs, despite not being dependent on the
events.
To err on the side of caution, we're going to roll this back and look to
optimize the aggregation queries before re-implementing. And we may be a
bit more cautious in giving _some_ but not _all_ priority to the
`opened` events.
ref https://linear.app/tryghost/issue/ENG-1518
After releasing the analytics job improvements, it appears for large
sites we're awfully close to missing some Mailgun events because of an
unexpected behavior of the aggregateStats call for just the opened
events job. This is taking 2-5x(+) the amount of time that the aggregate
queries take for the other jobs, despite not being dependent on the
events.
To err on the side of caution, we're going to roll this back and look to
optimize the aggregation queries before re-implementing. And we may be a
bit more cautious in giving _some_ but not _all_ priority to the
`opened` events.
Buttons were previously part of the bookmark card, but that's no longer
the case. This makes it easier to scale between different types of
posts/emails.
ref PLG-196
- Added comprehensive unit tests to cover various scenarios for the
createDonationCheckoutSession function.
- Verified correct handling of customer object, customerEmail and
metadata.
- Ensured accurate parameter passing to Stripe API, including
success_url and cancel_url.
ref DES-731
- improved mobile styles for the social buttons in the modal
- fixed the flow for publishing/scheduling pages
- redirect to post list only when a post doesn't involve any email
Comprehensive, context aware, Bosnian translations for all available
strings.
Informal addressing of users (as is widely accepted in Bosnia and
Herzegovina).
Minor text fixes.
ref
https://linear.app/tryghost/issue/ENG-1505/start-monitoring-event-loop-utilization-in-production-with
- The two main constraints we've observed in Ghost are the database connection pool and the CPU usage. However, there is a third constraint that we may be hitting, but can't currently observe: the event loop.
- This commit re-enabled OpenTelemetry (behind a config flag), removes the problematic tracing instrumentation which was breaking the frontend, and adds a Prometheus endpoint to export the eventLoopUtilization metric.
- This should give us visibility into whether we are hitting constraints in the event loop and address the root cause if we are.
ref KTLO-1
These tokens should prevent untargeted attacks, as the magic link
endpoint needs a token that was generated by the server, similar to a
CSRF token, but without needing any server-side state, or a cookie to
be set for unauthenticated users.
ref https://linear.app/tryghost/issue/ENG-952
- added persistence to the job timestamps
This set of changes reduces the potential for gaps in our email event
processing by adding persistence to the job timestamps. This avoids
expensive queries on the `email_recipients` table after every boot, and
reduces reliance on fallbacks in periods of heavy processing or reboot.
This is our first use of the jobs table to create a persistent line,
instead of its initial use case of single-run jobs. We may expand this
capability and move to use of the jobs model over knex.raw in order to
make this a bit friendlier.
Note: this works with sqlite but datetimes are stored as ints. It still
works fine. https://github.com/knex/knex/pull/5272
ref https://linear.app/tryghost/issue/ENG-1489/
The changes to improve posts loading in admin broke the shift selection
functionality. This restores that, as we need to be able to crawl across
the (now) three models when present.
The new modal for the updated publishing flow has an entirely new
layout, based on feedback previously received. In addition, this PR
includes a few tweaks to the underlying logic.
ref https://linear.app/tryghost/issue/ONC-263/
- labels dropdown in sign up card was not successfully fetching labels
- offers dropdown suffered the same fate
When introducing the second editor instance, it appears we ran into some
race conditions with the Ember tasks used to fetch the resources. The
init instance was beating the other to the punch, and so the state was
never successfully updated, as it is only fetched once on mounting the
card.
ref PLG-153
- Scoped one-time payments (`donation_event`) under the "payments"
category in the member activity feed filter.
- Updated `toggleEventType` logic to ensure that toggling "payments"
also toggles one-time payments when the `tipsAndDonations` feature is
enabled.
- Refactored event type handling into utility functions for easier
testing.
- Added unit tests for the new utility functions to ensure correct
behaviour.
- Added acceptance testing.
ref
https://linear.app/tryghost/issue/ONC-242/frontend-routing-prioritizes-collections-over-taxonomies
- Under a fairly specific edge case with a collection route that conflicts with a default, built-in route ("taxonomy" — like tags, authors, etc), the frontend routing would prioritize the collection over the taxonomy.
- For example, with the following in a custom `routes.yaml`:
```
collections:
/:
permalink: /{primary_tag}/{slug}/
template: index
```
If a post exists with the same slug as its primary tag's slug, the frontend routing would redirect the `/tag/{slug}/` route to the post in the collection, rather than serving the tag itself.
- This commit changes that, so if a collection's route conflicts with e.g. a `/tag/{slug}/` default route, Ghost will still return the built in route, rather than the collection.
no issue
- Safari's `innerText` sometimes includes a trailing `\n` depending on the markup used so we need to use `.to.have.rendered.trimmed.text()`
- reproducible when running tests via http://localhost:4200/tests (we don't yet run in Safari via our testem config)
closes https://linear.app/tryghost/issue/PLG-176
The editor files were previously stubbed for testing because we didn't have a way to load the externally-hosted files. This made testing slow and difficult because the only way to test the Admin integration was via Ghost's e2e browser tests.
- unstubbed the editor globals so `fetchKoenigLexical()` actually tries to import the external assets
- updated `ember-cli-build` to copy the Koenig UMD file over to the assets directory in development/test builds
- updated `environment.js` to set the required filename for the default asset import to successfully hit the test environment hosted files
- updated lexical editor acceptance tests to demonstrate the editor loads successfully for new and existing posts
ref https://linear.app/tryghost/issue/ONC-261
- Previous method of hiding the second Lexical instance using `width:
0`, `height: 0`, and `overflow: hidden` caused CPU usage to spike,
likely due to CodeMirror continuously processing the element.
ref https://linear.app/tryghost/issue/ONC-261
- Previous method of hiding the second Lexical instance using `width:
0`, `height: 0`, and `overflow: hidden` caused CPU usage to spike,
likely due to CodeMirror continuously processing the element.
ref https://linear.app/tryghost/issue/ENG-1477
- updated email analytics job to prioritize open events
- put limits on non-open event fetching
- updated job to now restart itself until processing is at a
sufficiently low volume
Previously the EmailAnalytics job would process all event data equally.
When there's sufficient recipients (>20k), we could see delays in the
open rate data in Admin because of all the delivered events being
processed. Open events are far more important to users, so we've now
prioritized processing those events before any others.
Processing of events shouldn't be any faster or slower with this as this
doesn't change throughput, just order.
NOTE: Use the mailgun-mock-server in TryGhost/Toolbox for testing.
ref DES-697
- used client side navigation on the flow instead of hard refresh
- reduced layout shift on the analytics screen
- made the modal transition a bit smoother
ref https://github.com/TryGhost/Ghost/pull/20793
- now that there's a non-zero suggested value amount Stripe's UI shows a button to change the amount rather than showing the amount input field immediately
- added extra click to the tests and an expectation that the default value is set correctly
closes https://linear.app/tryghost/issue/PLG-156
- updated all default fixtures to use `500` ($5) as the default suggested donation value
- added migration to update existing settings using the old default of `0` to `500`
- this is fine to apply because the feature hasn't been released so there's no explicit `0` values in the wild
- added an acceptance test for the adminx-settings tips & donations section
ref DES-609
- On top of bookmark card, this update adds default background color and font to file, audio, and product cards
- The main purpose of this is keeping embedded look across these cards consistencly regardless of the theme background color and default font
- Themes can still override the styles
ref https://linear.app/tryghost/issue/PLG-174
- forcing autosave on excerpt blur caused posts to revert to `draft` and save immediately even when they were published/scheduled
- updated the save-on-excerpt-blur to only autosave drafts
- added acceptance tests for title and excerpt change+blur on published posts
ref [ENG-661](https://linear.app/tryghost/issue/ENG-661/)
ref [ONC-253](https://linear.app/tryghost/issue/ONC-253/)
ref [PLG-174](https://linear.app/tryghost/issue/PLG-174/)
- restored the original but reverted fix for unsaved changes modal from https://github.com/TryGhost/Ghost/pull/20687
- updated code to remove some incorrect early-falsy-return logic in `editorController.hasDirtyAttributes` that prevented save of unsaved changes on the underlying model (e.g. excerpt)
- updated unit tests so they are testing real post model instances and therefore are testing what we expect them to test
- added acceptance tests to ensure autosave is working for title and excerpt fields
---------
Co-authored-by: Ronald Langeveld <hi@ronaldlangeveld.com>
ref https://linear.app/tryghost/issue/PLG-176
- we recently had to revert code because it caused unexpected problems despite the unit tests passing
- the setup for the editor unit tests was very basic and did not represent the real world because it tested a simple EmberObject instance rather than an Ember Model instance meaning the tests weren't covering the real code paths
- updated the unit test with a more realistic environment ready for re-working the reverted code+tests
ref https://linear.app/tryghost/issue/PLG-174
- added `blur` handler to excerpt field so it acts the same as the title field and triggers a save when it loses focus
