`wezterm.font` and `wezterm.font_with_fallback` can now specify
harfbuzz_features and freetype load/render target and flags as
options on a per-font basis.
This allows you to do things such as adjust shaping (eg: ligatures) or
rendering (eg: disable bitmaps, or adjust hinting) for a single font in
a fallback rather than globally for all fonts.
The error message in this issue sounds a lot like the freetype
regression that caused bitmap fonts to fail to render in a recent
freetype release.
Our workaround for that is to used our understanding of whether
a font is a bitmap font or not to avoid calling render.
What we normally see for bitmap TTFs is that setting the scale
fails and we then fall back to using a bitmap.
For Monaco.dfont it appears as though setting the scale succeeds.
This commit introduces some skepticism and prefers to use bitmaps
when available.
This might potentially cause problems in the future if there are
fonts that legitimately have both scaled and bitmap fonts, but
lets see if this helps for now.
refs: https://github.com/wez/wezterm/issues/1419
This is a fairly far-reaching commit. The idea is:
* Introduce a unicode_version config that specifies the default level
of unicode conformance for each newly created Terminal (each Pane)
* The unicode_version is passed down to the `grapheme_column_width`
function which interprets the width based on the version
* `Cell` records the width so that later calculations don't need to
know the unicode version
In a subsequent diff, I will introduce an escape sequence that allows
setting/pushing/popping the unicode version so that it can be overridden
via eg: a shell alias prior to launching an application that uses a
different version of unicode from the default.
This approach allows output from multiple applications with differing
understanding of unicode to coexist on the same screen a little more
sanely.
Note that the default `unicode_version` is set to 9, which means that
emoji presentation selectors are now by-default ignored. This was
selected to better match the level of support in widely deployed
applications.
I expect to raise that default version in the future.
Also worth noting: there are a number of callers of
`unicode_column_width` in things like overlays and lua helper functions
that pass `None` for the unicode version: these will assume the latest
known-to-wezterm/termwiz version of unicode to be desired. If those
overlays do things with emoji presentation selectors, then there may be
some alignment artifacts. That can be tackled in a follow up commit.
refs: #1231
refs: #997
We rely on using freetype in order to support more fonts in more
situations, and we have a deeper existing integration with harfbuzz.
I'm unlikely to come back to allsorts to complete our integration,
and in the meantime, it just adds overhead to build/test and those
builds are taking longer and longer.
I loved the idea of using pure rust for all the font stuff, but
its time is not now.
closes: #587closes: #66
It appears as though Menlo is the only font on macos to contain the
heavy ballot cross symbol, which is commonly used on macos (eg: in
`brew` output).
Our fallback list, despite starting with Menlo, didn't include menlo
itself in the candidates.
Furthermore, `ls-fonts` wouldn never see the result of the system
fallback resolution because it didn't know to try again, and was
using the list of handles from before the fallback.
This commit resolves all of these concerns.
refs: #849
A user reported a problem matching `等距更纱黑体 SC` against
a font that wezterm thought was really named `Sarasa Mono SC`.
This commit attempts to match against other language names,
although the Sarasa font that I found on the MS store doesn't
return `等距更纱黑体` in any of the additional SFNT name tables,
so this isn't a successful change.
The previous commit was partially OK, but the main cause of
emoji being wonky was this bit of macos specific code that I
added ages ago.
Remove that hack and the portion of the code from the previous
commit that was working to undo it.
This should make the baselines consistent across all platforms.
refs: #1203
We now compute the cap-height from the rasterized glyph data.
Moved the scaling action of use_cap_height_to_scale_fallback_fonts from
glyphcache into the font resolver: when enabled, and we have data
about the baseline font and the font being resolved, then the resolving
font will be scaled such that the cap-height of both fonts has the same
pixel size.
The effect of this is that `I` glyphs from both fonts should appear to
have the same height.
Added a row of `I`'s in differing styles at the bottom of styles.txt
to make this easier to visualize.
refs: #1189
This is to handle situations such as some versions of the Terminus
bitmap font, where the individual bitmap strike sizes are broken
out across multiple individual files.
Font matching now passes down the nominal pixel height based on
the current DPI and font scale factor, and will use that to select
the font file that has the closest pixel size.
Previously, it would be potentially undefined which of the Terminus
font files would be selected.
refs: https://github.com/wez/wezterm/issues/1189
This font is a bit funky; the bitmap strikes are only 4px wide:
```
; ftdump gohufont.otb
There is 1 face in this file.
----- Face number: 0 -----
font name entries
family: Gohu GohuFont
style: Regular
postscript: UNAVAILABLE
created: 1904-01-01
modified: 1904-01-01
revision: 1.00
glyph count: 1694
font type entries
FreeType driver: truetype
sfnt wrapped: yes
type: fixed size
direction: horizontal
fixed width: no
glyph names: no
fixed size
0: height 11, width 3
size 11.000, x_ppem 11.000, y_ppem 11.000
1: height 14, width 4
size 14.000, x_ppem 14.000, y_ppem 14.000
charmaps (1)
0: format 4, platform 3, encoding 1 language 0 (active)
```
but using that the cell size isn't right.
We draw from the metrics we compute via cell_metrics to get more
information; we don't get a valid height from that (just 0!), but
we do get the much more plausible width of 8 pixels.
So we take the max of the two techniques for figuring the metrics.
That appears to work out, and also doesn't appear to break emoji
fonts.
refs: #1165
This was a bit of a PITA to run down; the essence of the problem
was that the shaper was returning an x_advance of 0 for U+3000,
which caused wezterm's shaping layer to elide that glyph.
I eventually tracked down the x_advance to be the result of
scaling by an x_scale of 0, which in turn is the result of
harfbuzz not knowing the font size.
The critical portion of this diff is the line that advises
harfbuzz that the font has changed after we've applied the
font size.
The rest is just stuff to make it easier to debug and verify.
This:
```
printf "x\u3000x."
```
Now correctly renders on screen as "x x".
fixes: #1161
- If possible use fontconfig to obtain character coverage instead of
going through all glyphs using freetype.
- `FT_Get_First_Char` typically returns ranges of continuous glyphs itself
and it is far cheaper (I measured a speedup of about 7 times while
catting a large file with lots of funny unicode) to add a range to the
glyph coverage instead of adding each glyph individually.
- Permit adding a range to a RangeSet without performing checks to speed
up things even further.
@H-M-H noticed and suggested this; rather than spawning a thread
for potentially every cluster of graphemes that are being displayed
before we've located a font, constrain things to a single thread
so that we don't burn CPU trying to process the same results
in an excessive number of threads.
Since fonts now use dual source blending, the pixel colors are
interpreted as individual alpha channels. The A component should
be set to 1.0, so that's what we do here.
refs: #1025
The introduction of the Emoji vs Text VS processing means that we might
in some cases not find a glyph with the requested presentation.
In that case, we'd rather show the emoji presentation glyph than none at
all, so we'll retry fallback processing with unspecified presentation.
refs: #997
The recent presentation logic needs to be tweaked to ensure that
we ignore presentation when we reach the fallback font, otherwise
we'll end up in a bad error stack and crash the program.
This commit annotates fonts with a boolean that indicates whether
we think it contains glyphs with emoji presentation, and then
passes the cluster.presentation field down to the shaper.
If the presentation doesn't match the current font in the fallback,
then it will be skipped until we exhaust its options.
`wezterm ls-fonts` also shows whether we think a font has emoji
presentation.
refs: #997
This commit hooks up DECRQM so that we can report that we implement
synchronized updates, and then refines the code that manages sending
data to the terminal model; the first cut at synchronized updates
was a bit simplistic, and now we make a point of "flushing" pending
actions when we start a sync point, and again as soon as we release
the sync point.
This smooths out the jaggies around the orca that I mentioned in
dcbbda7702
and while testing this, I realized that recent parser changes had
mangled processing bundled dec private mode sequences where multiple
modes were specified in the same overall escape sequence. I've
added the missing unit test case for this and made that work again.
refs: https://github.com/wez/wezterm/issues/955
refs: https://github.com/wez/wezterm/issues/882
opentype allows a font to have a weight in the range 0-1000.
MacOS has its own concept of symbolic weight names and opentype
values that is a slightly different scale of boldness to Windows
and Linux.
That means that Medium could be a different range of opentype
weight values depending on the system.
To further complicate things, the font designer can name their
variant with any name they like and assign it an arbitrary
opentype weight value.
For the Operator Mono font, it has Book variant with opentype
weight 325 and a Light variant with an opentype weight of 300.
wezterm was considering these both to have `FontWeight::Light` because
that's how those values were bucketed, which results in amibiguity in
resolve the font and frustration in not being able to access one of the
variants.
This commit changes the `FontWeight` type to now hold the unambiguous
opentype weight value, and to define some symbolic aliases for
some specified weights.
When serializing, if the weight matches a symbolic alias, then that
name will be used in the canonical name (eg: as listed via ls-fonts).
Otherwise, the numeric value will be used.
When parsing the font configuration, wezterm will allow both symbolic
and numeric values.
This allows all of the Operator Mono variants to be referenced
unambiguously, although some variants have to be specified via the
numeric weight:
```
wezterm.font("Operator Mono", {weight=275, stretch="Normal", italic=false}) -- /Users/wez/.fonts/OperatorMono-XLight.otf, FontDirs
wezterm.font("Operator Mono", {weight="Light", stretch="Normal", italic=false}) -- /Users/wez/.fonts/OperatorMono-Light.otf, FontDirs
wezterm.font("Operator Mono", {weight=325, stretch="Normal", italic=false}) -- /Users/wez/.fonts/OperatorMono-Book.otf, FontDirs
wezterm.font("Operator Mono", {weight="DemiLight", stretch="Normal", italic=false}) -- /Users/wez/.fonts/OperatorMono-Medium.otf, FontDirs
wezterm.font("Operator Mono", {weight="Regular", stretch="Normal", italic=false}) -- /Users/wez/.fonts/OperatorMono-Bold.otf, FontDirs
```
https://github.com/wez/wezterm/issues/849#issuecomment-873454483
Change the loader so that it has better matching weight and stretch
characteristics, and ask core text to return all possible candidates
so that we can then apply our CSS-style font matching rules.
Previously, the font descriptor we created would only match the
family name and return the normal/regular variant only.
refs: https://github.com/wez/wezterm/issues/873
This commit adds a slant to *scalable* (not bitmap!) fonts whose
originating font attributes requested italics but for for which
the resolved face is not italic.
refs: #815
This commit introduces the knowledge about whether a font is
scalable or was using bitmap strikes (eg: color emoji bitmaps).
Then that information is used to help figure out whether and
how to scale a glyph.
refs: https://github.com/wez/wezterm/issues/685
I *think* the heart of the issue is that the problematic fonts
don't define a `spacing` property, and we were being stric
about matching it.
This commit changes the behavior to strictly match the spacing
value when it is defined, but to allow an undefined spacing
value to match.
refs: https://github.com/wez/wezterm/issues/726
It now outputs something that you could conceivably put into
your config file, although the intent is to show the canonical
way to reference the individual fonts that were found, rather
than to specify a fully baked list to paste into a config.
eg:
```
; ./target/debug/wezterm ls-fonts
Primary font:
wezterm.font_with_fallback({
-- /home/wez/.fonts/OperatorMonoSSmLig-Medium.otf, FontDirs
{family="Operator Mono SSm Lig", weight="DemiLight"},
-- /home/wez/.fonts/OperatorMonoSSmLig-Medium.otf, FontConfig
{family="Operator Mono SSm Lig", weight="DemiLight"},
-- /home/wez/.fonts/MaterialDesignIconsDesktop.ttf, FontDirs
"Material Design Icons Desktop",
-- /home/wez/.fonts/terminus-bold.otb, FontDirs
{family="Terminus", weight="Bold"},
-- /home/wez/.fonts/JetBrainsMono-Regular.ttf, FontDirs
"JetBrains Mono",
-- /home/wez/.fonts/NotoColorEmoji.ttf, FontDirs
"Noto Color Emoji",
-- /home/wez/.fonts/MaterialDesignIconsDesktop.ttf, FontConfig
"Material Design Icons Desktop",
-- /usr/share/fonts/terminus-fonts/ter-u32n.otb, FontConfig
"Terminus",
-- /home/wez/.fonts/JetBrainsMono-Regular.ttf, FontConfig
"JetBrains Mono",
-- /home/wez/.fonts/NotoColorEmoji.ttf, FontConfig
"Noto Color Emoji",
-- <built-in>, BuiltIn
"Last Resort High-Efficiency",
})
```
In earlier times, in an effort to avoid bleeding resources into
child processes, the fontconfig wrapper grew some logic to keep
track of how many fontconfig objects we'd loaded so that we could
aggressively unload it when there were none.
Since that time we've evolved differently pessimistic logic that forces
random fds closed when we spawn children so the critical need for unloading
fontconfig is no longer present.
Importantly, with the over-eager unloading, each font query we make
effectively needs to initialize fontconfig from scratch, which is a
fixed minimum cost of ~5-6ms on my system, and I've seen some traces
with a number as high as 100ms (those systems need `fc-cache` to be
run).
Removing the unloading keeps fontconfig initialized so we only pay
the 5ms cost once, then subsequent queries are in the order of 100us.
It seems difficult/impossible to phrase precisely the constraints
that we want when making a font config query, so this changes our
queries to use relative broad family and postscriptname list operations
and then we parse and filter using our own CSS-inspired font matching
criteria.
refs: #689
we now compute the ratio of the cap height (the height of a capital
letter) vs. the em-square (which relates to our chosen point size) to
understand what proportion of the font point-size that a given font
occupies when rendered.
When rendering glyphs from secondary fonts we can use the cap height
ratios of both to scale the secondary font such that its effective
cap height matches that of the primary font.
In plainer-english: if you mix say bold, italic and regular text
style in the same line, and you have different font families for
those fonts, then they will now appear to be the same height where
previously they may have varied more noticeably.
For emoji and symbol fonts there may not be a cap-height metric
encoded in the font. We can however, improve our scaling: prior
to this commit we'd use the ratio of the cell metrics of the two
fonts to scale the icon/emoji glyph, but this could cause the glyph
to be slightly oversized as seen in https://github.com/wez/wezterm/issues/624
If we know the cap-height of the primary font then we can additionaly
apply that factor to scale the emoji to better fit the cell.
While looking at this, I noticed that the aspect ratio calculation
for when to apply to the allow_square_glyphs_to_overflow_width option
had width and height flipped :-(
See also: https://tonsky.me/blog/font-size/
refs: https://github.com/wez/wezterm/issues/624
Sort the available fonts list by family name (rather than full name),
then the styling attributes.
Display the font name in the preferred form for inclusion in the
wezterm config.
* Log which codepoints we're about to perform fallback resolution for
* Rank the fallback fonts by decreasing amount of coverage
* If a fallback covers the desired codepoints, remove those codepoints
from the set and reduce, so that we only add the minimal set of
fallback fonts for the set of codepoints
You can see what triggered fallback processing using:
```
; WEZTERM_LOG=wezterm_font=trace wezterm -n 2> /tmp/font.txt
; grep 'fallback font' /tmp/font.txt
2021-04-11T21:41:09.653Z TRACE wezterm_font > Looking for \u{d604}\u{c7ac}\u{be0c}\u{b79c}\u{ce58} in fallback fonts
2021-04-11T21:41:12.132Z TRACE wezterm_font > Looking for \u{f4e9} in fallback fonts
```
refs: https://github.com/wez/wezterm/issues/559#issuecomment-817260512
Now that all platforms know whether the system fallbacks
covered the requested glyph range, it is reasonable to
restore the configuration error window to advise the user
if they are missing fonts for the text they want to display.
refs: https://github.com/wez/wezterm/issues/671
Previously, we would add a list of ~100 or so fallback fonts to
the shaper's fallback list.
In pathological cases where a wide range of glyphs that have no
corresponding font are repeatedly emitted to the output, we'd
keep loading and unloading that large list of fallbacks in the
hope of finding a match.
Since that code was written, we're now able to compute the
codepoint coverage for ourselves, so teach the core text locator
how to reduce the the list of fallback fonts to just those that
contain the missing glyphs.
Furthermore, we restrict that list to just the normal/regular
weight/stretch/style fonts.
refs: https://github.com/wez/wezterm/issues/671
The intent is to reveal more context on what's happening in
https://github.com/wez/wezterm/issues/671
As a nice side benefit, this avoids the potential inability
to open paths that are not utf8 or representable as c-strings
on Windows.
And on top of that: this enables memory mapped file IO as well,
which wasn't enabled previously. This should help to reduce
extraneous copies of the font in memory, have fewer open files
and minimize the chances of racing with O_CLOEXEC.
When we process the system fallback list, we can produce a long list
of fonts to be speculatively processed by the shaper.
Until this commit, the shaper would always keep the associated
freetype face open forever, which increases the number of open
files and the amount of allocated memory.
This commit allows the shaper to release a font if it has never
produced any valid shaper results, which keeps the list down
to just the fonts that are in use.
This commit allows the x11 window implementation to detect changes
in the DPI that occur after a window is created.
These can occur when changing desktop resolution or when changing
the accessibility option for "Large Text" in gnome.
In order to avoid continually polling for the value on every resize,
we look for the `_GTK_EDGE_CONSTRAINTS` atom in our property change
notifications. This seems to be sent at least as often as the
dpi/scaling changes.
It's also worth noting that some dpi changes don't generate resize
events, so we can't just read the dpi value on every resize, because
we'd miss some of those changes.
Part of this commit changes the font scaling logic: previously
we'd keep a notion of "dpi scale" to apply. That dates from an
earlier time in wezterm where we didn't think that we knew an
actual dpi value.
The way that worked was that we'd compare our current guestimate
of the DPI against what we though the baseline OS dpi should be to
produce a scaling factor.
On X11 that dpi value is global and we'd effectively always produce
a revised scaling factor of 1 after we'd set up the initial window.
This commit changes that logic to just pass down the actual DPI value
to the font code. That DPI value already accounts for HiDPI scaling
so this is hopefully a NOP change for the other systems.
refs: https://github.com/wez/wezterm/issues/667
Replaces the last usage of ttf-parser with calling into freetype.
This removes a source of inconsistency, as ttf-parser doesn't support
all of the things that freetype does.
Notably, this prevents a weird error from blowing up codepoint coverage
calculations on a system where I have helvetica.bdf in my font dir for
long-forgotten reasons.
```
WEZTERM_LOG=wezterm_font=trace,info wezterm
```
will list fonts found in font_dirs and the builtin fonts.
It can't show all fonts found via the system locator, because
that only has an interface for finding fonts by matching name,
not listing all of them.
With this configuration:
```lua
local wezterm = require 'wezterm'
return {
font_dirs = {"/Users/wez/Downloads/Inconsolata"},
font = wezterm.font("Inconsolata"),
font_locator = "ConfigDirsOnly"
}
```
wezterm is now able to see the 74 variations that are available
in the single inconsolata ttf.
Running `WEZTERM_LOG=wezterm_font=trace wezterm` will log the
variations.
refs: https://github.com/wez/wezterm/issues/655
If shaping can't resolve some glyphs, queue the font locator
fallback resolution to another thread; meanwhile, a last resort
glyph is used.
That thread can trigger an invalidation once the fallback resolve
is complete, the window is invalidated and the last resort glyph
is replaced by the resolve glyph.
refs: https://github.com/wez/wezterm/issues/559
refs: https://github.com/wez/wezterm/issues/508
Keep track of the glyphs we've already advised about (until the
config is reloaded) so that we don't keep spamming the user.
refs: https://github.com/wez/wezterm/issues/559
* Check built-in fonts before asking the system for codepoint coverage
* If one of the earlier stages resolved some fonts, skip the remaining
stages and speculatively shape what we have. This avoids triggering
the system font lookup for fonts that are present in the font_dirs
or that are built-in, such as powerline symbols.
refs: https://github.com/wez/wezterm/issues/559
The dwrote crate offers functions that can extract the underlying
font file name(s) from the system, so let's use those to get
OnDisk font handles and save some memory.
refs: https://github.com/wez/wezterm/issues/559
In an earlier incarnation of both wezterm and freetype, FT_New_Face
would lead file descriptors into child processes because it didn't
set O_CLOEXEC. That led to the slightly pessimistic approach of
loading the font into memory for the lifetime of the wezterm process.
With the improved fallback handling on macos, this can result in
hundreds of MB of font data being loaded, in some cases multiple
times.
Since those days, freetype now sets O_CLOEXEC and wezterm has some
logic to close other random fds, so the descriptor leaking problem
is gone and we can now let freetype manage a file handle instead
of a memory-baked font.
This reduces the memory utilization by at least 1GB in the case
that a glyphs need to be resolved from the system fallback fonts.
refs: https://github.com/wez/wezterm/issues/559
terminus-bold.otb reports 0 height!
Detect and force that case to go through the bitmap strike loading
code path.
Improve the size selection heuristic for bitmap strikes: previously,
we would just pick the largest bitmap and allow it to be scaled down,
which was OK for emoji fonts that just had 128px square glyphs, but
is not ok for pre-rendered pixel strikes like terminus.
Note that IncreaseFontSize works in terms of percentages only,
so using a font like this may have "gaps" when ctrl-+ or - to change
the font size.
refs: https://github.com/wez/wezterm/issues/560
This is a bit unfortunate, but necessary, because the system fallback
list contains a handful of special fonts that apple doesn't ship on
disk in ttf/otf files.
One of those is `.AppleSymbolsFB` which would normally satisfy
the symbol lookup.
This commit hard codes the "Apple Symbols" font to use instead,
which is a disk based font. I don't know what the difference
is between it and `.AppleSymbolsFB`, but this is sufficient
to satisfy the glyph in question from the referenced issue.
refs: https://github.com/wez/wezterm/issues/506
Continuing along the same lines as the prior commit, the goal
of this commit is to remove the buffer transformation that was
part of uploading the texture to the GPU provided surface.
In order to do so:
* The sense of our local textures needs to change from bgra32 to rgba32.
bgra32 was a hangover from earlier versions of our window crate that
allowed direct-to-fb writes in software mode. We had to pick bgra32
for that for the broadest OS compatibility. I believe that that
constraint has been totally removed, although there is a chance that
this will flip the colors on macos.
* There was an additional linear-to-srgb conversion inlined in that
buffer transformation. I have no idea where that is needed because
the source data is carefully constructed as SRGB. I don't yet know
how to signal that, but for now I've moved that gamma correction
into the shader when we sample the texture.
With this change, timg playback now has vtparse as the hottest
region of code.
refs: #537
This commit expands on the prior commits to introduce the concept
of per-window configuration overrides.
Each TermWindow maintains json compatible object value holding
a map of config key -> config value overrides.
When the window notices that the config has changed, the config
file is loaded, the CLI overrides (if any) are applied, and then
finally the per-window overrides, before attempting to coerce
the resultant lua value into a Config object.
This mechanism has some important constraints:
* Only data can be assigned to the overrides. Closures or special
lua userdata object handles are not permitted. This is because
the lifetime of those objects is tied to the lua context in which
they were parsed, which doesn't really exist in the context of
the window.
* Only simple keys are supported for the per-window overrides.
That means that trying to override a very specific field of
a deeply structured value (eg: something like `font_rules[1].italic = false`
isn't able to be expressed in this scheme. Instead, you would
need to assign the entire `font_rules` key. I don't anticipate
this being a common desire at this time; if more advance manipulations
are required, then I have some thoughts on an event where arbitrary
lua modifications can be applied.
The implementation details are fairly straight-forward, but in testing
the two examplary use cases I noticed that some hangovers from
supporting overrides for a couple of font related options meant that the
window-specific config wasn't being honored. I've removed the code that
handled those overrides in favor of the newer more general CLI option
override support, and threaded the config through to the font code.
closes: #469closes: #329
