rework positioning/rendering and enable softwrap/virtual text (#5420)

* rework positioning/rendering, enables softwrap/virtual text

This commit is a large rework of the core text positioning and
rendering code in helix to remove the assumption that on-screen
columns/lines correspond to text columns/lines.

A generic `DocFormatter` is introduced that positions graphemes on
and is used both for rendering and for movements/scrolling.
Both virtual text support (inline, grapheme overlay and multi-line)
and a capable softwrap implementation is included.

fix picker highlight

cleanup doc formatter, use word bondaries for wrapping

make visual vertical movement a seperate commnad

estimate line gutter width to improve performance

cache cursor position

cleanup and optimize doc formatter

cleanup documentation

fix typos

Co-authored-by: Daniel Hines <d4hines@gmail.com>

update documentation

fix panic in last_visual_line funciton

improve soft-wrap documentation

add extend_visual_line_up/down commands

fix non-visual vertical movement

streamline virtual text highlighting, add softwrap indicator

fix cursor position if softwrap is disabled

improve documentation of text_annotations module

avoid crashes if view anchor is out of bounds

fix: consider horizontal offset when traslation char_idx -> vpos

improve default configuration

fix: mixed up horizontal and vertical offset

reset view position after config reload

apply suggestions from review

disabled softwrap for very small screens to avoid endless spin

fix wrap_indicator setting

fix bar cursor disappearring on the EOF character

add keybinding for linewise vertical movement

fix: inconsistent gutter highlights

improve virtual text API

make scope idx lookup more ergonomic

allow overlapping overlays

correctly track char_pos for virtual text

adjust configuration

deprecate old position fucntions

fix infinite loop in highlight lookup

fix gutter style

fix formatting

document max-line-width interaction with softwrap

change wrap-indicator example to use empty string

fix: rare panic when view is in invalid state (bis)

* Apply suggestions from code review

Co-authored-by: Michael Davis <mcarsondavis@gmail.com>

* improve documentation for positoning functions

* simplify tests

* fix documentation of Grapheme::width

* Apply suggestions from code review

Co-authored-by: Michael Davis <mcarsondavis@gmail.com>

* add explicit drop invocation

* Add explicit MoveFn type alias

* add docuntation to Editor::cursor_cache

* fix a few typos

* explain use of allow(deprecated)

* make gj and gk extend in select mode

* remove unneded debug and TODO

* mark tab_width_at #[inline]

* add fast-path to move_vertically_visual in case softwrap is disabled

* rename first_line to first_visual_line

* simplify duplicate if/else

---------

Co-authored-by: Michael Davis <mcarsondavis@gmail.com>

book/src/configuration.md

@@ -307,4 +307,25 @@ Currently unused
 
 #### `[editor.gutters.spacer]` Section
 
-Currently unused
+Currently unused
+
+### `[editor.soft-wrap]` Section
+
+Options for soft wrapping lines that exceed the view width
+
+| Key                 | Description                                                  | Default |
+| ---                 | ---                                                          | ---     |
+| `enable`            | Whether soft wrapping is enabled.                            | `false` |
+| `max-wrap`          | Maximum free space left at the end of the line.              | `20`    |
+| `max-indent-retain` | Maximum indentation to carry over when soft wrapping a line. | `40`    |
+| `wrap-indicator`    | Text inserted before soft wrapped lines, highlighted with `ui.virtual.wrap` | `↪ `    |
+
+Example:
+
+```toml
+[editor.soft-wrap]
+enable = true
+max-wrap = 25 # increase value to reduce forced mid-word wrapping
+max-indent-retain = 0
+wrap-indicator = ""  # set wrap-indicator to "" to hide it
+```

book/src/languages.md

@@ -61,7 +61,7 @@ These configuration keys are available:
 | `config`              | Language Server configuration                                 |
 | `grammar`             | The tree-sitter grammar to use (defaults to the value of `name`) |
 | `formatter`           | The formatter for the language, it will take precedence over the lsp when defined. The formatter must be able to take the original file as input from stdin and write the formatted file to stdout |
-| `max-line-length`     | Maximum line length. Used for the `:reflow` command           |
+| `max-line-length`     | Maximum line length. Used for the `:reflow` command and soft-wrapping           |
 
 ### File-type detection and the `file-types` key
 

helix-core/src/doc_formatter.rs

@@ -0,0 +1,384 @@
+//! The `DocumentFormatter` forms the bridge between the raw document text
+//! and onscreen positioning. It yields the text graphemes as an iterator
+//! and traverses (part) of the document text. During that traversal it
+//! handles grapheme detection, softwrapping and annotations.
+//! It yields `FormattedGrapheme`s and their corresponding visual coordinates.
+//!
+//! As both virtual text and softwrapping can insert additional lines into the document
+//! it is generally not possible to find the start of the previous visual line.
+//! Instead the `DocumentFormatter` starts at the last "checkpoint" (usually a linebreak)
+//! called a "block" and the caller must advance it as needed.
+
+use std::borrow::Cow;
+use std::fmt::Debug;
+use std::mem::{replace, take};
+
+#[cfg(test)]
+mod test;
+
+use unicode_segmentation::{Graphemes, UnicodeSegmentation};
+
+use crate::graphemes::{Grapheme, GraphemeStr};
+use crate::syntax::Highlight;
+use crate::text_annotations::TextAnnotations;
+use crate::{Position, RopeGraphemes, RopeSlice};
+
+/// TODO make Highlight a u32 to reduce the size of this enum to a single word.
+#[derive(Debug, Clone, Copy)]
+pub enum GraphemeSource {
+    Document {
+        codepoints: u32,
+    },
+    /// Inline virtual text can not be highlighted with a `Highlight` iterator
+    /// because it's not part of the document. Instead the `Highlight`
+    /// is emitted right by the document formatter
+    VirtualText {
+        highlight: Option<Highlight>,
+    },
+}
+
+#[derive(Debug, Clone)]
+pub struct FormattedGrapheme<'a> {
+    pub grapheme: Grapheme<'a>,
+    pub source: GraphemeSource,
+}
+
+impl<'a> FormattedGrapheme<'a> {
+    pub fn new(
+        g: GraphemeStr<'a>,
+        visual_x: usize,
+        tab_width: u16,
+        source: GraphemeSource,
+    ) -> FormattedGrapheme<'a> {
+        FormattedGrapheme {
+            grapheme: Grapheme::new(g, visual_x, tab_width),
+            source,
+        }
+    }
+    /// Returns whether this grapheme is virtual inline text
+    pub fn is_virtual(&self) -> bool {
+        matches!(self.source, GraphemeSource::VirtualText { .. })
+    }
+
+    pub fn placeholder() -> Self {
+        FormattedGrapheme {
+            grapheme: Grapheme::Other { g: " ".into() },
+            source: GraphemeSource::Document { codepoints: 0 },
+        }
+    }
+
+    pub fn doc_chars(&self) -> usize {
+        match self.source {
+            GraphemeSource::Document { codepoints } => codepoints as usize,
+            GraphemeSource::VirtualText { .. } => 0,
+        }
+    }
+
+    pub fn is_whitespace(&self) -> bool {
+        self.grapheme.is_whitespace()
+    }
+
+    pub fn width(&self) -> usize {
+        self.grapheme.width()
+    }
+
+    pub fn is_word_boundary(&self) -> bool {
+        self.grapheme.is_word_boundary()
+    }
+}
+
+#[derive(Debug, Clone)]
+pub struct TextFormat {
+    pub soft_wrap: bool,
+    pub tab_width: u16,
+    pub max_wrap: u16,
+    pub max_indent_retain: u16,
+    pub wrap_indicator: Box<str>,
+    pub wrap_indicator_highlight: Option<Highlight>,
+    pub viewport_width: u16,
+}
+
+// test implementation is basically only used for testing or when softwrap is always disabled
+impl Default for TextFormat {
+    fn default() -> Self {
+        TextFormat {
+            soft_wrap: false,
+            tab_width: 4,
+            max_wrap: 3,
+            max_indent_retain: 4,
+            wrap_indicator: Box::from(" "),
+            viewport_width: 17,
+            wrap_indicator_highlight: None,
+        }
+    }
+}
+
+#[derive(Debug)]
+pub struct DocumentFormatter<'t> {
+    text_fmt: &'t TextFormat,
+    annotations: &'t TextAnnotations,
+
+    /// The visual position at the end of the last yielded word boundary
+    visual_pos: Position,
+    graphemes: RopeGraphemes<'t>,
+    /// The character pos of the `graphemes` iter used for inserting annotations
+    char_pos: usize,
+    /// The line pos of the `graphemes` iter used for inserting annotations
+    line_pos: usize,
+    exhausted: bool,
+
+    /// Line breaks to be reserved for virtual text
+    /// at the next line break
+    virtual_lines: usize,
+    inline_anntoation_graphemes: Option<(Graphemes<'t>, Option<Highlight>)>,
+
+    // softwrap specific
+    /// The indentation of the current line
+    /// Is set to `None` if the indentation level is not yet known
+    /// because no non-whitespace graphemes have been encountered yet
+    indent_level: Option<usize>,
+    /// In case a long word needs to be split a single grapheme might need to be wrapped
+    /// while the rest of the word stays on the same line
+    peeked_grapheme: Option<(FormattedGrapheme<'t>, usize)>,
+    /// A first-in first-out (fifo) buffer for the Graphemes of any given word
+    word_buf: Vec<FormattedGrapheme<'t>>,
+    /// The index of the next grapheme that will be yielded from the `word_buf`
+    word_i: usize,
+}
+
+impl<'t> DocumentFormatter<'t> {
+    /// Creates a new formatter at the last block before `char_idx`.
+    /// A block is a chunk which always ends with a linebreak.
+    /// This is usually just a normal line break.
+    /// However very long lines are always wrapped at constant intervals that can be cheaply calculated
+    /// to avoid pathological behaviour.
+    pub fn new_at_prev_checkpoint(
+        text: RopeSlice<'t>,
+        text_fmt: &'t TextFormat,
+        annotations: &'t TextAnnotations,
+        char_idx: usize,
+    ) -> (Self, usize) {
+        // TODO divide long lines into blocks to avoid bad performance for long lines
+        let block_line_idx = text.char_to_line(char_idx.min(text.len_chars()));
+        let block_char_idx = text.line_to_char(block_line_idx);
+        annotations.reset_pos(block_char_idx);
+        (
+            DocumentFormatter {
+                text_fmt,
+                annotations,
+                visual_pos: Position { row: 0, col: 0 },
+                graphemes: RopeGraphemes::new(text.slice(block_char_idx..)),
+                char_pos: block_char_idx,
+                exhausted: false,
+                virtual_lines: 0,
+                indent_level: None,
+                peeked_grapheme: None,
+                word_buf: Vec::with_capacity(64),
+                word_i: 0,
+                line_pos: block_line_idx,
+                inline_anntoation_graphemes: None,
+            },
+            block_char_idx,
+        )
+    }
+
+    fn next_inline_annotation_grapheme(&mut self) -> Option<(&'t str, Option<Highlight>)> {
+        loop {
+            if let Some(&mut (ref mut annotation, highlight)) =
+                self.inline_anntoation_graphemes.as_mut()
+            {
+                if let Some(grapheme) = annotation.next() {
+                    return Some((grapheme, highlight));
+                }
+            }
+
+            if let Some((annotation, highlight)) =
+                self.annotations.next_inline_annotation_at(self.char_pos)
+            {
+                self.inline_anntoation_graphemes = Some((
+                    UnicodeSegmentation::graphemes(&*annotation.text, true),
+                    highlight,
+                ))
+            } else {
+                return None;
+            }
+        }
+    }
+
+    fn advance_grapheme(&mut self, col: usize) -> Option<FormattedGrapheme<'t>> {
+        let (grapheme, source) =
+            if let Some((grapheme, highlight)) = self.next_inline_annotation_grapheme() {
+                (grapheme.into(), GraphemeSource::VirtualText { highlight })
+            } else if let Some(grapheme) = self.graphemes.next() {
+                self.virtual_lines += self.annotations.annotation_lines_at(self.char_pos);
+                let codepoints = grapheme.len_chars() as u32;
+
+                let overlay = self.annotations.overlay_at(self.char_pos);
+                let grapheme = match overlay {
+                    Some((overlay, _)) => overlay.grapheme.as_str().into(),
+                    None => Cow::from(grapheme).into(),
+                };
+
+                self.char_pos += codepoints as usize;
+                (grapheme, GraphemeSource::Document { codepoints })
+            } else {
+                if self.exhausted {
+                    return None;
+                }
+                self.exhausted = true;
+                // EOF grapheme is required for rendering
+                // and correct position computations
+                return Some(FormattedGrapheme {
+                    grapheme: Grapheme::Other { g: " ".into() },
+                    source: GraphemeSource::Document { codepoints: 0 },
+                });
+            };
+
+        let grapheme = FormattedGrapheme::new(grapheme, col, self.text_fmt.tab_width, source);
+
+        Some(grapheme)
+    }
+
+    /// Move a word to the next visual line
+    fn wrap_word(&mut self, virtual_lines_before_word: usize) -> usize {
+        // softwrap this word to the next line
+        let indent_carry_over = if let Some(indent) = self.indent_level {
+            if indent as u16 <= self.text_fmt.max_indent_retain {
+                indent as u16
+            } else {
+                0
+            }
+        } else {
+            // ensure the indent stays 0
+            self.indent_level = Some(0);
+            0
+        };
+
+        self.visual_pos.col = indent_carry_over as usize;
+        self.virtual_lines -= virtual_lines_before_word;
+        self.visual_pos.row += 1 + virtual_lines_before_word;
+        let mut i = 0;
+        let mut word_width = 0;
+        let wrap_indicator = UnicodeSegmentation::graphemes(&*self.text_fmt.wrap_indicator, true)
+            .map(|g| {
+                i += 1;
+                let grapheme = FormattedGrapheme::new(
+                    g.into(),
+                    self.visual_pos.col + word_width,
+                    self.text_fmt.tab_width,
+                    GraphemeSource::VirtualText {
+                        highlight: self.text_fmt.wrap_indicator_highlight,
+                    },
+                );
+                word_width += grapheme.width();
+                grapheme
+            });
+        self.word_buf.splice(0..0, wrap_indicator);
+
+        for grapheme in &mut self.word_buf[i..] {
+            let visual_x = self.visual_pos.col + word_width;
+            grapheme
+                .grapheme
+                .change_position(visual_x, self.text_fmt.tab_width);
+            word_width += grapheme.width();
+        }
+        word_width
+    }
+
+    fn advance_to_next_word(&mut self) {
+        self.word_buf.clear();
+        let mut word_width = 0;
+        let virtual_lines_before_word = self.virtual_lines;
+        let mut virtual_lines_before_grapheme = self.virtual_lines;
+
+        loop {
+            // softwrap word if necessary
+            if word_width + self.visual_pos.col >= self.text_fmt.viewport_width as usize {
+                // wrapping this word would move too much text to the next line
+                // split the word at the line end instead
+                if word_width > self.text_fmt.max_wrap as usize {
+                    // Usually we stop accomulating graphemes as soon as softwrapping becomes necessary.
+                    // However if the last grapheme is multiple columns wide it might extend beyond the EOL.
+                    // The condition below ensures that this grapheme is not cutoff and instead wrapped to the next line
+                    if word_width + self.visual_pos.col > self.text_fmt.viewport_width as usize {
+                        self.peeked_grapheme = self.word_buf.pop().map(|grapheme| {
+                            (grapheme, self.virtual_lines - virtual_lines_before_grapheme)
+                        });
+                        self.virtual_lines = virtual_lines_before_grapheme;
+                    }
+                    return;
+                }
+
+                word_width = self.wrap_word(virtual_lines_before_word);
+            }
+
+            virtual_lines_before_grapheme = self.virtual_lines;
+
+            let grapheme = if let Some((grapheme, virtual_lines)) = self.peeked_grapheme.take() {
+                self.virtual_lines += virtual_lines;
+                grapheme
+            } else if let Some(grapheme) = self.advance_grapheme(self.visual_pos.col + word_width) {
+                grapheme
+            } else {
+                return;
+            };
+
+            // Track indentation
+            if !grapheme.is_whitespace() && self.indent_level.is_none() {
+                self.indent_level = Some(self.visual_pos.col);
+            } else if grapheme.grapheme == Grapheme::Newline {
+                self.indent_level = None;
+            }
+
+            let is_word_boundary = grapheme.is_word_boundary();
+            word_width += grapheme.width();
+            self.word_buf.push(grapheme);
+
+            if is_word_boundary {
+                return;
+            }
+        }
+    }
+
+    /// returns the document line pos of the **next** grapheme that will be yielded
+    pub fn line_pos(&self) -> usize {
+        self.line_pos
+    }
+
+    /// returns the visual pos of the **next** grapheme that will be yielded
+    pub fn visual_pos(&self) -> Position {
+        self.visual_pos
+    }
+}
+
+impl<'t> Iterator for DocumentFormatter<'t> {
+    type Item = (FormattedGrapheme<'t>, Position);
+
+    fn next(&mut self) -> Option<Self::Item> {
+        let grapheme = if self.text_fmt.soft_wrap {
+            if self.word_i >= self.word_buf.len() {
+                self.advance_to_next_word();
+                self.word_i = 0;
+            }
+            let grapheme = replace(
+                self.word_buf.get_mut(self.word_i)?,
+                FormattedGrapheme::placeholder(),
+            );
+            self.word_i += 1;
+            grapheme
+        } else {
+            self.advance_grapheme(self.visual_pos.col)?
+        };
+
+        let pos = self.visual_pos;
+        if grapheme.grapheme == Grapheme::Newline {
+            self.visual_pos.row += 1;
+            self.visual_pos.row += take(&mut self.virtual_lines);
+            self.visual_pos.col = 0;
+            self.line_pos += 1;
+        } else {
+            self.visual_pos.col += grapheme.width();
+        }
+        Some((grapheme, pos))
+    }
+}

helix-core/src/doc_formatter/test.rs

@@ -0,0 +1,222 @@
+use std::rc::Rc;
+
+use crate::doc_formatter::{DocumentFormatter, TextFormat};
+use crate::text_annotations::{InlineAnnotation, Overlay, TextAnnotations};
+
+impl TextFormat {
+    fn new_test(softwrap: bool) -> Self {
+        TextFormat {
+            soft_wrap: softwrap,
+            tab_width: 2,
+            max_wrap: 3,
+            max_indent_retain: 4,
+            wrap_indicator: ".".into(),
+            wrap_indicator_highlight: None,
+            // use a prime number to allow lining up too often with repeat
+            viewport_width: 17,
+        }
+    }
+}
+
+impl<'t> DocumentFormatter<'t> {
+    fn collect_to_str(&mut self) -> String {
+        use std::fmt::Write;
+        let mut res = String::new();
+        let viewport_width = self.text_fmt.viewport_width;
+        let mut line = 0;
+
+        for (grapheme, pos) in self {
+            if pos.row != line {
+                line += 1;
+                assert_eq!(pos.row, line);
+                write!(res, "\n{}", ".".repeat(pos.col)).unwrap();
+                assert!(
+                    pos.col <= viewport_width as usize,
+                    "softwrapped failed {}<={viewport_width}",
+                    pos.col
+                );
+            }
+            write!(res, "{}", grapheme.grapheme).unwrap();
+        }
+
+        res
+    }
+}
+
+fn softwrap_text(text: &str) -> String {
+    DocumentFormatter::new_at_prev_checkpoint(
+        text.into(),
+        &TextFormat::new_test(true),
+        &TextAnnotations::default(),
+        0,
+    )
+    .0
+    .collect_to_str()
+}
+
+#[test]
+fn basic_softwrap() {
+    assert_eq!(
+        softwrap_text(&"foo ".repeat(10)),
+        "foo foo foo foo \n.foo foo foo foo \n.foo foo  "
+    );
+    assert_eq!(
+        softwrap_text(&"fooo ".repeat(10)),
+        "fooo fooo fooo \n.fooo fooo fooo \n.fooo fooo fooo \n.fooo  "
+    );
+
+    // check that we don't wrap unnecessarily
+    assert_eq!(softwrap_text("\t\txxxx1xxxx2xx\n"), "    xxxx1xxxx2xx \n ");
+}
+
+#[test]
+fn softwrap_indentation() {
+    assert_eq!(
+        softwrap_text("\t\tfoo1 foo2 foo3 foo4 foo5 foo6\n"),
+        "    foo1 foo2 \n.....foo3 foo4 \n.....foo5 foo6 \n "
+    );
+    assert_eq!(
+        softwrap_text("\t\t\tfoo1 foo2 foo3 foo4 foo5 foo6\n"),
+        "      foo1 foo2 \n.foo3 foo4 foo5 \n.foo6 \n "
+    );
+}
+
+#[test]
+fn long_word_softwrap() {
+    assert_eq!(
+        softwrap_text("\t\txxxx1xxxx2xxxx3xxxx4xxxx5xxxx6xxxx7xxxx8xxxx9xxx\n"),
+        "    xxxx1xxxx2xxx\n.....x3xxxx4xxxx5\n.....xxxx6xxxx7xx\n.....xx8xxxx9xxx \n "
+    );
+    assert_eq!(
+        softwrap_text("xxxxxxxx1xxxx2xxx\n"),
+        "xxxxxxxx1xxxx2xxx\n. \n "
+    );
+    assert_eq!(
+        softwrap_text("\t\txxxx1xxxx 2xxxx3xxxx4xxxx5xxxx6xxxx7xxxx8xxxx9xxx\n"),
+        "    xxxx1xxxx \n.....2xxxx3xxxx4x\n.....xxx5xxxx6xxx\n.....x7xxxx8xxxx9\n.....xxx \n "
+    );
+    assert_eq!(
+        softwrap_text("\t\txxxx1xxx 2xxxx3xxxx4xxxx5xxxx6xxxx7xxxx8xxxx9xxx\n"),
+        "    xxxx1xxx 2xxx\n.....x3xxxx4xxxx5\n.....xxxx6xxxx7xx\n.....xx8xxxx9xxx \n "
+    );
+}
+
+fn overlay_text(text: &str, char_pos: usize, softwrap: bool, overlays: &[Overlay]) -> String {
+    DocumentFormatter::new_at_prev_checkpoint(
+        text.into(),
+        &TextFormat::new_test(softwrap),
+        TextAnnotations::default().add_overlay(overlays.into(), None),
+        char_pos,
+    )
+    .0
+    .collect_to_str()
+}
+
+#[test]
+fn overlay() {
+    assert_eq!(
+        overlay_text(
+            "foobar",
+            0,
+            false,
+            &[
+                Overlay {
+                    char_idx: 0,
+                    grapheme: "X".into(),
+                },
+                Overlay {
+                    char_idx: 2,
+                    grapheme: "\t".into(),
+                },
+            ]
+        ),
+        "Xo  bar "
+    );
+    assert_eq!(
+        overlay_text(
+            &"foo ".repeat(10),
+            0,
+            true,
+            &[
+                Overlay {
+                    char_idx: 2,
+                    grapheme: "\t".into(),
+                },
+                Overlay {
+                    char_idx: 5,
+                    grapheme: "\t".into(),
+                },
+                Overlay {
+                    char_idx: 16,
+                    grapheme: "X".into(),
+                },
+            ]
+        ),
+        "fo   f  o foo \n.foo Xoo foo foo \n.foo foo foo  "
+    );
+}
+
+fn annotate_text(text: &str, softwrap: bool, annotations: &[InlineAnnotation]) -> String {
+    DocumentFormatter::new_at_prev_checkpoint(
+        text.into(),
+        &TextFormat::new_test(softwrap),
+        TextAnnotations::default().add_inline_annotations(annotations.into(), None),
+        0,
+    )
+    .0
+    .collect_to_str()
+}
+
+#[test]
+fn annotation() {
+    assert_eq!(
+        annotate_text(
+            "bar",
+            false,
+            &[InlineAnnotation {
+                char_idx: 0,
+                text: "foo".into(),
+            }]
+        ),
+        "foobar "
+    );
+    assert_eq!(
+        annotate_text(
+            &"foo ".repeat(10),
+            true,
+            &[InlineAnnotation {
+                char_idx: 0,
+                text: "foo ".into(),
+            }]
+        ),
+        "foo foo foo foo \n.foo foo foo foo \n.foo foo foo  "
+    );
+}
+#[test]
+fn annotation_and_overlay() {
+    assert_eq!(
+        DocumentFormatter::new_at_prev_checkpoint(
+            "bbar".into(),
+            &TextFormat::new_test(false),
+            TextAnnotations::default()
+                .add_inline_annotations(
+                    Rc::new([InlineAnnotation {
+                        char_idx: 0,
+                        text: "fooo".into(),
+                    }]),
+                    None
+                )
+                .add_overlay(
+                    Rc::new([Overlay {
+                        char_idx: 0,
+                        grapheme: "\t".into(),
+                    }]),
+                    None
+                ),
+            0,
+        )
+        .0
+        .collect_to_str(),
+        "fooo  bar "
+    );
+}

helix-core/src/graphemes.rs

@@ -5,7 +5,88 @@ use ropey::{iter::Chunks, str_utils::byte_to_char_idx, RopeSlice};
 use unicode_segmentation::{GraphemeCursor, GraphemeIncomplete};
 use unicode_width::UnicodeWidthStr;
 
-use std::fmt;
+use std::borrow::Cow;
+use std::fmt::{self, Debug, Display};
+use std::marker::PhantomData;
+use std::ops::Deref;
+use std::ptr::NonNull;
+use std::{slice, str};
+
+use crate::chars::{char_is_whitespace, char_is_word};
+use crate::LineEnding;
+
+#[inline]
+pub fn tab_width_at(visual_x: usize, tab_width: u16) -> usize {
+    tab_width as usize - (visual_x % tab_width as usize)
+}
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum Grapheme<'a> {
+    Newline,
+    Tab { width: usize },
+    Other { g: GraphemeStr<'a> },
+}
+
+impl<'a> Grapheme<'a> {
+    pub fn new(g: GraphemeStr<'a>, visual_x: usize, tab_width: u16) -> Grapheme<'a> {
+        match g {
+            g if g == "\t" => Grapheme::Tab {
+                width: tab_width_at(visual_x, tab_width),
+            },
+            _ if LineEnding::from_str(&g).is_some() => Grapheme::Newline,
+            _ => Grapheme::Other { g },
+        }
+    }
+
+    pub fn change_position(&mut self, visual_x: usize, tab_width: u16) {
+        if let Grapheme::Tab { width } = self {
+            *width = tab_width_at(visual_x, tab_width)
+        }
+    }
+
+    /// Returns the a visual width of this grapheme,
+    #[inline]
+    pub fn width(&self) -> usize {
+        match *self {
+            // width is not cached because we are dealing with
+            // ASCII almost all the time which already has a fastpath
+            // it's okay to convert to u16 here because no codepoint has a width larger
+            // than 2 and graphemes are usually atmost two visible codepoints wide
+            Grapheme::Other { ref g } => grapheme_width(g),
+            Grapheme::Tab { width } => width,
+            Grapheme::Newline => 1,
+        }
+    }
+
+    pub fn is_whitespace(&self) -> bool {
+        !matches!(&self, Grapheme::Other { g } if !g.chars().all(char_is_whitespace))
+    }
+
+    // TODO currently word boundaries are used for softwrapping.
+    // This works best for programming languages and well for prose.
+    // This could however be improved in the future by considering unicode
+    // character classes but
+    pub fn is_word_boundary(&self) -> bool {
+        !matches!(&self, Grapheme::Other { g,.. } if g.chars().all(char_is_word))
+    }
+}
+
+impl Display for Grapheme<'_> {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        match *self {
+            Grapheme::Newline => write!(f, " "),
+            Grapheme::Tab { width } => {
+                for _ in 0..width {
+                    write!(f, " ")?;
+                }
+                Ok(())
+            }
+            Grapheme::Other { ref g } => {
+                write!(f, "{g}")
+            }
+        }
+    }
+}
 
 #[must_use]
 pub fn grapheme_width(g: &str) -> usize {
@@ -27,6 +108,8 @@ pub fn grapheme_width(g: &str) -> usize {
         // We use max(1) here because all grapeheme clusters--even illformed
         // ones--should have at least some width so they can be edited
         // properly.
+        // TODO properly handle unicode width for all codepoints
+        // example of where unicode width is currently wrong: 🤦🏼‍♂️ (taken from https://hsivonen.fi/string-length/)
         UnicodeWidthStr::width(g).max(1)
     }
 }
@@ -341,3 +424,101 @@ impl<'a> Iterator for RopeGraphemes<'a> {
         }
     }
 }
+
+/// A highly compressed Cow<'a, str> that holds
+/// atmost u31::MAX bytes and is readonly
+pub struct GraphemeStr<'a> {
+    ptr: NonNull<u8>,
+    len: u32,
+    phantom: PhantomData<&'a str>,
+}
+
+impl GraphemeStr<'_> {
+    const MASK_OWNED: u32 = 1 << 31;
+
+    fn compute_len(&self) -> usize {
+        (self.len & !Self::MASK_OWNED) as usize
+    }
+}
+
+impl Deref for GraphemeStr<'_> {
+    type Target = str;
+    fn deref(&self) -> &Self::Target {
+        unsafe {
+            let bytes = slice::from_raw_parts(self.ptr.as_ptr(), self.compute_len());
+            str::from_utf8_unchecked(bytes)
+        }
+    }
+}
+
+impl Drop for GraphemeStr<'_> {
+    fn drop(&mut self) {
+        if self.len & Self::MASK_OWNED != 0 {
+            // free allocation
+            unsafe {
+                drop(Box::from_raw(slice::from_raw_parts_mut(
+                    self.ptr.as_ptr(),
+                    self.compute_len(),
+                )));
+            }
+        }
+    }
+}
+
+impl<'a> From<&'a str> for GraphemeStr<'a> {
+    fn from(g: &'a str) -> Self {
+        GraphemeStr {
+            ptr: unsafe { NonNull::new_unchecked(g.as_bytes().as_ptr() as *mut u8) },
+            len: i32::try_from(g.len()).unwrap() as u32,
+            phantom: PhantomData,
+        }
+    }
+}
+
+impl<'a> From<String> for GraphemeStr<'a> {
+    fn from(g: String) -> Self {
+        let len = g.len();
+        let ptr = Box::into_raw(g.into_bytes().into_boxed_slice()) as *mut u8;
+        GraphemeStr {
+            ptr: unsafe { NonNull::new_unchecked(ptr) },
+            len: i32::try_from(len).unwrap() as u32,
+            phantom: PhantomData,
+        }
+    }
+}
+
+impl<'a> From<Cow<'a, str>> for GraphemeStr<'a> {
+    fn from(g: Cow<'a, str>) -> Self {
+        match g {
+            Cow::Borrowed(g) => g.into(),
+            Cow::Owned(g) => g.into(),
+        }
+    }
+}
+
+impl<T: Deref<Target = str>> PartialEq<T> for GraphemeStr<'_> {
+    fn eq(&self, other: &T) -> bool {
+        self.deref() == other.deref()
+    }
+}
+impl PartialEq<str> for GraphemeStr<'_> {
+    fn eq(&self, other: &str) -> bool {
+        self.deref() == other
+    }
+}
+impl Eq for GraphemeStr<'_> {}
+impl Debug for GraphemeStr<'_> {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        Debug::fmt(self.deref(), f)
+    }
+}
+impl Display for GraphemeStr<'_> {
+    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+        Display::fmt(self.deref(), f)
+    }
+}
+impl Clone for GraphemeStr<'_> {
+    fn clone(&self) -> Self {
+        self.deref().to_owned().into()
+    }
+}

helix-core/src/lib.rs

@@ -6,6 +6,7 @@ pub mod comment;
 pub mod config;
 pub mod diagnostic;
 pub mod diff;
+pub mod doc_formatter;
 pub mod graphemes;
 pub mod history;
 pub mod increment;
@@ -24,6 +25,7 @@ pub mod shellwords;
 pub mod surround;
 pub mod syntax;
 pub mod test;
+pub mod text_annotations;
 pub mod textobject;
 mod transaction;
 pub mod wrap;
@@ -95,8 +97,12 @@ pub use {regex, tree_sitter};
 
 pub use graphemes::RopeGraphemes;
 pub use position::{
-    coords_at_pos, pos_at_coords, pos_at_visual_coords, visual_coords_at_pos, Position,
+    char_idx_at_visual_offset, coords_at_pos, pos_at_coords, visual_offset_from_anchor,
+    visual_offset_from_block, Position,
 };
+#[allow(deprecated)]
+pub use position::{pos_at_visual_coords, visual_coords_at_pos};
+
 pub use selection::{Range, Selection};
 pub use smallvec::{smallvec, SmallVec};
 pub use syntax::Syntax;

helix-core/src/movement.rs

@@ -4,16 +4,19 @@ use ropey::iter::Chars;
 use tree_sitter::{Node, QueryCursor};
 
 use crate::{
+    char_idx_at_visual_offset,
     chars::{categorize_char, char_is_line_ending, CharCategory},
+    doc_formatter::TextFormat,
     graphemes::{
         next_grapheme_boundary, nth_next_grapheme_boundary, nth_prev_grapheme_boundary,
         prev_grapheme_boundary,
     },
     line_ending::rope_is_line_ending,
-    pos_at_visual_coords,
+    position::char_idx_at_visual_block_offset,
     syntax::LanguageConfiguration,
+    text_annotations::TextAnnotations,
     textobject::TextObject,
-    visual_coords_at_pos, Position, Range, RopeSlice,
+    visual_offset_from_block, Range, RopeSlice,
 };
 
 #[derive(Debug, Copy, Clone, PartialEq, Eq)]
@@ -34,7 +37,8 @@ pub fn move_horizontally(
     dir: Direction,
     count: usize,
     behaviour: Movement,
-    _: usize,
+    _: &TextFormat,
+    _: &mut TextAnnotations,
 ) -> Range {
     let pos = range.cursor(slice);
 
@@ -48,35 +52,116 @@ pub fn move_horizontally(
     range.put_cursor(slice, new_pos, behaviour == Movement::Extend)
 }
 
+pub fn move_vertically_visual(
+    slice: RopeSlice,
+    range: Range,
+    dir: Direction,
+    count: usize,
+    behaviour: Movement,
+    text_fmt: &TextFormat,
+    annotations: &mut TextAnnotations,
+) -> Range {
+    if !text_fmt.soft_wrap {
+        move_vertically(slice, range, dir, count, behaviour, text_fmt, annotations);
+    }
+    annotations.clear_line_annotations();
+    let pos = range.cursor(slice);
+
+    // Compute the current position's 2d coordinates.
+    let (visual_pos, block_off) = visual_offset_from_block(slice, pos, pos, text_fmt, annotations);
+    let new_col = range
+        .old_visual_position
+        .map_or(visual_pos.col as u32, |(_, col)| col);
+
+    // Compute the new position.
+    let mut row_off = match dir {
+        Direction::Forward => count as isize,
+        Direction::Backward => -(count as isize),
+    };
+
+    // TODO how to handle inline annotations that span an entire visual line (very unlikely).
+
+    // Compute visual offset relative to block start to avoid trasversing the block twice
+    row_off += visual_pos.row as isize;
+    let new_pos = char_idx_at_visual_offset(
+        slice,
+        block_off,
+        row_off,
+        new_col as usize,
+        text_fmt,
+        annotations,
+    )
+    .0;
+
+    // Special-case to avoid moving to the end of the last non-empty line.
+    if behaviour == Movement::Extend && slice.line(slice.char_to_line(new_pos)).len_chars() == 0 {
+        return range;
+    }
+
+    let mut new_range = range.put_cursor(slice, new_pos, behaviour == Movement::Extend);
+    new_range.old_visual_position = Some((0, new_col));
+    new_range
+}
+
 pub fn move_vertically(
     slice: RopeSlice,
     range: Range,
     dir: Direction,
     count: usize,
     behaviour: Movement,
-    tab_width: usize,
+    text_fmt: &TextFormat,
+    annotations: &mut TextAnnotations,
 ) -> Range {
+    annotations.clear_line_annotations();
     let pos = range.cursor(slice);
+    let line_idx = slice.char_to_line(pos);
+    let line_start = slice.line_to_char(line_idx);
 
     // Compute the current position's 2d coordinates.
-    let Position { row, col } = visual_coords_at_pos(slice, pos, tab_width);
-    let horiz = range.horiz.unwrap_or(col as u32);
+    let visual_pos = visual_offset_from_block(slice, line_start, pos, text_fmt, annotations).0;
+    let (mut new_row, new_col) = range
+        .old_visual_position
+        .map_or((visual_pos.row as u32, visual_pos.col as u32), |pos| pos);
+    new_row = new_row.max(visual_pos.row as u32);
+    let line_idx = slice.char_to_line(pos);
 
     // Compute the new position.
-    let new_row = match dir {
-        Direction::Forward => (row + count).min(slice.len_lines().saturating_sub(1)),
-        Direction::Backward => row.saturating_sub(count),
+    let mut new_line_idx = match dir {
+        Direction::Forward => line_idx.saturating_add(count),
+        Direction::Backward => line_idx.saturating_sub(count),
     };
-    let new_col = col.max(horiz as usize);
-    let new_pos = pos_at_visual_coords(slice, Position::new(new_row, new_col), tab_width);
+
+    let line = if new_line_idx >= slice.len_lines() - 1 {
+        // there is no line terminator for the last line
+        // so the logic below is not necessary here
+        new_line_idx = slice.len_lines() - 1;
+        slice
+    } else {
+        // char_idx_at_visual_block_offset returns a one-past-the-end index
+        // in case it reaches the end of the slice
+        // to avoid moving to the nextline in that case the line terminator is removed from the line
+        let new_line_end = prev_grapheme_boundary(slice, slice.line_to_char(new_line_idx + 1));
+        slice.slice(..new_line_end)
+    };
+
+    let new_line_start = line.line_to_char(new_line_idx);
+
+    let (new_pos, _) = char_idx_at_visual_block_offset(
+        line,
+        new_line_start,
+        new_row as usize,
+        new_col as usize,
+        text_fmt,
+        annotations,
+    );
 
     // Special-case to avoid moving to the end of the last non-empty line.
-    if behaviour == Movement::Extend && slice.line(new_row).len_chars() == 0 {
+    if behaviour == Movement::Extend && slice.line(new_line_idx).len_chars() == 0 {
         return range;
     }
 
     let mut new_range = range.put_cursor(slice, new_pos, behaviour == Movement::Extend);
-    new_range.horiz = Some(horiz);
+    new_range.old_visual_position = Some((new_row, new_col));
     new_range
 }
 
@@ -473,7 +558,16 @@ mod test {
         assert_eq!(
             coords_at_pos(
                 slice,
-                move_vertically(slice, range, Direction::Forward, 1, Movement::Move, 4).head
+                move_vertically_visual(
+                    slice,
+                    range,
+                    Direction::Forward,
+                    1,
+                    Movement::Move,
+                    &TextFormat::default(),
+                    &mut TextAnnotations::default(),
+                )
+                .head
             ),
             (1, 3).into()
         );
@@ -497,7 +591,15 @@ mod test {
         ];
 
         for ((direction, amount), coordinates) in moves_and_expected_coordinates {
-            range = move_horizontally(slice, range, direction, amount, Movement::Move, 0);
+            range = move_horizontally(
+                slice,
+                range,
+                direction,
+                amount,
+                Movement::Move,
+                &TextFormat::default(),
+                &mut TextAnnotations::default(),
+            );
             assert_eq!(coords_at_pos(slice, range.head), coordinates.into())
         }
     }
@@ -523,7 +625,15 @@ mod test {
         ];
 
         for ((direction, amount), coordinates) in moves_and_expected_coordinates {
-            range = move_horizontally(slice, range, direction, amount, Movement::Move, 0);
+            range = move_horizontally(
+                slice,
+                range,
+                direction,
+                amount,
+                Movement::Move,
+                &TextFormat::default(),
+                &mut TextAnnotations::default(),
+            );
             assert_eq!(coords_at_pos(slice, range.head), coordinates.into());
             assert_eq!(range.head, range.anchor);
         }
@@ -545,7 +655,15 @@ mod test {
         ];
 
         for (direction, amount) in moves {
-            range = move_horizontally(slice, range, direction, amount, Movement::Extend, 0);
+            range = move_horizontally(
+                slice,
+                range,
+                direction,
+                amount,
+                Movement::Extend,
+                &TextFormat::default(),
+                &mut TextAnnotations::default(),
+            );
             assert_eq!(range.anchor, original_anchor);
         }
     }
@@ -569,7 +687,15 @@ mod test {
         ];
 
         for ((direction, amount), coordinates) in moves_and_expected_coordinates {
-            range = move_vertically(slice, range, direction, amount, Movement::Move, 4);
+            range = move_vertically_visual(
+                slice,
+                range,
+                direction,
+                amount,
+                Movement::Move,
+                &TextFormat::default(),
+                &mut TextAnnotations::default(),
+            );
             assert_eq!(coords_at_pos(slice, range.head), coordinates.into());
             assert_eq!(range.head, range.anchor);
         }
@@ -603,8 +729,24 @@ mod test {
 
         for ((axis, direction, amount), coordinates) in moves_and_expected_coordinates {
             range = match axis {
-                Axis::H => move_horizontally(slice, range, direction, amount, Movement::Move, 0),
-                Axis::V => move_vertically(slice, range, direction, amount, Movement::Move, 4),
+                Axis::H => move_horizontally(
+                    slice,
+                    range,
+                    direction,
+                    amount,
+                    Movement::Move,
+                    &TextFormat::default(),
+                    &mut TextAnnotations::default(),
+                ),
+                Axis::V => move_vertically_visual(
+                    slice,
+                    range,
+                    direction,
+                    amount,
+                    Movement::Move,
+                    &TextFormat::default(),
+                    &mut TextAnnotations::default(),
+                ),
             };
             assert_eq!(coords_at_pos(slice, range.head), coordinates.into());
             assert_eq!(range.head, range.anchor);
@@ -638,8 +780,24 @@ mod test {
 
         for ((axis, direction, amount), coordinates) in moves_and_expected_coordinates {
             range = match axis {
-                Axis::H => move_horizontally(slice, range, direction, amount, Movement::Move, 0),
-                Axis::V => move_vertically(slice, range, direction, amount, Movement::Move, 4),
+                Axis::H => move_horizontally(
+                    slice,
+                    range,
+                    direction,
+                    amount,
+                    Movement::Move,
+                    &TextFormat::default(),
+                    &mut TextAnnotations::default(),
+                ),
+                Axis::V => move_vertically_visual(
+                    slice,
+                    range,
+                    direction,
+                    amount,
+                    Movement::Move,
+                    &TextFormat::default(),
+                    &mut TextAnnotations::default(),
+                ),
             };
             assert_eq!(coords_at_pos(slice, range.head), coordinates.into());
             assert_eq!(range.head, range.anchor);

helix-core/src/position.rs

@@ -1,9 +1,11 @@
-use std::borrow::Cow;
+use std::{borrow::Cow, cmp::Ordering};
 
 use crate::{
     chars::char_is_line_ending,
+    doc_formatter::{DocumentFormatter, TextFormat},
     graphemes::{ensure_grapheme_boundary_prev, grapheme_width, RopeGraphemes},
     line_ending::line_end_char_index,
+    text_annotations::TextAnnotations,
     RopeSlice,
 };
 
@@ -73,6 +75,13 @@ pub fn coords_at_pos(text: RopeSlice, pos: usize) -> Position {
 /// Takes \t, double-width characters (CJK) into account as well as text
 /// not in the document in the future.
 /// See [`coords_at_pos`] for an "objective" one.
+///
+/// This function should be used very rarely. Usually `visual_offset_from_anchor`
+/// or `visual_offset_from_block` is preferable. However when you want to compute the
+/// actual visual row/column in the text (not what is actually shown on screen)
+/// then you should use this function. For example aligning text should ignore virtual
+/// text and softwrap.
+#[deprecated = "Doesn't account for softwrap or decorations, use visual_offset_from_anchor instead"]
 pub fn visual_coords_at_pos(text: RopeSlice, pos: usize, tab_width: usize) -> Position {
     let line = text.char_to_line(pos);
 
@@ -93,6 +102,82 @@ pub fn visual_coords_at_pos(text: RopeSlice, pos: usize, tab_width: usize) -> Po
     Position::new(line, col)
 }
 
+/// Returns the visual offset from the start of the first visual line
+/// in the block that contains anchor.
+/// Text is always wrapped at blocks, they usually correspond to
+/// actual line breaks but for very long lines
+/// softwrapping positions are estimated with an O(1) algorithm
+/// to ensure consistent performance for large lines (currently unimplemented)
+///
+/// Usualy you want to use `visual_offset_from_anchor` instead but this function
+/// can be useful (and faster) if
+/// * You already know the visual position of the block
+/// * You only care about the horizontal offset (column) and not the vertical offset (row)
+pub fn visual_offset_from_block(
+    text: RopeSlice,
+    anchor: usize,
+    pos: usize,
+    text_fmt: &TextFormat,
+    annotations: &TextAnnotations,
+) -> (Position, usize) {
+    let mut last_pos = Position::default();
+    let (formatter, block_start) =
+        DocumentFormatter::new_at_prev_checkpoint(text, text_fmt, annotations, anchor);
+    let mut char_pos = block_start;
+
+    for (grapheme, vpos) in formatter {
+        last_pos = vpos;
+        char_pos += grapheme.doc_chars();
+
+        if char_pos > pos {
+            return (last_pos, block_start);
+        }
+    }
+
+    (last_pos, block_start)
+}
+
+/// Returns the visual offset from the start of the visual line
+/// that contains anchor.
+pub fn visual_offset_from_anchor(
+    text: RopeSlice,
+    anchor: usize,
+    pos: usize,
+    text_fmt: &TextFormat,
+    annotations: &TextAnnotations,
+    max_rows: usize,
+) -> Option<(Position, usize)> {
+    let (formatter, block_start) =
+        DocumentFormatter::new_at_prev_checkpoint(text, text_fmt, annotations, anchor);
+    let mut char_pos = block_start;
+    let mut anchor_line = None;
+    let mut last_pos = Position::default();
+
+    for (grapheme, vpos) in formatter {
+        last_pos = vpos;
+        char_pos += grapheme.doc_chars();
+
+        if char_pos > anchor && anchor_line.is_none() {
+            anchor_line = Some(last_pos.row);
+        }
+        if char_pos > pos {
+            last_pos.row -= anchor_line.unwrap();
+            return Some((last_pos, block_start));
+        }
+
+        if let Some(anchor_line) = anchor_line {
+            if vpos.row >= anchor_line + max_rows {
+                return None;
+            }
+        }
+    }
+
+    let anchor_line = anchor_line.unwrap_or(last_pos.row);
+    last_pos.row -= anchor_line;
+
+    Some((last_pos, block_start))
+}
+
 /// Convert (line, column) coordinates to a character index.
 ///
 /// If the `line` coordinate is beyond the end of the file, the EOF
@@ -140,6 +225,11 @@ pub fn pos_at_coords(text: RopeSlice, coords: Position, limit_before_line_ending
 /// If the `column` coordinate is past the end of the given line, the
 /// line-end position (in this case, just before the line ending
 /// character) will be returned.
+/// This function should be used very rarely. Usually `char_idx_at_visual_offset` is preferable.
+/// However when you want to compute a char position from the visual row/column in the text
+/// (not what is actually shown on screen) then you should use this function.
+/// For example aligning text should ignore virtual text and softwrap.
+#[deprecated = "Doesn't account for softwrap or decorations, use char_idx_at_visual_offset instead"]
 pub fn pos_at_visual_coords(text: RopeSlice, coords: Position, tab_width: usize) -> usize {
     let Position { mut row, col } = coords;
     row = row.min(text.len_lines() - 1);
@@ -169,6 +259,120 @@ pub fn pos_at_visual_coords(text: RopeSlice, coords: Position, tab_width: usize)
     line_start + col_char_offset
 }
 
+/// Returns the char index on the visual line `row_offset` below the visual line of
+/// the provided char index `anchor` that is closest to the supplied visual `column`.
+///
+/// If the targeted visual line is entirely covered by virtual text the last
+/// char position before the virtual text and a virtual offset is returned instead.
+///
+/// If no (text) grapheme starts at exactly at the specified column the
+/// start of the grapheme to the left is returned. If there is no grapheme
+/// to the left (for example if the line starts with virtual text) then the positiong
+/// of the next grapheme to the right is returned.
+///
+/// If the `line` coordinate is beyond the end of the file, the EOF
+/// position will be returned.
+///
+/// If the `column` coordinate is past the end of the given line, the
+/// line-end position (in this case, just before the line ending
+/// character) will be returned.
+///
+/// # Returns
+///
+/// `(real_char_idx, virtual_lines)`
+///
+/// The nearest character idx "closest" (see above) to the specified visual offset
+/// on the visual line is returned if the visual line contains any text:
+/// If the visual line at the specified offset is a virtual line generated by a `LineAnnotation`
+/// the previous char_index is returned, together with the remaining vertical offset (`virtual_lines`)
+pub fn char_idx_at_visual_offset<'a>(
+    text: RopeSlice<'a>,
+    mut anchor: usize,
+    mut row_offset: isize,
+    column: usize,
+    text_fmt: &TextFormat,
+    annotations: &TextAnnotations,
+) -> (usize, usize) {
+    // convert row relative to visual line containing anchor to row relative to a block containing anchor (anchor may change)
+    loop {
+        let (visual_pos_in_block, block_char_offset) =
+            visual_offset_from_block(text, anchor, anchor, text_fmt, annotations);
+        row_offset += visual_pos_in_block.row as isize;
+        anchor = block_char_offset;
+        if row_offset >= 0 {
+            break;
+        }
+
+        if block_char_offset == 0 {
+            row_offset = 0;
+            break;
+        }
+        // the row_offset is negative so we need to look at the previous block
+        // set the anchor to the last char before the current block
+        // this char index is also always a line earlier so increase the row_offset by 1
+        anchor -= 1;
+        row_offset += 1;
+    }
+
+    char_idx_at_visual_block_offset(
+        text,
+        anchor,
+        row_offset as usize,
+        column,
+        text_fmt,
+        annotations,
+    )
+}
+
+/// This function behaves the same as `char_idx_at_visual_offset`, except that
+/// the vertical offset `row` is always computed relative to the block that contains `anchor`
+/// instead of the visual line that contains `anchor`.
+/// Usually `char_idx_at_visual_offset` is more useful but this function can be
+/// used in some situations as an optimization when `visual_offset_from_block` was used
+///
+/// # Returns
+///
+/// `(real_char_idx, virtual_lines)`
+///
+/// See `char_idx_at_visual_offset` for details
+pub fn char_idx_at_visual_block_offset(
+    text: RopeSlice,
+    anchor: usize,
+    row: usize,
+    column: usize,
+    text_fmt: &TextFormat,
+    annotations: &TextAnnotations,
+) -> (usize, usize) {
+    let (formatter, mut char_idx) =
+        DocumentFormatter::new_at_prev_checkpoint(text, text_fmt, annotations, anchor);
+    let mut last_char_idx = char_idx;
+    let mut last_char_idx_on_line = None;
+    let mut last_row = 0;
+    for (grapheme, grapheme_pos) in formatter {
+        match grapheme_pos.row.cmp(&row) {
+            Ordering::Equal => {
+                if grapheme_pos.col + grapheme.width() > column {
+                    if !grapheme.is_virtual() {
+                        return (char_idx, 0);
+                    } else if let Some(char_idx) = last_char_idx_on_line {
+                        return (char_idx, 0);
+                    }
+                } else if !grapheme.is_virtual() {
+                    last_char_idx_on_line = Some(char_idx)
+                }
+            }
+            Ordering::Greater => return (last_char_idx, row - last_row),
+            _ => (),
+        }
+
+        last_char_idx = char_idx;
+        last_row = grapheme_pos.row;
+        char_idx += grapheme.doc_chars();
+    }
+
+    (char_idx, 0)
+}
+
 #[cfg(test)]
 mod test {
     use super::*;
@@ -228,6 +432,7 @@ mod test {
     }
 
     #[test]
+    #[allow(deprecated)]
     fn test_visual_coords_at_pos() {
         let text = Rope::from("ḧëḷḷö\nẅöṛḷḋ");
         let slice = text.slice(..);
@@ -275,6 +480,130 @@ mod test {
         assert_eq!(visual_coords_at_pos(slice, 2, 8), (0, 9).into());
     }
 
+    #[test]
+    fn test_visual_off_from_block() {
+        let text = Rope::from("ḧëḷḷö\nẅöṛḷḋ");
+        let slice = text.slice(..);
+        let annot = TextAnnotations::default();
+        let text_fmt = TextFormat::default();
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 0, &text_fmt, &annot).0,
+            (0, 0).into()
+        );
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 5, &text_fmt, &annot).0,
+            (0, 5).into()
+        ); // position on \n
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 6, &text_fmt, &annot).0,
+            (1, 0).into()
+        ); // position on w
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 7, &text_fmt, &annot).0,
+            (1, 1).into()
+        ); // position on o
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 10, &text_fmt, &annot).0,
+            (1, 4).into()
+        ); // position on d
+
+        // Test with wide characters.
+        let text = Rope::from("今日はいい\n");
+        let slice = text.slice(..);
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 0, &text_fmt, &annot).0,
+            (0, 0).into()
+        );
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 1, &text_fmt, &annot).0,
+            (0, 2).into()
+        );
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 2, &text_fmt, &annot).0,
+            (0, 4).into()
+        );
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 3, &text_fmt, &annot).0,
+            (0, 6).into()
+        );
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 4, &text_fmt, &annot).0,
+            (0, 8).into()
+        );
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 5, &text_fmt, &annot).0,
+            (0, 10).into()
+        );
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 6, &text_fmt, &annot).0,
+            (1, 0).into()
+        );
+
+        // Test with grapheme clusters.
+        let text = Rope::from("a̐éö̲\r\n");
+        let slice = text.slice(..);
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 0, &text_fmt, &annot).0,
+            (0, 0).into()
+        );
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 2, &text_fmt, &annot).0,
+            (0, 1).into()
+        );
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 4, &text_fmt, &annot).0,
+            (0, 2).into()
+        );
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 7, &text_fmt, &annot).0,
+            (0, 3).into()
+        );
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 9, &text_fmt, &annot).0,
+            (1, 0).into()
+        );
+
+        // Test with wide-character grapheme clusters.
+        // TODO: account for cluster.
+        let text = Rope::from("किमपि\n");
+        let slice = text.slice(..);
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 0, &text_fmt, &annot).0,
+            (0, 0).into()
+        );
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 2, &text_fmt, &annot).0,
+            (0, 2).into()
+        );
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 3, &text_fmt, &annot).0,
+            (0, 3).into()
+        );
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 5, &text_fmt, &annot).0,
+            (0, 5).into()
+        );
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 6, &text_fmt, &annot).0,
+            (1, 0).into()
+        );
+
+        // Test with tabs.
+        let text = Rope::from("\tHello\n");
+        let slice = text.slice(..);
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 0, &text_fmt, &annot).0,
+            (0, 0).into()
+        );
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 1, &text_fmt, &annot).0,
+            (0, 4).into()
+        );
+        assert_eq!(
+            visual_offset_from_block(slice, 0, 2, &text_fmt, &annot).0,
+            (0, 5).into()
+        );
+    }
     #[test]
     fn test_pos_at_coords() {
         let text = Rope::from("ḧëḷḷö\nẅöṛḷḋ");
@@ -341,6 +670,7 @@ mod test {
     }
 
     #[test]
+    #[allow(deprecated)]
     fn test_pos_at_visual_coords() {
         let text = Rope::from("ḧëḷḷö\nẅöṛḷḋ");
         let slice = text.slice(..);
@@ -405,4 +735,100 @@ mod test {
         assert_eq!(pos_at_visual_coords(slice, (0, 10).into(), 4), 0);
         assert_eq!(pos_at_visual_coords(slice, (10, 10).into(), 4), 0);
     }
+
+    #[test]
+    fn test_char_idx_at_visual_row_offset() {
+        let text = Rope::from("ḧëḷḷö\nẅöṛḷḋ\nfoo");
+        let slice = text.slice(..);
+        let mut text_fmt = TextFormat::default();
+        for i in 0isize..3isize {
+            for j in -2isize..=2isize {
+                if !(0..3).contains(&(i + j)) {
+                    continue;
+                }
+                println!("{i} {j}");
+                assert_eq!(
+                    char_idx_at_visual_offset(
+                        slice,
+                        slice.line_to_char(i as usize),
+                        j,
+                        3,
+                        &text_fmt,
+                        &TextAnnotations::default(),
+                    )
+                    .0,
+                    slice.line_to_char((i + j) as usize) + 3
+                );
+            }
+        }
+
+        text_fmt.soft_wrap = true;
+        let mut softwrapped_text = "foo ".repeat(10);
+        softwrapped_text.push('\n');
+        let last_char = softwrapped_text.len() - 1;
+
+        let text = Rope::from(softwrapped_text.repeat(3));
+        let slice = text.slice(..);
+        assert_eq!(
+            char_idx_at_visual_offset(
+                slice,
+                last_char,
+                0,
+                0,
+                &text_fmt,
+                &TextAnnotations::default(),
+            )
+            .0,
+            32
+        );
+        assert_eq!(
+            char_idx_at_visual_offset(
+                slice,
+                last_char,
+                -1,
+                0,
+                &text_fmt,
+                &TextAnnotations::default(),
+            )
+            .0,
+            16
+        );
+        assert_eq!(
+            char_idx_at_visual_offset(
+                slice,
+                last_char,
+                -2,
+                0,
+                &text_fmt,
+                &TextAnnotations::default(),
+            )
+            .0,
+            0
+        );
+        assert_eq!(
+            char_idx_at_visual_offset(
+                slice,
+                softwrapped_text.len() + last_char,
+                -2,
+                0,
+                &text_fmt,
+                &TextAnnotations::default(),
+            )
+            .0,
+            softwrapped_text.len()
+        );
+
+        assert_eq!(
+            char_idx_at_visual_offset(
+                slice,
+                softwrapped_text.len() + last_char,
+                -5,
+                0,
+                &text_fmt,
+                &TextAnnotations::default(),
+            )
+            .0,
+            0
+        );
+    }
 }

helix-core/src/selection.rs

@@ -53,7 +53,9 @@ pub struct Range {
     pub anchor: usize,
     /// The head of the range, moved when extending.
     pub head: usize,
-    pub horiz: Option<u32>,
+    /// The previous visual offset (softwrapped lines and columns) from
+    /// the start of the line
+    pub old_visual_position: Option<(u32, u32)>,
 }
 
 impl Range {
@@ -61,7 +63,7 @@ impl Range {
         Self {
             anchor,
             head,
-            horiz: None,
+            old_visual_position: None,
         }
     }
 
@@ -127,7 +129,7 @@ impl Range {
         Self {
             anchor: self.head,
             head: self.anchor,
-            horiz: self.horiz,
+            old_visual_position: self.old_visual_position,
         }
     }
 
@@ -185,7 +187,7 @@ impl Range {
         Self {
             anchor,
             head,
-            horiz: None,
+            old_visual_position: None,
         }
     }
 
@@ -198,13 +200,13 @@ impl Range {
             Self {
                 anchor: self.anchor.min(from),
                 head: self.head.max(to),
-                horiz: None,
+                old_visual_position: None,
             }
         } else {
             Self {
                 anchor: self.anchor.max(to),
                 head: self.head.min(from),
-                horiz: None,
+                old_visual_position: None,
             }
         }
     }
@@ -219,13 +221,13 @@ impl Range {
             Range {
                 anchor: self.anchor.max(other.anchor),
                 head: self.head.min(other.head),
-                horiz: None,
+                old_visual_position: None,
             }
         } else {
             Range {
                 anchor: self.from().min(other.from()),
                 head: self.to().max(other.to()),
-                horiz: None,
+                old_visual_position: None,
             }
         }
     }
@@ -279,8 +281,8 @@ impl Range {
         Range {
             anchor: new_anchor,
             head: new_head,
-            horiz: if new_anchor == self.anchor {
-                self.horiz
+            old_visual_position: if new_anchor == self.anchor {
+                self.old_visual_position
             } else {
                 None
             },
@@ -306,7 +308,7 @@ impl Range {
             Range {
                 anchor: self.anchor,
                 head: next_grapheme_boundary(slice, self.head),
-                horiz: self.horiz,
+                old_visual_position: self.old_visual_position,
             }
         } else {
             *self
@@ -378,7 +380,7 @@ impl From<(usize, usize)> for Range {
         Self {
             anchor,
             head,
-            horiz: None,
+            old_visual_position: None,
         }
     }
 }
@@ -482,7 +484,7 @@ impl Selection {
             ranges: smallvec![Range {
                 anchor,
                 head,
-                horiz: None
+                old_visual_position: None
             }],
             primary_index: 0,
         }
@@ -566,9 +568,9 @@ impl Selection {
     }
 
     /// Takes a closure and maps each `Range` over the closure.
-    pub fn transform<F>(mut self, f: F) -> Self
+    pub fn transform<F>(mut self, mut f: F) -> Self
     where
-        F: Fn(Range) -> Range,
+        F: FnMut(Range) -> Range,
     {
         for range in self.ranges.iter_mut() {
             *range = f(*range)

helix-core/src/text_annotations.rs

@@ -0,0 +1,271 @@
+use std::cell::Cell;
+use std::convert::identity;
+use std::ops::Range;
+use std::rc::Rc;
+
+use crate::syntax::Highlight;
+use crate::Tendril;
+
+/// An inline annotation is continuous text shown
+/// on the screen before the grapheme that starts at
+/// `char_idx`
+#[derive(Debug, Clone)]
+pub struct InlineAnnotation {
+    pub text: Tendril,
+    pub char_idx: usize,
+}
+
+/// Represents a **single Grapheme** that is part of the document
+/// that start at `char_idx` that will be replaced with
+/// a different `grapheme`.
+/// If `grapheme` contains multiple graphemes the text
+/// will render incorrectly.
+/// If you want to overlay multiple graphemes simply
+/// use multiple `Overlays`.
+///
+/// # Examples
+///
+/// The following examples are valid overlays for the following text:
+///
+/// `aX͎̊͢͜͝͡bc`
+///
+/// ```
+/// use helix_core::text_annotations::Overlay;
+///
+/// // replaces a
+/// Overlay {
+///   char_idx: 0,
+///   grapheme: "X".into(),
+/// };
+///
+/// // replaces X͎̊͢͜͝͡
+/// Overlay{
+///   char_idx: 1,
+///   grapheme: "\t".into(),
+/// };
+///
+/// // replaces b
+/// Overlay{
+///   char_idx: 6,
+///   grapheme: "X̢̢̟͖̲͌̋̇͑͝".into(),
+/// };
+/// ```
+///
+/// The following examples are invalid uses
+///
+/// ```
+/// use helix_core::text_annotations::Overlay;
+///
+/// // overlay is not aligned at grapheme boundary
+/// Overlay{
+///   char_idx: 3,
+///   grapheme: "x".into(),
+/// };
+///
+/// // overlay contains multiple graphemes
+/// Overlay{
+///   char_idx: 0,
+///   grapheme: "xy".into(),
+/// };
+/// ```
+#[derive(Debug, Clone)]
+pub struct Overlay {
+    pub char_idx: usize,
+    pub grapheme: Tendril,
+}
+
+/// Line annotations allow for virtual text between normal
+/// text lines. They cause `height` empty lines to be inserted
+/// below the document line that contains `anchor_char_idx`.
+///
+/// These lines can be filled with text in the rendering code
+/// as their contents have no effect beyond visual appearance.
+///
+/// To insert a line after a document line simply set
+/// `anchor_char_idx` to `doc.line_to_char(line_idx)`
+#[derive(Debug, Clone)]
+pub struct LineAnnotation {
+    pub anchor_char_idx: usize,
+    pub height: usize,
+}
+
+#[derive(Debug)]
+struct Layer<A, M> {
+    annotations: Rc<[A]>,
+    current_index: Cell<usize>,
+    metadata: M,
+}
+
+impl<A, M: Clone> Clone for Layer<A, M> {
+    fn clone(&self) -> Self {
+        Layer {
+            annotations: self.annotations.clone(),
+            current_index: self.current_index.clone(),
+            metadata: self.metadata.clone(),
+        }
+    }
+}
+
+impl<A, M> Layer<A, M> {
+    pub fn reset_pos(&self, char_idx: usize, get_char_idx: impl Fn(&A) -> usize) {
+        let new_index = self
+            .annotations
+            .binary_search_by_key(&char_idx, get_char_idx)
+            .unwrap_or_else(identity);
+
+        self.current_index.set(new_index);
+    }
+
+    pub fn consume(&self, char_idx: usize, get_char_idx: impl Fn(&A) -> usize) -> Option<&A> {
+        let annot = self.annotations.get(self.current_index.get())?;
+        debug_assert!(get_char_idx(annot) >= char_idx);
+        if get_char_idx(annot) == char_idx {
+            self.current_index.set(self.current_index.get() + 1);
+            Some(annot)
+        } else {
+            None
+        }
+    }
+}
+
+impl<A, M> From<(Rc<[A]>, M)> for Layer<A, M> {
+    fn from((annotations, metadata): (Rc<[A]>, M)) -> Layer<A, M> {
+        Layer {
+            annotations,
+            current_index: Cell::new(0),
+            metadata,
+        }
+    }
+}
+
+fn reset_pos<A, M>(layers: &[Layer<A, M>], pos: usize, get_pos: impl Fn(&A) -> usize) {
+    for layer in layers {
+        layer.reset_pos(pos, &get_pos)
+    }
+}
+
+/// Annotations that change that is displayed when the document is render.
+/// Also commonly called virtual text.
+#[derive(Default, Debug, Clone)]
+pub struct TextAnnotations {
+    inline_annotations: Vec<Layer<InlineAnnotation, Option<Highlight>>>,
+    overlays: Vec<Layer<Overlay, Option<Highlight>>>,
+    line_annotations: Vec<Layer<LineAnnotation, ()>>,
+}
+
+impl TextAnnotations {
+    /// Prepare the TextAnnotations for iteration starting at char_idx
+    pub fn reset_pos(&self, char_idx: usize) {
+        reset_pos(&self.inline_annotations, char_idx, |annot| annot.char_idx);
+        reset_pos(&self.overlays, char_idx, |annot| annot.char_idx);
+        reset_pos(&self.line_annotations, char_idx, |annot| {
+            annot.anchor_char_idx
+        });
+    }
+
+    pub fn collect_overlay_highlights(
+        &self,
+        char_range: Range<usize>,
+    ) -> Vec<(usize, Range<usize>)> {
+        let mut highlights = Vec::new();
+        self.reset_pos(char_range.start);
+        for char_idx in char_range {
+            if let Some((_, Some(highlight))) = self.overlay_at(char_idx) {
+                // we don't know the number of chars the original grapheme takes
+                // however it doesn't matter as highlight bounderies are automatically
+                // aligned to grapheme boundaries in the rendering code
+                highlights.push((highlight.0, char_idx..char_idx + 1))
+            }
+        }
+
+        highlights
+    }
+
+    /// Add new inline annotations.
+    ///
+    /// The annotations grapheme will be rendered with `highlight`
+    /// patched on top of `ui.text`.
+    ///
+    /// The annotations **must be sorted** by their `char_idx`.
+    /// Multiple annotations with the same `char_idx` are allowed,
+    /// they will be display in the order that they are present in the layer.
+    ///
+    /// If multiple layers contain annotations at the same position
+    /// the annotations that belong to the layers added first will be shown first.
+    pub fn add_inline_annotations(
+        &mut self,
+        layer: Rc<[InlineAnnotation]>,
+        highlight: Option<Highlight>,
+    ) -> &mut Self {
+        self.inline_annotations.push((layer, highlight).into());
+        self
+    }
+
+    /// Add new grapheme overlays.
+    ///
+    /// The overlayed grapheme will be rendered with `highlight`
+    /// patched on top of `ui.text`.
+    ///
+    /// The overlays **must be sorted** by their `char_idx`.
+    /// Multiple overlays with the same `char_idx` **are allowed**.
+    ///
+    /// If multiple layers contain overlay at the same position
+    /// the overlay from the layer added last will be show.
+    pub fn add_overlay(&mut self, layer: Rc<[Overlay]>, highlight: Option<Highlight>) -> &mut Self {
+        self.overlays.push((layer, highlight).into());
+        self
+    }
+
+    /// Add new annotation lines.
+    ///
+    /// The line annotations **must be sorted** by their `char_idx`.
+    /// Multiple line annotations with the same `char_idx` **are not allowed**.
+    pub fn add_line_annotation(&mut self, layer: Rc<[LineAnnotation]>) -> &mut Self {
+        self.line_annotations.push((layer, ()).into());
+        self
+    }
+
+    /// Removes all line annotations, useful for vertical motions
+    /// so that virtual text lines are automatically skipped.
+    pub fn clear_line_annotations(&mut self) {
+        self.line_annotations.clear();
+    }
+
+    pub(crate) fn next_inline_annotation_at(
+        &self,
+        char_idx: usize,
+    ) -> Option<(&InlineAnnotation, Option<Highlight>)> {
+        self.inline_annotations.iter().find_map(|layer| {
+            let annotation = layer.consume(char_idx, |annot| annot.char_idx)?;
+            Some((annotation, layer.metadata))
+        })
+    }
+
+    pub(crate) fn overlay_at(&self, char_idx: usize) -> Option<(&Overlay, Option<Highlight>)> {
+        let mut overlay = None;
+        for layer in &self.overlays {
+            while let Some(new_overlay) = layer.consume(char_idx, |annot| annot.char_idx) {
+                overlay = Some((new_overlay, layer.metadata));
+            }
+        }
+        overlay
+    }
+
+    pub(crate) fn annotation_lines_at(&self, char_idx: usize) -> usize {
+        self.line_annotations
+            .iter()
+            .map(|layer| {
+                let mut lines = 0;
+                while let Some(annot) = layer.annotations.get(layer.current_index.get()) {
+                    if annot.anchor_char_idx == char_idx {
+                        layer.current_index.set(layer.current_index.get() + 1);
+                        lines += annot.height
+                    } else {
+                        break;
+                    }
+                }
+                lines
+            })
+            .sum()
+    }
+}

helix-term/src/application.rs

@@ -172,7 +172,7 @@ impl Application {
             area,
             theme_loader.clone(),
             syn_loader.clone(),
-            Box::new(Map::new(Arc::clone(&config), |config: &Config| {
+            Arc::new(Map::new(Arc::clone(&config), |config: &Config| {
                 &config.editor
             })),
         );
@@ -309,8 +309,10 @@ impl Application {
         let surface = self.terminal.current_buffer_mut();
 
         self.compositor.render(area, surface, &mut cx);
-
         let (pos, kind) = self.compositor.cursor(area, &self.editor);
+        // reset cursor cache
+        self.editor.cursor_cache.set(None);
+
         let pos = pos.map(|pos| (pos.col as u16, pos.row as u16));
         self.terminal.draw(pos, kind).unwrap();
     }
@@ -395,6 +397,13 @@ impl Application {
         // Update all the relevant members in the editor after updating
         // the configuration.
         self.editor.refresh_config();
+
+        // reset view position in case softwrap was enabled/disabled
+        let scrolloff = self.editor.config().scrolloff;
+        for (view, _) in self.editor.tree.views_mut() {
+            let doc = &self.editor.documents[&view.doc];
+            view.ensure_cursor_in_view(doc, scrolloff)
+        }
     }
 
     /// refresh language config after config change

helix-term/src/commands.rs

@@ -9,21 +9,25 @@ use tui::widgets::Row;
 pub use typed::*;
 
 use helix_core::{
-    comment, coords_at_pos, encoding, find_first_non_whitespace_char, find_root, graphemes,
+    char_idx_at_visual_offset, comment,
+    doc_formatter::TextFormat,
+    encoding, find_first_non_whitespace_char, find_root, graphemes,
     history::UndoKind,
     increment, indent,
     indent::IndentStyle,
     line_ending::{get_line_ending_of_str, line_end_char_index, str_is_line_ending},
     match_brackets,
-    movement::{self, Direction},
-    object, pos_at_coords, pos_at_visual_coords,
+    movement::{self, move_vertically_visual, Direction},
+    object, pos_at_coords,
     regex::{self, Regex, RegexBuilder},
     search::{self, CharMatcher},
-    selection, shellwords, surround, textobject,
+    selection, shellwords, surround,
+    text_annotations::TextAnnotations,
+    textobject,
     tree_sitter::Node,
     unicode::width::UnicodeWidthChar,
-    visual_coords_at_pos, LineEnding, Position, Range, Rope, RopeGraphemes, RopeSlice, Selection,
-    SmallVec, Tendril, Transaction,
+    visual_offset_from_block, LineEnding, Position, Range, Rope, RopeGraphemes, RopeSlice,
+    Selection, SmallVec, Tendril, Transaction,
 };
 use helix_view::{
     clipboard::ClipboardType,
@@ -200,10 +204,14 @@ impl MappableCommand {
         move_char_right, "Move right",
         move_line_up, "Move up",
         move_line_down, "Move down",
+        move_visual_line_up, "Move up",
+        move_visual_line_down, "Move down",
         extend_char_left, "Extend left",
         extend_char_right, "Extend right",
         extend_line_up, "Extend up",
         extend_line_down, "Extend down",
+        extend_visual_line_up, "Extend up",
+        extend_visual_line_down, "Extend down",
         copy_selection_on_next_line, "Copy selection on next line",
         copy_selection_on_prev_line, "Copy selection on previous line",
         move_next_word_start, "Move to start of next word",
@@ -538,18 +546,27 @@ impl PartialEq for MappableCommand {
 
 fn no_op(_cx: &mut Context) {}
 
-fn move_impl<F>(cx: &mut Context, move_fn: F, dir: Direction, behaviour: Movement)
-where
-    F: Fn(RopeSlice, Range, Direction, usize, Movement, usize) -> Range,
-{
+type MoveFn =
+    fn(RopeSlice, Range, Direction, usize, Movement, &TextFormat, &mut TextAnnotations) -> Range;
+
+fn move_impl(cx: &mut Context, move_fn: MoveFn, dir: Direction, behaviour: Movement) {
     let count = cx.count();
     let (view, doc) = current!(cx.editor);
     let text = doc.text().slice(..);
+    let text_fmt = doc.text_format(view.inner_area(doc).width, None);
+    let mut annotations = view.text_annotations(doc, None);
 
-    let selection = doc
-        .selection(view.id)
-        .clone()
-        .transform(|range| move_fn(text, range, dir, count, behaviour, doc.tab_width()));
+    let selection = doc.selection(view.id).clone().transform(|range| {
+        move_fn(
+            text,
+            range,
+            dir,
+            count,
+            behaviour,
+            &text_fmt,
+            &mut annotations,
+        )
+    });
     doc.set_selection(view.id, selection);
 }
 
@@ -571,6 +588,24 @@ fn move_line_down(cx: &mut Context) {
     move_impl(cx, move_vertically, Direction::Forward, Movement::Move)
 }
 
+fn move_visual_line_up(cx: &mut Context) {
+    move_impl(
+        cx,
+        move_vertically_visual,
+        Direction::Backward,
+        Movement::Move,
+    )
+}
+
+fn move_visual_line_down(cx: &mut Context) {
+    move_impl(
+        cx,
+        move_vertically_visual,
+        Direction::Forward,
+        Movement::Move,
+    )
+}
+
 fn extend_char_left(cx: &mut Context) {
     move_impl(cx, move_horizontally, Direction::Backward, Movement::Extend)
 }
@@ -587,6 +622,24 @@ fn extend_line_down(cx: &mut Context) {
     move_impl(cx, move_vertically, Direction::Forward, Movement::Extend)
 }
 
+fn extend_visual_line_up(cx: &mut Context) {
+    move_impl(
+        cx,
+        move_vertically_visual,
+        Direction::Backward,
+        Movement::Extend,
+    )
+}
+
+fn extend_visual_line_down(cx: &mut Context) {
+    move_impl(
+        cx,
+        move_vertically_visual,
+        Direction::Forward,
+        Movement::Extend,
+    )
+}
+
 fn goto_line_end_impl(view: &mut View, doc: &mut Document, movement: Movement) {
     let text = doc.text().slice(..);
 
@@ -814,7 +867,10 @@ fn trim_selections(cx: &mut Context) {
 }
 
 // align text in selection
+#[allow(deprecated)]
 fn align_selections(cx: &mut Context) {
+    use helix_core::visual_coords_at_pos;
+
     let (view, doc) = current!(cx.editor);
     let text = doc.text().slice(..);
     let selection = doc.selection(view.id);
@@ -891,17 +947,22 @@ fn goto_window(cx: &mut Context, align: Align) {
     // as we type
     let scrolloff = config.scrolloff.min(height.saturating_sub(1) / 2);
 
-    let last_line = view.last_line(doc);
+    let last_visual_line = view.last_visual_line(doc);
 
-    let line = match align {
-        Align::Top => view.offset.row + scrolloff + count,
-        Align::Center => view.offset.row + ((last_line - view.offset.row) / 2),
-        Align::Bottom => last_line.saturating_sub(scrolloff + count),
+    let visual_line = match align {
+        Align::Top => view.offset.vertical_offset + scrolloff + count,
+        Align::Center => view.offset.vertical_offset + (last_visual_line / 2),
+        Align::Bottom => {
+            view.offset.vertical_offset + last_visual_line.saturating_sub(scrolloff + count)
+        }
     }
-    .max(view.offset.row + scrolloff)
-    .min(last_line.saturating_sub(scrolloff));
+    .max(view.offset.vertical_offset + scrolloff)
+    .min(view.offset.vertical_offset + last_visual_line.saturating_sub(scrolloff));
+
+    let pos = view
+        .pos_at_visual_coords(doc, visual_line as u16, 0, false)
+        .expect("visual_line was constrained to the view area");
 
-    let pos = doc.text().line_to_char(line);
     let text = doc.text().slice(..);
     let selection = doc
         .selection(view.id)
@@ -1385,53 +1446,72 @@ pub fn scroll(cx: &mut Context, offset: usize, direction: Direction) {
     let range = doc.selection(view.id).primary();
     let text = doc.text().slice(..);
 
-    let cursor = visual_coords_at_pos(text, range.cursor(text), doc.tab_width());
-    let doc_last_line = doc.text().len_lines().saturating_sub(1);
-
-    let last_line = view.last_line(doc);
-
-    if direction == Backward && view.offset.row == 0
-        || direction == Forward && last_line == doc_last_line
-    {
-        return;
-    }
-
+    let cursor = range.cursor(text);
     let height = view.inner_height();
 
     let scrolloff = config.scrolloff.min(height / 2);
+    let offset = match direction {
+        Forward => offset as isize,
+        Backward => -(offset as isize),
+    };
 
-    view.offset.row = match direction {
-        Forward => view.offset.row + offset,
-        Backward => view.offset.row.saturating_sub(offset),
+    let doc_text = doc.text().slice(..);
+    let viewport = view.inner_area(doc);
+    let text_fmt = doc.text_format(viewport.width, None);
+    let annotations = view.text_annotations(doc, None);
+    (view.offset.anchor, view.offset.vertical_offset) = char_idx_at_visual_offset(
+        doc_text,
+        view.offset.anchor,
+        view.offset.vertical_offset as isize + offset,
+        0,
+        &text_fmt,
+        &annotations,
+    );
+
+    let head;
+    match direction {
+        Forward => {
+            head = char_idx_at_visual_offset(
+                doc_text,
+                view.offset.anchor,
+                (view.offset.vertical_offset + scrolloff) as isize,
+                0,
+                &text_fmt,
+                &annotations,
+            )
+            .0;
+            if head <= cursor {
+                return;
+            }
+        }
+        Backward => {
+            head = char_idx_at_visual_offset(
+                doc_text,
+                view.offset.anchor,
+                (view.offset.vertical_offset + height - scrolloff) as isize,
+                0,
+                &text_fmt,
+                &annotations,
+            )
+            .0;
+            if head >= cursor {
+                return;
+            }
+        }
     }
-    .min(doc_last_line);
 
-    // recalculate last line
-    let last_line = view.last_line(doc);
+    let anchor = if cx.editor.mode == Mode::Select {
+        range.anchor
+    } else {
+        head
+    };
 
-    // clamp into viewport
-    let line = cursor
-        .row
-        .max(view.offset.row + scrolloff)
-        .min(last_line.saturating_sub(scrolloff));
-
-    // If cursor needs moving, replace primary selection
-    if line != cursor.row {
-        let head = pos_at_visual_coords(text, Position::new(line, cursor.col), doc.tab_width()); // this func will properly truncate to line end
-
-        let anchor = if cx.editor.mode == Mode::Select {
-            range.anchor
-        } else {
-            head
-        };
-
-        // replace primary selection with an empty selection at cursor pos
-        let prim_sel = Range::new(anchor, head);
-        let mut sel = doc.selection(view.id).clone();
-        let idx = sel.primary_index();
-        sel = sel.replace(idx, prim_sel);
-        doc.set_selection(view.id, sel);
-    }
+    // replace primary selection with an empty selection at cursor pos
+    let prim_sel = Range::new(anchor, head);
+    let mut sel = doc.selection(view.id).clone();
+    let idx = sel.primary_index();
+    sel = sel.replace(idx, prim_sel);
+    doc.set_selection(view.id, sel);
 }
 
 fn page_up(cx: &mut Context) {
@@ -1458,7 +1538,15 @@ fn half_page_down(cx: &mut Context) {
     scroll(cx, offset, Direction::Forward);
 }
 
+#[allow(deprecated)]
+// currently uses the deprected `visual_coords_at_pos`/`pos_at_visual_coords` functions
+// as this function ignores softwrapping (and virtual text) and instead only cares
+// about "text visual position"
+//
+// TODO: implement a variant of that uses visual lines and respects virtual text
 fn copy_selection_on_line(cx: &mut Context, direction: Direction) {
+    use helix_core::{pos_at_visual_coords, visual_coords_at_pos};
+
     let count = cx.count();
     let (view, doc) = current!(cx.editor);
     let text = doc.text().slice(..);
@@ -4475,11 +4563,19 @@ fn align_view_bottom(cx: &mut Context) {
 
 fn align_view_middle(cx: &mut Context) {
     let (view, doc) = current!(cx.editor);
-    let text = doc.text().slice(..);
-    let pos = doc.selection(view.id).primary().cursor(text);
-    let pos = coords_at_pos(text, pos);
+    let inner_width = view.inner_width(doc);
+    let text_fmt = doc.text_format(inner_width, None);
+    // there is no horizontal position when softwrap is enabled
+    if text_fmt.soft_wrap {
+        return;
+    }
+    let doc_text = doc.text().slice(..);
+    let annotations = view.text_annotations(doc, None);
+    let pos = doc.selection(view.id).primary().cursor(doc_text);
+    let pos =
+        visual_offset_from_block(doc_text, view.offset.anchor, pos, &text_fmt, &annotations).0;
 
-    view.offset.col = pos
+    view.offset.horizontal_offset = pos
         .col
         .saturating_sub((view.inner_area(doc).width as usize) / 2);
 }

helix-term/src/keymap/default.rs

@@ -7,8 +7,8 @@ use helix_core::hashmap;
 pub fn default() -> HashMap<Mode, Keymap> {
     let normal = keymap!({ "Normal mode"
         "h" | "left" => move_char_left,
-        "j" | "down" => move_line_down,
-        "k" | "up" => move_line_up,
+        "j" | "down" => move_visual_line_down,
+        "k" | "up" => move_visual_line_up,
         "l" | "right" => move_char_right,
 
         "t" => find_till_char,
@@ -55,6 +55,8 @@ pub fn default() -> HashMap<Mode, Keymap> {
             "m" => goto_last_modified_file,
             "n" => goto_next_buffer,
             "p" => goto_previous_buffer,
+            "k" => move_line_up,
+            "j" => move_line_down,
             "." => goto_last_modification,
         },
         ":" => command_mode,
@@ -321,8 +323,8 @@ pub fn default() -> HashMap<Mode, Keymap> {
     let mut select = normal.clone();
     select.merge_nodes(keymap!({ "Select mode"
         "h" | "left" => extend_char_left,
-        "j" | "down" => extend_line_down,
-        "k" | "up" => extend_line_up,
+        "j" | "down" => extend_visual_line_down,
+        "k" | "up" => extend_visual_line_up,
         "l" | "right" => extend_char_right,
 
         "w" => extend_next_word_start,
@@ -345,6 +347,10 @@ pub fn default() -> HashMap<Mode, Keymap> {
         "esc" => exit_select_mode,
 
         "v" => normal_mode,
+        "g" => { "Goto"
+            "k" => extend_line_up,
+            "j" => extend_line_down,
+        },
     }));
     let insert = keymap!({ "Insert mode"
         "esc" => normal_mode,
@@ -362,8 +368,8 @@ pub fn default() -> HashMap<Mode, Keymap> {
         "C-j" | "ret" => insert_newline,
         "tab" => insert_tab,
 
-        "up" => move_line_up,
-        "down" => move_line_down,
+        "up" => move_visual_line_up,
+        "down" => move_visual_line_down,
         "left" => move_char_left,
         "right" => move_char_right,
         "pageup" => page_up,

helix-term/src/ui/completion.rs

@@ -391,8 +391,10 @@ impl Component for Completion {
             let language = doc.language_name().unwrap_or("");
             let text = doc.text().slice(..);
             let cursor_pos = doc.selection(view.id).primary().cursor(text);
-            let coords = helix_core::visual_coords_at_pos(text, cursor_pos, doc.tab_width());
-            let cursor_pos = (coords.row - view.offset.row) as u16;
+            let coords = view
+                .screen_coords_at_pos(doc, text, cursor_pos)
+                .expect("cursor must be in view");
+            let cursor_pos = coords.row as u16;
 
             let mut markdown_doc = match &option.documentation {
                 Some(lsp::Documentation::String(contents))

helix-term/src/ui/document.rs

@@ -0,0 +1,475 @@
+use std::cmp::min;
+
+use helix_core::doc_formatter::{DocumentFormatter, GraphemeSource, TextFormat};
+use helix_core::graphemes::Grapheme;
+use helix_core::str_utils::char_to_byte_idx;
+use helix_core::syntax::Highlight;
+use helix_core::syntax::HighlightEvent;
+use helix_core::text_annotations::TextAnnotations;
+use helix_core::{visual_offset_from_block, Position, RopeSlice};
+use helix_view::editor::{WhitespaceConfig, WhitespaceRenderValue};
+use helix_view::graphics::Rect;
+use helix_view::theme::Style;
+use helix_view::view::ViewPosition;
+use helix_view::Document;
+use helix_view::Theme;
+use tui::buffer::Buffer as Surface;
+
+pub trait LineDecoration {
+    fn render_background(&mut self, _renderer: &mut TextRenderer, _pos: LinePos) {}
+    fn render_foreground(
+        &mut self,
+        _renderer: &mut TextRenderer,
+        _pos: LinePos,
+        _end_char_idx: usize,
+    ) {
+    }
+}
+
+impl<F: FnMut(&mut TextRenderer, LinePos)> LineDecoration for F {
+    fn render_background(&mut self, renderer: &mut TextRenderer, pos: LinePos) {
+        self(renderer, pos)
+    }
+}
+
+/// A wrapper around a HighlightIterator
+/// that merges the layered highlights to create the final text style
+/// and yields the active text style and the char_idx where the active
+/// style will have to be recomputed.
+struct StyleIter<'a, H: Iterator<Item = HighlightEvent>> {
+    text_style: Style,
+    active_highlights: Vec<Highlight>,
+    highlight_iter: H,
+    theme: &'a Theme,
+}
+
+impl<H: Iterator<Item = HighlightEvent>> Iterator for StyleIter<'_, H> {
+    type Item = (Style, usize);
+    fn next(&mut self) -> Option<(Style, usize)> {
+        while let Some(event) = self.highlight_iter.next() {
+            match event {
+                HighlightEvent::HighlightStart(highlights) => {
+                    self.active_highlights.push(highlights)
+                }
+                HighlightEvent::HighlightEnd => {
+                    self.active_highlights.pop();
+                }
+                HighlightEvent::Source { start, end } => {
+                    if start == end {
+                        continue;
+                    }
+                    let style = self
+                        .active_highlights
+                        .iter()
+                        .fold(self.text_style, |acc, span| {
+                            acc.patch(self.theme.highlight(span.0))
+                        });
+                    return Some((style, end));
+                }
+            }
+        }
+        None
+    }
+}
+
+#[derive(Debug, PartialEq, Eq, Copy, Clone)]
+pub struct LinePos {
+    /// Indicates whether the given visual line
+    /// is the first visual line of the given document line
+    pub first_visual_line: bool,
+    /// The line index of the document line that contains the given visual line
+    pub doc_line: usize,
+    /// Vertical offset from the top of the inner view area
+    pub visual_line: u16,
+    /// The first char index of this visual line.
+    /// Note that if the visual line is entirely filled by
+    /// a very long inline virtual text then this index will point
+    /// at the next (non-virtual) char after this visual line
+    pub start_char_idx: usize,
+}
+
+pub type TranslatedPosition<'a> = (usize, Box<dyn FnMut(&mut TextRenderer, Position) + 'a>);
+
+#[allow(clippy::too_many_arguments)]
+pub fn render_document(
+    surface: &mut Surface,
+    viewport: Rect,
+    doc: &Document,
+    offset: ViewPosition,
+    doc_annotations: &TextAnnotations,
+    highlight_iter: impl Iterator<Item = HighlightEvent>,
+    theme: &Theme,
+    line_decoration: &mut [Box<dyn LineDecoration + '_>],
+    translated_positions: &mut [TranslatedPosition],
+) {
+    let mut renderer = TextRenderer::new(surface, doc, theme, offset.horizontal_offset, viewport);
+    render_text(
+        &mut renderer,
+        doc.text().slice(..),
+        offset,
+        &doc.text_format(viewport.width, Some(theme)),
+        doc_annotations,
+        highlight_iter,
+        theme,
+        line_decoration,
+        translated_positions,
+    )
+}
+
+fn translate_positions(
+    char_pos: usize,
+    first_visisble_char_idx: usize,
+    translated_positions: &mut [TranslatedPosition],
+    text_fmt: &TextFormat,
+    renderer: &mut TextRenderer,
+    pos: Position,
+) {
+    // check if any positions translated on the fly (like cursor) has been reached
+    for (char_idx, callback) in &mut *translated_positions {
+        if *char_idx < char_pos && *char_idx >= first_visisble_char_idx {
+            // by replacing the char_index with usize::MAX large number we ensure
+            // that the same position is only translated once
+            // text will never reach usize::MAX as rust memory allocations are limited
+            // to isize::MAX
+            *char_idx = usize::MAX;
+
+            if text_fmt.soft_wrap {
+                callback(renderer, pos)
+            } else if pos.col >= renderer.col_offset
+                && pos.col - renderer.col_offset < renderer.viewport.width as usize
+            {
+                callback(
+                    renderer,
+                    Position {
+                        row: pos.row,
+                        col: pos.col - renderer.col_offset,
+                    },
+                )
+            }
+        }
+    }
+}
+
+#[allow(clippy::too_many_arguments)]
+pub fn render_text<'t>(
+    renderer: &mut TextRenderer,
+    text: RopeSlice<'t>,
+    offset: ViewPosition,
+    text_fmt: &TextFormat,
+    text_annotations: &TextAnnotations,
+    highlight_iter: impl Iterator<Item = HighlightEvent>,
+    theme: &Theme,
+    line_decorations: &mut [Box<dyn LineDecoration + '_>],
+    translated_positions: &mut [TranslatedPosition],
+) {
+    let (
+        Position {
+            row: mut row_off, ..
+        },
+        mut char_pos,
+    ) = visual_offset_from_block(
+        text,
+        offset.anchor,
+        offset.anchor,
+        text_fmt,
+        text_annotations,
+    );
+    row_off += offset.vertical_offset;
+    assert_eq!(0, offset.vertical_offset);
+
+    let (mut formatter, mut first_visible_char_idx) =
+        DocumentFormatter::new_at_prev_checkpoint(text, text_fmt, text_annotations, offset.anchor);
+    let mut styles = StyleIter {
+        text_style: renderer.text_style,
+        active_highlights: Vec::with_capacity(64),
+        highlight_iter,
+        theme,
+    };
+
+    let mut last_line_pos = LinePos {
+        first_visual_line: false,
+        doc_line: usize::MAX,
+        visual_line: u16::MAX,
+        start_char_idx: usize::MAX,
+    };
+    let mut is_in_indent_area = true;
+    let mut last_line_indent_level = 0;
+    let mut style_span = styles
+        .next()
+        .unwrap_or_else(|| (Style::default(), usize::MAX));
+
+    loop {
+        // formattter.line_pos returns to line index of the next grapheme
+        // so it must be called before formatter.next
+        let doc_line = formatter.line_pos();
+        // TODO refactor with let .. else once MSRV reaches 1.65
+        let (grapheme, mut pos) = if let Some(it) = formatter.next() {
+            it
+        } else {
+            let mut last_pos = formatter.visual_pos();
+            last_pos.col -= 1;
+            // check if any positions translated on the fly (like cursor) are at the EOF
+            translate_positions(
+                char_pos + 1,
+                first_visible_char_idx,
+                translated_positions,
+                text_fmt,
+                renderer,
+                last_pos,
+            );
+            break;
+        };
+
+        // skip any graphemes on visual lines before the block start
+        if pos.row < row_off {
+            if char_pos >= style_span.1 {
+                // TODO refactor using let..else once MSRV reaches 1.65
+                style_span = if let Some(style_span) = styles.next() {
+                    style_span
+                } else {
+                    break;
+                }
+            }
+            char_pos += grapheme.doc_chars();
+            first_visible_char_idx = char_pos + 1;
+            continue;
+        }
+        pos.row -= row_off;
+
+        // if the end of the viewport is reached stop rendering
+        if pos.row as u16 >= renderer.viewport.height {
+            break;
+        }
+
+        // apply decorations before rendering a new line
+        if pos.row as u16 != last_line_pos.visual_line {
+            if pos.row > 0 {
+                renderer.draw_indent_guides(last_line_indent_level, last_line_pos.visual_line);
+                is_in_indent_area = true;
+                for line_decoration in &mut *line_decorations {
+                    line_decoration.render_foreground(renderer, last_line_pos, char_pos);
+                }
+            }
+            last_line_pos = LinePos {
+                first_visual_line: doc_line != last_line_pos.doc_line,
+                doc_line,
+                visual_line: pos.row as u16,
+                start_char_idx: char_pos,
+            };
+            for line_decoration in &mut *line_decorations {
+                line_decoration.render_background(renderer, last_line_pos);
+            }
+        }
+
+        // aquire the correct grapheme style
+        if char_pos >= style_span.1 {
+            // TODO refactor using let..else once MSRV reaches 1.65
+            style_span = if let Some(style_span) = styles.next() {
+                style_span
+            } else {
+                (Style::default(), usize::MAX)
+            }
+        }
+        char_pos += grapheme.doc_chars();
+
+        // check if any positions translated on the fly (like cursor) has been reached
+        translate_positions(
+            char_pos,
+            first_visible_char_idx,
+            translated_positions,
+            text_fmt,
+            renderer,
+            pos,
+        );
+
+        let grapheme_style = if let GraphemeSource::VirtualText { highlight } = grapheme.source {
+            let style = renderer.text_style;
+            if let Some(highlight) = highlight {
+                style.patch(theme.highlight(highlight.0))
+            } else {
+                style
+            }
+        } else {
+            style_span.0
+        };
+
+        renderer.draw_grapheme(
+            grapheme.grapheme,
+            grapheme_style,
+            &mut last_line_indent_level,
+            &mut is_in_indent_area,
+            pos,
+        );
+    }
+
+    renderer.draw_indent_guides(last_line_indent_level, last_line_pos.visual_line);
+    for line_decoration in &mut *line_decorations {
+        line_decoration.render_foreground(renderer, last_line_pos, char_pos);
+    }
+}
+
+#[derive(Debug)]
+pub struct TextRenderer<'a> {
+    pub surface: &'a mut Surface,
+    pub text_style: Style,
+    pub whitespace_style: Style,
+    pub indent_guide_char: String,
+    pub indent_guide_style: Style,
+    pub newline: String,
+    pub nbsp: String,
+    pub space: String,
+    pub tab: String,
+    pub tab_width: u16,
+    pub starting_indent: usize,
+    pub draw_indent_guides: bool,
+    pub col_offset: usize,
+    pub viewport: Rect,
+}
+
+impl<'a> TextRenderer<'a> {
+    pub fn new(
+        surface: &'a mut Surface,
+        doc: &Document,
+        theme: &Theme,
+        col_offset: usize,
+        viewport: Rect,
+    ) -> TextRenderer<'a> {
+        let editor_config = doc.config.load();
+        let WhitespaceConfig {
+            render: ws_render,
+            characters: ws_chars,
+        } = &editor_config.whitespace;
+
+        let tab_width = doc.tab_width();
+        let tab = if ws_render.tab() == WhitespaceRenderValue::All {
+            std::iter::once(ws_chars.tab)
+                .chain(std::iter::repeat(ws_chars.tabpad).take(tab_width - 1))
+                .collect()
+        } else {
+            " ".repeat(tab_width)
+        };
+        let newline = if ws_render.newline() == WhitespaceRenderValue::All {
+            ws_chars.newline.into()
+        } else {
+            " ".to_owned()
+        };
+
+        let space = if ws_render.space() == WhitespaceRenderValue::All {
+            ws_chars.space.into()
+        } else {
+            " ".to_owned()
+        };
+        let nbsp = if ws_render.nbsp() == WhitespaceRenderValue::All {
+            ws_chars.nbsp.into()
+        } else {
+            " ".to_owned()
+        };
+
+        let text_style = theme.get("ui.text");
+
+        TextRenderer {
+            surface,
+            indent_guide_char: editor_config.indent_guides.character.into(),
+            newline,
+            nbsp,
+            space,
+            tab_width: tab_width as u16,
+            tab,
+            whitespace_style: theme.get("ui.virtual.whitespace"),
+            starting_indent: (col_offset / tab_width)
+                + editor_config.indent_guides.skip_levels as usize,
+            indent_guide_style: text_style.patch(
+                theme
+                    .try_get("ui.virtual.indent-guide")
+                    .unwrap_or_else(|| theme.get("ui.virtual.whitespace")),
+            ),
+            text_style,
+            draw_indent_guides: editor_config.indent_guides.render,
+            viewport,
+            col_offset,
+        }
+    }
+
+    /// Draws a single `grapheme` at the current render position with a specified `style`.
+    pub fn draw_grapheme(
+        &mut self,
+        grapheme: Grapheme,
+        mut style: Style,
+        last_indent_level: &mut usize,
+        is_in_indent_area: &mut bool,
+        position: Position,
+    ) {
+        let cut_off_start = self.col_offset.saturating_sub(position.col as usize);
+        let is_whitespace = grapheme.is_whitespace();
+
+        // TODO is it correct to apply the whitspace style to all unicode white spaces?
+        if is_whitespace {
+            style = style.patch(self.whitespace_style);
+        }
+
+        let width = grapheme.width();
+        let grapheme = match grapheme {
+            Grapheme::Tab { width } => {
+                let grapheme_tab_width = char_to_byte_idx(&self.tab, width as usize);
+                &self.tab[..grapheme_tab_width]
+            }
+            // TODO special rendering for other whitespaces?
+            Grapheme::Other { ref g } if g == " " => &self.space,
+            Grapheme::Other { ref g } if g == "\u{00A0}" => &self.nbsp,
+            Grapheme::Other { ref g } => &*g,
+            Grapheme::Newline => &self.newline,
+        };
+
+        let in_bounds = self.col_offset <= (position.col as usize)
+            && (position.col as usize) < self.viewport.width as usize + self.col_offset;
+
+        if in_bounds {
+            self.surface.set_string(
+                self.viewport.x + (position.col - self.col_offset) as u16,
+                self.viewport.y + position.row as u16,
+                grapheme,
+                style,
+            );
+        } else if cut_off_start != 0 && cut_off_start < width as usize {
+            // partially on screen
+            let rect = Rect::new(
+                self.viewport.x as u16,
+                self.viewport.y + position.row as u16,
+                (width - cut_off_start) as u16,
+                1,
+            );
+            self.surface.set_style(rect, style);
+        }
+
+        if *is_in_indent_area && !is_whitespace {
+            *last_indent_level = position.col;
+            *is_in_indent_area = false;
+        }
+    }
+
+    /// Overlay indentation guides ontop of a rendered line
+    /// The indentation level is computed in `draw_lines`.
+    /// Therefore this function must always be called afterwards.
+    pub fn draw_indent_guides(&mut self, indent_level: usize, row: u16) {
+        if !self.draw_indent_guides {
+            return;
+        }
+
+        // Don't draw indent guides outside of view
+        let end_indent = min(
+            indent_level,
+            // Add tab_width - 1 to round up, since the first visible
+            // indent might be a bit after offset.col
+            self.col_offset + self.viewport.width as usize + (self.tab_width - 1) as usize,
+        ) / self.tab_width as usize;
+
+        for i in self.starting_indent..end_indent {
+            let x =
+                (self.viewport.x as usize + (i * self.tab_width as usize) - self.col_offset) as u16;
+            let y = self.viewport.y + row;
+            debug_assert!(self.surface.in_bounds(x, y));
+            self.surface
+                .set_string(x, y, &self.indent_guide_char, self.indent_guide_style);
+        }
+    }
+}

helix-term/src/ui/editor.rs

@@ -4,7 +4,10 @@ use crate::{
     job::{self, Callback},
     key,
     keymap::{KeymapResult, Keymaps},
-    ui::{Completion, ProgressSpinners},
+    ui::{
+        document::{render_document, LinePos, TextRenderer, TranslatedPosition},
+        Completion, ProgressSpinners,
+    },
 };
 
 use helix_core::{
@@ -13,8 +16,9 @@ use helix_core::{
     },
     movement::Direction,
     syntax::{self, HighlightEvent},
+    text_annotations::TextAnnotations,
     unicode::width::UnicodeWidthStr,
-    visual_coords_at_pos, LineEnding, Position, Range, Selection, Transaction,
+    visual_offset_from_block, Position, Range, Selection, Transaction,
 };
 use helix_view::{
     document::{Mode, SCRATCH_BUFFER_NAME},
@@ -24,12 +28,12 @@ use helix_view::{
     keyboard::{KeyCode, KeyModifiers},
     Document, Editor, Theme, View,
 };
-use std::{borrow::Cow, cmp::min, num::NonZeroUsize, path::PathBuf};
+use std::{num::NonZeroUsize, path::PathBuf, rc::Rc};
 
 use tui::buffer::Buffer as Surface;
 
-use super::lsp::SignatureHelp;
 use super::statusline;
+use super::{document::LineDecoration, lsp::SignatureHelp};
 
 pub struct EditorView {
     pub keymaps: Keymaps,
@@ -83,6 +87,10 @@ impl EditorView {
         let theme = &editor.theme;
         let config = editor.config();
 
+        let text_annotations = view.text_annotations(doc, Some(theme));
+        let mut line_decorations: Vec<Box<dyn LineDecoration>> = Vec::new();
+        let mut translated_positions: Vec<TranslatedPosition> = Vec::new();
+
         // DAP: Highlight current stack frame position
         let stack_frame = editor.debugger.as_ref().and_then(|debugger| {
             if let (Some(frame), Some(thread_id)) = (debugger.active_frame, debugger.thread_id) {
@@ -103,28 +111,40 @@ impl EditorView {
                     == doc.path()
             {
                 let line = frame.line - 1; // convert to 0-indexing
-                if line >= view.offset.row && line < view.offset.row + area.height as usize {
-                    surface.set_style(
-                        Rect::new(
-                            area.x,
-                            area.y + (line - view.offset.row) as u16,
-                            area.width,
-                            1,
-                        ),
-                        theme.get("ui.highlight"),
-                    );
-                }
+                let style = theme.get("ui.highlight");
+                let line_decoration = move |renderer: &mut TextRenderer, pos: LinePos| {
+                    if pos.doc_line != line {
+                        return;
+                    }
+                    renderer
+                        .surface
+                        .set_style(Rect::new(area.x, pos.visual_line, area.width, 1), style);
+                };
+
+                line_decorations.push(Box::new(line_decoration));
             }
         }
 
         if is_focused && config.cursorline {
-            Self::highlight_cursorline(doc, view, surface, theme);
-        }
-        if is_focused && config.cursorcolumn {
-            Self::highlight_cursorcolumn(doc, view, surface, theme);
+            line_decorations.push(Self::cursorline_decorator(doc, view, theme))
+        }
+
+        if is_focused && config.cursorcolumn {
+            Self::highlight_cursorcolumn(doc, view, surface, theme, inner, &text_annotations);
+        }
+
+        let mut highlights =
+            Self::doc_syntax_highlights(doc, view.offset.anchor, inner.height, theme);
+        let overlay_highlights = Self::overlay_syntax_highlights(
+            doc,
+            view.offset.anchor,
+            inner.height,
+            &text_annotations,
+        );
+        if !overlay_highlights.is_empty() {
+            highlights = Box::new(syntax::merge(highlights, overlay_highlights));
         }
 
-        let mut highlights = Self::doc_syntax_highlights(doc, view.offset, inner.height, theme);
         for diagnostic in Self::doc_diagnostics_highlights(doc, theme) {
             // Most of the `diagnostic` Vecs are empty most of the time. Skipping
             // a merge for any empty Vec saves a significant amount of work.
@@ -133,8 +153,9 @@ impl EditorView {
             }
             highlights = Box::new(syntax::merge(highlights, diagnostic));
         }
+
         let highlights: Box<dyn Iterator<Item = HighlightEvent>> = if is_focused {
-            Box::new(syntax::merge(
+            let highlights = syntax::merge(
                 highlights,
                 Self::doc_selection_highlights(
                     editor.mode(),
@@ -143,19 +164,52 @@ impl EditorView {
                     theme,
                     &config.cursor_shape,
                 ),
-            ))
+            );
+            let focused_view_elements = Self::highlight_focused_view_elements(view, doc, theme);
+            if focused_view_elements.is_empty() {
+                Box::new(highlights)
+            } else {
+                Box::new(syntax::merge(highlights, focused_view_elements))
+            }
         } else {
             Box::new(highlights)
         };
 
-        Self::render_text_highlights(doc, view.offset, inner, surface, theme, highlights, &config);
-        Self::render_gutter(editor, doc, view, view.area, surface, theme, is_focused);
-        Self::render_rulers(editor, doc, view, inner, surface, theme);
+        Self::render_gutter(
+            editor,
+            doc,
+            view,
+            view.area,
+            theme,
+            is_focused,
+            &mut line_decorations,
+        );
 
         if is_focused {
-            Self::render_focused_view_elements(view, doc, inner, theme, surface);
+            let cursor = doc
+                .selection(view.id)
+                .primary()
+                .cursor(doc.text().slice(..));
+            // set the cursor_cache to out of view in case the position is not found
+            editor.cursor_cache.set(Some(None));
+            let update_cursor_cache =
+                |_: &mut TextRenderer, pos| editor.cursor_cache.set(Some(Some(pos)));
+            translated_positions.push((cursor, Box::new(update_cursor_cache)));
         }
 
+        render_document(
+            surface,
+            inner,
+            doc,
+            view.offset,
+            &text_annotations,
+            highlights,
+            theme,
+            &mut line_decorations,
+            &mut *translated_positions,
+        );
+        Self::render_rulers(editor, doc, view, inner, surface, theme);
+
         // if we're not at the edge of the screen, draw a right border
         if viewport.right() != view.area.right() {
             let x = area.right();
@@ -203,31 +257,53 @@ impl EditorView {
             .iter()
             // View might be horizontally scrolled, convert from absolute distance
             // from the 1st column to relative distance from left of viewport
-            .filter_map(|ruler| ruler.checked_sub(1 + view.offset.col as u16))
+            .filter_map(|ruler| ruler.checked_sub(1 + view.offset.horizontal_offset as u16))
             .filter(|ruler| ruler < &viewport.width)
             .map(|ruler| viewport.clip_left(ruler).with_width(1))
             .for_each(|area| surface.set_style(area, ruler_theme))
     }
 
+    pub fn overlay_syntax_highlights(
+        doc: &Document,
+        anchor: usize,
+        height: u16,
+        text_annotations: &TextAnnotations,
+    ) -> Vec<(usize, std::ops::Range<usize>)> {
+        let text = doc.text().slice(..);
+        let row = text.char_to_line(anchor.min(text.len_chars()));
+
+        let range = {
+            // Calculate viewport byte ranges:
+            // Saturating subs to make it inclusive zero indexing.
+            let last_line = text.len_lines().saturating_sub(1);
+            let last_visible_line = (row + height as usize).saturating_sub(1).min(last_line);
+            let start = text.line_to_byte(row.min(last_line));
+            let end = text.line_to_byte(last_visible_line + 1);
+
+            start..end
+        };
+
+        text_annotations.collect_overlay_highlights(range)
+    }
+
     /// Get syntax highlights for a document in a view represented by the first line
     /// and column (`offset`) and the last line. This is done instead of using a view
     /// directly to enable rendering syntax highlighted docs anywhere (eg. picker preview)
     pub fn doc_syntax_highlights<'doc>(
         doc: &'doc Document,
-        offset: Position,
+        anchor: usize,
         height: u16,
         _theme: &Theme,
     ) -> Box<dyn Iterator<Item = HighlightEvent> + 'doc> {
         let text = doc.text().slice(..);
+        let row = text.char_to_line(anchor.min(text.len_chars()));
 
         let range = {
             // Calculate viewport byte ranges:
             // Saturating subs to make it inclusive zero indexing.
-            let last_line = doc.text().len_lines().saturating_sub(1);
-            let last_visible_line = (offset.row + height as usize)
-                .saturating_sub(1)
-                .min(last_line);
-            let start = text.line_to_byte(offset.row.min(last_line));
+            let last_line = text.len_lines().saturating_sub(1);
+            let last_visible_line = (row + height as usize).saturating_sub(1).min(last_line);
+            let start = text.line_to_byte(row.min(last_line));
             let end = text.line_to_byte(last_visible_line + 1);
 
             start..end
@@ -272,11 +348,11 @@ impl EditorView {
         use helix_core::diagnostic::Severity;
         let get_scope_of = |scope| {
             theme
-            .find_scope_index(scope)
+            .find_scope_index_exact(scope)
             // get one of the themes below as fallback values
-            .or_else(|| theme.find_scope_index("diagnostic"))
-            .or_else(|| theme.find_scope_index("ui.cursor"))
-            .or_else(|| theme.find_scope_index("ui.selection"))
+            .or_else(|| theme.find_scope_index_exact("diagnostic"))
+            .or_else(|| theme.find_scope_index_exact("ui.cursor"))
+            .or_else(|| theme.find_scope_index_exact("ui.selection"))
             .expect(
                 "at least one of the following scopes must be defined in the theme: `diagnostic`, `ui.cursor`, or `ui.selection`",
             )
@@ -339,29 +415,29 @@ impl EditorView {
         let cursor_is_block = cursorkind == CursorKind::Block;
 
         let selection_scope = theme
-            .find_scope_index("ui.selection")
+            .find_scope_index_exact("ui.selection")
             .expect("could not find `ui.selection` scope in the theme!");
         let primary_selection_scope = theme
-            .find_scope_index("ui.selection.primary")
+            .find_scope_index_exact("ui.selection.primary")
             .unwrap_or(selection_scope);
         let base_cursor_scope = theme
-            .find_scope_index("ui.cursor")
+            .find_scope_index_exact("ui.cursor")
             .unwrap_or(selection_scope);
         let base_primary_cursor_scope = theme
             .find_scope_index("ui.cursor.primary")
             .unwrap_or(base_cursor_scope);
 
         let cursor_scope = match mode {
-            Mode::Insert => theme.find_scope_index("ui.cursor.insert"),
-            Mode::Select => theme.find_scope_index("ui.cursor.select"),
-            Mode::Normal => theme.find_scope_index("ui.cursor.normal"),
+            Mode::Insert => theme.find_scope_index_exact("ui.cursor.insert"),
+            Mode::Select => theme.find_scope_index_exact("ui.cursor.select"),
+            Mode::Normal => theme.find_scope_index_exact("ui.cursor.normal"),
         }
         .unwrap_or(base_cursor_scope);
 
         let primary_cursor_scope = match mode {
-            Mode::Insert => theme.find_scope_index("ui.cursor.primary.insert"),
-            Mode::Select => theme.find_scope_index("ui.cursor.primary.select"),
-            Mode::Normal => theme.find_scope_index("ui.cursor.primary.normal"),
+            Mode::Insert => theme.find_scope_index_exact("ui.cursor.primary.insert"),
+            Mode::Select => theme.find_scope_index_exact("ui.cursor.primary.select"),
+            Mode::Normal => theme.find_scope_index_exact("ui.cursor.primary.normal"),
         }
         .unwrap_or(base_primary_cursor_scope);
 
@@ -424,248 +500,26 @@ impl EditorView {
         spans
     }
 
-    pub fn render_text_highlights<H: Iterator<Item = HighlightEvent>>(
-        doc: &Document,
-        offset: Position,
-        viewport: Rect,
-        surface: &mut Surface,
-        theme: &Theme,
-        highlights: H,
-        config: &helix_view::editor::Config,
-    ) {
-        let whitespace = &config.whitespace;
-        use helix_view::editor::WhitespaceRenderValue;
-
-        // It's slightly more efficient to produce a full RopeSlice from the Rope, then slice that a bunch
-        // of times than it is to always call Rope::slice/get_slice (it will internally always hit RSEnum::Light).
-        let text = doc.text().slice(..);
-
-        let characters = &whitespace.characters;
-
-        let mut spans = Vec::new();
-        let mut visual_x = 0usize;
-        let mut line = 0u16;
-        let tab_width = doc.tab_width();
-        let tab = if whitespace.render.tab() == WhitespaceRenderValue::All {
-            std::iter::once(characters.tab)
-                .chain(std::iter::repeat(characters.tabpad).take(tab_width - 1))
-                .collect()
-        } else {
-            " ".repeat(tab_width)
-        };
-        let space = characters.space.to_string();
-        let nbsp = characters.nbsp.to_string();
-        let newline = if whitespace.render.newline() == WhitespaceRenderValue::All {
-            characters.newline.to_string()
-        } else {
-            " ".to_string()
-        };
-        let indent_guide_char = config.indent_guides.character.to_string();
-
-        let text_style = theme.get("ui.text");
-        let whitespace_style = theme.get("ui.virtual.whitespace");
-
-        let mut is_in_indent_area = true;
-        let mut last_line_indent_level = 0;
-
-        // use whitespace style as fallback for indent-guide
-        let indent_guide_style = text_style.patch(
-            theme
-                .try_get("ui.virtual.indent-guide")
-                .unwrap_or_else(|| theme.get("ui.virtual.whitespace")),
-        );
-
-        let draw_indent_guides = |indent_level, line, surface: &mut Surface| {
-            if !config.indent_guides.render {
-                return;
-            }
-
-            let starting_indent =
-                (offset.col / tab_width) + config.indent_guides.skip_levels as usize;
-
-            // Don't draw indent guides outside of view
-            let end_indent = min(
-                indent_level,
-                // Add tab_width - 1 to round up, since the first visible
-                // indent might be a bit after offset.col
-                offset.col + viewport.width as usize + (tab_width - 1),
-            ) / tab_width;
-
-            for i in starting_indent..end_indent {
-                let x = (viewport.x as usize + (i * tab_width) - offset.col) as u16;
-                let y = viewport.y + line;
-                debug_assert!(surface.in_bounds(x, y));
-                surface.set_string(x, y, &indent_guide_char, indent_guide_style);
-            }
-        };
-
-        'outer: for event in highlights {
-            match event {
-                HighlightEvent::HighlightStart(span) => {
-                    spans.push(span);
-                }
-                HighlightEvent::HighlightEnd => {
-                    spans.pop();
-                }
-                HighlightEvent::Source { start, end } => {
-                    let is_trailing_cursor = text.len_chars() < end;
-
-                    // `unwrap_or_else` part is for off-the-end indices of
-                    // the rope, to allow cursor highlighting at the end
-                    // of the rope.
-                    let text = text.get_slice(start..end).unwrap_or_else(|| " ".into());
-                    let style = spans
-                        .iter()
-                        .fold(text_style, |acc, span| acc.patch(theme.highlight(span.0)));
-
-                    let space = if whitespace.render.space() == WhitespaceRenderValue::All
-                        && !is_trailing_cursor
-                    {
-                        &space
-                    } else {
-                        " "
-                    };
-
-                    let nbsp = if whitespace.render.nbsp() == WhitespaceRenderValue::All
-                        && text.len_chars() < end
-                    {
-                        &nbsp
-                    } else {
-                        " "
-                    };
-
-                    use helix_core::graphemes::{grapheme_width, RopeGraphemes};
-
-                    for grapheme in RopeGraphemes::new(text) {
-                        let out_of_bounds = offset.col > visual_x
-                            || visual_x >= viewport.width as usize + offset.col;
-
-                        if LineEnding::from_rope_slice(&grapheme).is_some() {
-                            if !out_of_bounds {
-                                // we still want to render an empty cell with the style
-                                surface.set_string(
-                                    (viewport.x as usize + visual_x - offset.col) as u16,
-                                    viewport.y + line,
-                                    &newline,
-                                    style.patch(whitespace_style),
-                                );
-                            }
-
-                            draw_indent_guides(last_line_indent_level, line, surface);
-
-                            visual_x = 0;
-                            line += 1;
-                            is_in_indent_area = true;
-
-                            // TODO: with proper iter this shouldn't be necessary
-                            if line >= viewport.height {
-                                break 'outer;
-                            }
-                        } else {
-                            let grapheme = Cow::from(grapheme);
-                            let is_whitespace;
-
-                            let (display_grapheme, width) = if grapheme == "\t" {
-                                is_whitespace = true;
-                                // make sure we display tab as appropriate amount of spaces
-                                let visual_tab_width = tab_width - (visual_x % tab_width);
-                                let grapheme_tab_width =
-                                    helix_core::str_utils::char_to_byte_idx(&tab, visual_tab_width);
-
-                                (&tab[..grapheme_tab_width], visual_tab_width)
-                            } else if grapheme == " " {
-                                is_whitespace = true;
-                                (space, 1)
-                            } else if grapheme == "\u{00A0}" {
-                                is_whitespace = true;
-                                (nbsp, 1)
-                            } else {
-                                is_whitespace = false;
-                                // Cow will prevent allocations if span contained in a single slice
-                                // which should really be the majority case
-                                let width = grapheme_width(&grapheme);
-                                (grapheme.as_ref(), width)
-                            };
-
-                            let cut_off_start = offset.col.saturating_sub(visual_x);
-
-                            if !out_of_bounds {
-                                // if we're offscreen just keep going until we hit a new line
-                                surface.set_string(
-                                    (viewport.x as usize + visual_x - offset.col) as u16,
-                                    viewport.y + line,
-                                    display_grapheme,
-                                    if is_whitespace {
-                                        style.patch(whitespace_style)
-                                    } else {
-                                        style
-                                    },
-                                );
-                            } else if cut_off_start != 0 && cut_off_start < width {
-                                // partially on screen
-                                let rect = Rect::new(
-                                    viewport.x,
-                                    viewport.y + line,
-                                    (width - cut_off_start) as u16,
-                                    1,
-                                );
-                                surface.set_style(
-                                    rect,
-                                    if is_whitespace {
-                                        style.patch(whitespace_style)
-                                    } else {
-                                        style
-                                    },
-                                );
-                            }
-
-                            if is_in_indent_area && !(grapheme == " " || grapheme == "\t") {
-                                draw_indent_guides(visual_x, line, surface);
-                                is_in_indent_area = false;
-                                last_line_indent_level = visual_x;
-                            }
-
-                            visual_x = visual_x.saturating_add(width);
-                        }
-                    }
-                }
-            }
-        }
-    }
-
     /// Render brace match, etc (meant for the focused view only)
-    pub fn render_focused_view_elements(
+    pub fn highlight_focused_view_elements(
         view: &View,
         doc: &Document,
-        viewport: Rect,
         theme: &Theme,
-        surface: &mut Surface,
-    ) {
+    ) -> Vec<(usize, std::ops::Range<usize>)> {
         // Highlight matching braces
         if let Some(syntax) = doc.syntax() {
             let text = doc.text().slice(..);
             use helix_core::match_brackets;
             let pos = doc.selection(view.id).primary().cursor(text);
 
-            let pos = match_brackets::find_matching_bracket(syntax, doc.text(), pos)
-                .and_then(|pos| view.screen_coords_at_pos(doc, text, pos));
-
-            if let Some(pos) = pos {
+            if let Some(pos) = match_brackets::find_matching_bracket(syntax, doc.text(), pos) {
                 // ensure col is on screen
-                if (pos.col as u16) < viewport.width + view.offset.col as u16
-                    && pos.col >= view.offset.col
-                {
-                    let style = theme.try_get("ui.cursor.match").unwrap_or_else(|| {
-                        Style::default()
-                            .add_modifier(Modifier::REVERSED)
-                            .add_modifier(Modifier::DIM)
-                    });
-
-                    surface[(viewport.x + pos.col as u16, viewport.y + pos.row as u16)]
-                        .set_style(style);
+                if let Some(highlight) = theme.find_scope_index_exact("ui.cursor.match") {
+                    return vec![(highlight, pos..pos + 1)];
                 }
             }
         }
+        Vec::new()
     }
 
     /// Render bufferline at the top
@@ -721,22 +575,17 @@ impl EditorView {
         }
     }
 
-    pub fn render_gutter(
-        editor: &Editor,
-        doc: &Document,
+    pub fn render_gutter<'d>(
+        editor: &'d Editor,
+        doc: &'d Document,
         view: &View,
         viewport: Rect,
-        surface: &mut Surface,
         theme: &Theme,
         is_focused: bool,
+        line_decorations: &mut Vec<Box<(dyn LineDecoration + 'd)>>,
     ) {
         let text = doc.text().slice(..);
-        let last_line = view.last_line(doc);
-
-        // it's used inside an iterator so the collect isn't needless:
-        // https://github.com/rust-lang/rust-clippy/issues/6164
-        #[allow(clippy::needless_collect)]
-        let cursors: Vec<_> = doc
+        let cursors: Rc<[_]> = doc
             .selection(view.id)
             .iter()
             .map(|range| range.cursor_line(text))
@@ -746,29 +595,36 @@ impl EditorView {
 
         let gutter_style = theme.get("ui.gutter");
         let gutter_selected_style = theme.get("ui.gutter.selected");
-
-        // avoid lots of small allocations by reusing a text buffer for each line
-        let mut text = String::with_capacity(8);
+        let gutter_style_virtual = theme.get("ui.gutter.virtual");
+        let gutter_selected_style_virtual = theme.get("ui.gutter.selected.virtual");
 
         for gutter_type in view.gutters() {
             let mut gutter = gutter_type.style(editor, doc, view, theme, is_focused);
             let width = gutter_type.width(view, doc);
-            text.reserve(width); // ensure there's enough space for the gutter
-            for (i, line) in (view.offset.row..(last_line + 1)).enumerate() {
-                let selected = cursors.contains(&line);
+            // avoid lots of small allocations by reusing a text buffer for each line
+            let mut text = String::with_capacity(width);
+            let cursors = cursors.clone();
+            let gutter_decoration = move |renderer: &mut TextRenderer, pos: LinePos| {
+                // TODO handle softwrap in gutters
+                let selected = cursors.contains(&pos.doc_line);
                 let x = viewport.x + offset;
-                let y = viewport.y + i as u16;
+                let y = viewport.y + pos.visual_line;
 
-                let gutter_style = if selected {
-                    gutter_selected_style
-                } else {
-                    gutter_style
+                let gutter_style = match (selected, pos.first_visual_line) {
+                    (false, true) => gutter_style,
+                    (true, true) => gutter_selected_style,
+                    (false, false) => gutter_style_virtual,
+                    (true, false) => gutter_selected_style_virtual,
                 };
 
-                if let Some(style) = gutter(line, selected, &mut text) {
-                    surface.set_stringn(x, y, &text, width, gutter_style.patch(style));
+                if let Some(style) =
+                    gutter(pos.doc_line, selected, pos.first_visual_line, &mut text)
+                {
+                    renderer
+                        .surface
+                        .set_stringn(x, y, &text, width, gutter_style.patch(style));
                 } else {
-                    surface.set_style(
+                    renderer.surface.set_style(
                         Rect {
                             x,
                             y,
@@ -779,7 +635,8 @@ impl EditorView {
                     );
                 }
                 text.clear();
-            }
+            };
+            line_decorations.push(Box::new(gutter_decoration));
 
             offset += width as u16;
         }
@@ -840,10 +697,13 @@ impl EditorView {
     }
 
     /// Apply the highlighting on the lines where a cursor is active
-    pub fn highlight_cursorline(doc: &Document, view: &View, surface: &mut Surface, theme: &Theme) {
+    pub fn cursorline_decorator(
+        doc: &Document,
+        view: &View,
+        theme: &Theme,
+    ) -> Box<dyn LineDecoration> {
         let text = doc.text().slice(..);
-        let last_line = view.last_line(doc);
-
+        // TODO only highlight the visual line that contains the cursor instead of the full visual line
         let primary_line = doc.selection(view.id).primary().cursor_line(text);
 
         // The secondary_lines do contain the primary_line, it doesn't matter
@@ -860,20 +720,23 @@ impl EditorView {
 
         let primary_style = theme.get("ui.cursorline.primary");
         let secondary_style = theme.get("ui.cursorline.secondary");
+        let viewport = view.area;
 
-        for line in view.offset.row..(last_line + 1) {
+        let line_decoration = move |renderer: &mut TextRenderer, pos: LinePos| {
             let area = Rect::new(
-                view.area.x,
-                view.area.y + (line - view.offset.row) as u16,
-                view.area.width,
+                viewport.x,
+                viewport.y + pos.visual_line as u16,
+                viewport.width,
                 1,
             );
-            if primary_line == line {
-                surface.set_style(area, primary_style);
-            } else if secondary_lines.binary_search(&line).is_ok() {
-                surface.set_style(area, secondary_style);
+            if primary_line == pos.doc_line {
+                renderer.surface.set_style(area, primary_style);
+            } else if secondary_lines.binary_search(&pos.doc_line).is_ok() {
+                renderer.surface.set_style(area, secondary_style);
             }
-        }
+        };
+
+        Box::new(line_decoration)
     }
 
     /// Apply the highlighting on the columns where a cursor is active
@@ -882,6 +745,8 @@ impl EditorView {
         view: &View,
         surface: &mut Surface,
         theme: &Theme,
+        viewport: Rect,
+        text_annotations: &TextAnnotations,
     ) {
         let text = doc.text().slice(..);
 
@@ -897,19 +762,23 @@ impl EditorView {
             .unwrap_or_else(|| theme.get("ui.cursorline.secondary"));
 
         let inner_area = view.inner_area(doc);
-        let offset = view.offset.col;
 
         let selection = doc.selection(view.id);
         let primary = selection.primary();
+        let text_format = doc.text_format(viewport.width, None);
         for range in selection.iter() {
             let is_primary = primary == *range;
+            let cursor = range.cursor(text);
+
+            let Position { col, .. } =
+                visual_offset_from_block(text, cursor, cursor, &text_format, text_annotations).0;
 
-            let Position { row: _, col } =
-                visual_coords_at_pos(text, range.cursor(text), doc.tab_width());
             // if the cursor is horizontally in the view
-            if col >= offset && inner_area.width > (col - offset) as u16 {
+            if col >= view.offset.horizontal_offset
+                && inner_area.width > (col - view.offset.horizontal_offset) as u16
+            {
                 let area = Rect::new(
-                    inner_area.x + (col - offset) as u16,
+                    inner_area.x + (col - view.offset.horizontal_offset) as u16,
                     view.area.y,
                     1,
                     view.area.height,
@@ -1149,7 +1018,7 @@ impl EditorView {
 
         let pos_and_view = |editor: &Editor, row, column| {
             editor.tree.views().find_map(|(view, _focus)| {
-                view.pos_at_screen_coords(&editor.documents[&view.doc], row, column)
+                view.pos_at_screen_coords(&editor.documents[&view.doc], row, column, true)
                     .map(|pos| (pos, view.id))
             })
         };
@@ -1191,8 +1060,10 @@ impl EditorView {
                         None => return EventResult::Ignored(None),
                     };
 
-                    let line = coords.row + view.offset.row;
-                    if line < doc.text().len_lines() {
+                    if let Some(char_idx) =
+                        view.pos_at_visual_coords(doc, coords.row as u16, coords.col as u16, true)
+                    {
+                        let line = doc.text().char_to_line(char_idx);
                         commands::dap_toggle_breakpoint_impl(cxt, path, line);
                         return EventResult::Consumed(None);
                     }
@@ -1204,7 +1075,7 @@ impl EditorView {
             MouseEventKind::Drag(MouseButton::Left) => {
                 let (view, doc) = current!(cxt.editor);
 
-                let pos = match view.pos_at_screen_coords(doc, row, column) {
+                let pos = match view.pos_at_screen_coords(doc, row, column, true) {
                     Some(pos) => pos,
                     None => return EventResult::Ignored(None),
                 };
@@ -1268,8 +1139,9 @@ impl EditorView {
                     cxt.editor.focus(view_id);
 
                     let (view, doc) = current!(cxt.editor);
-                    let line = coords.row + view.offset.row;
-                    if let Ok(pos) = doc.text().try_line_to_char(line) {
+                    if let Some(pos) =
+                        view.pos_at_visual_coords(doc, coords.row as u16, coords.col as u16, true)
+                    {
                         doc.set_selection(view_id, Selection::point(pos));
                         if modifiers == KeyModifiers::ALT {
                             commands::MappableCommand::dap_edit_log.execute(cxt);

helix-term/src/ui/lsp.rs

@@ -53,7 +53,10 @@ impl Component for SignatureHelp {
 
         let active_param_span = self.active_param_range.map(|(start, end)| {
             vec![(
-                cx.editor.theme.find_scope_index("ui.selection").unwrap(),
+                cx.editor
+                    .theme
+                    .find_scope_index_exact("ui.selection")
+                    .unwrap(),
                 start..end,
             )]
         });

helix-term/src/ui/mod.rs

@@ -1,4 +1,5 @@
 mod completion;
+mod document;
 pub(crate) mod editor;
 mod fuzzy_match;
 mod info;

helix-term/src/ui/picker.rs

@@ -2,7 +2,12 @@ use crate::{
     alt,
     compositor::{Component, Compositor, Context, Event, EventResult},
     ctrl, key, shift,
-    ui::{self, fuzzy_match::FuzzyQuery, EditorView},
+    ui::{
+        self,
+        document::{render_document, LineDecoration, LinePos, TextRenderer},
+        fuzzy_match::FuzzyQuery,
+        EditorView,
+    },
 };
 use futures_util::future::BoxFuture;
 use tui::{
@@ -19,11 +24,15 @@ use std::cmp::{self, Ordering};
 use std::{collections::HashMap, io::Read, path::PathBuf};
 
 use crate::ui::{Prompt, PromptEvent};
-use helix_core::{movement::Direction, unicode::segmentation::UnicodeSegmentation, Position};
+use helix_core::{
+    movement::Direction, text_annotations::TextAnnotations,
+    unicode::segmentation::UnicodeSegmentation, Position,
+};
 use helix_view::{
     editor::Action,
     graphics::{CursorKind, Margin, Modifier, Rect},
     theme::Style,
+    view::ViewPosition,
     Document, DocumentId, Editor,
 };
 
@@ -179,7 +188,7 @@ impl<T: Item> FilePicker<T> {
                             }
                             _ => {
                                 // TODO: enable syntax highlighting; blocked by async rendering
-                                Document::open(path, None, None)
+                                Document::open(path, None, None, editor.config.clone())
                                     .map(|doc| CachedPreview::Document(Box::new(doc)))
                                     .unwrap_or(CachedPreview::NotFound)
                             }
@@ -283,43 +292,57 @@ impl<T: Item + 'static> Component for FilePicker<T> {
                 })
                 .unwrap_or(0);
 
-            let offset = Position::new(first_line, 0);
+            let offset = ViewPosition {
+                anchor: doc.text().line_to_char(first_line),
+                horizontal_offset: 0,
+                vertical_offset: 0,
+            };
 
-            let mut highlights =
-                EditorView::doc_syntax_highlights(doc, offset, area.height, &cx.editor.theme);
+            let mut highlights = EditorView::doc_syntax_highlights(
+                doc,
+                offset.anchor,
+                area.height,
+                &cx.editor.theme,
+            );
             for spans in EditorView::doc_diagnostics_highlights(doc, &cx.editor.theme) {
                 if spans.is_empty() {
                     continue;
                 }
                 highlights = Box::new(helix_core::syntax::merge(highlights, spans));
             }
-            EditorView::render_text_highlights(
+            let mut decorations: Vec<Box<dyn LineDecoration>> = Vec::new();
+
+            if let Some((start, end)) = range {
+                let style = cx
+                    .editor
+                    .theme
+                    .try_get("ui.highlight")
+                    .unwrap_or_else(|| cx.editor.theme.get("ui.selection"));
+                let draw_highlight = move |renderer: &mut TextRenderer, pos: LinePos| {
+                    if (start..=end).contains(&pos.doc_line) {
+                        let area = Rect::new(
+                            renderer.viewport.x,
+                            renderer.viewport.y + pos.visual_line,
+                            renderer.viewport.width,
+                            1,
+                        );
+                        renderer.surface.set_style(area, style)
+                    }
+                };
+                decorations.push(Box::new(draw_highlight))
+            }
+
+            render_document(
+                surface,
+                inner,
                 doc,
                 offset,
-                inner,
-                surface,
-                &cx.editor.theme,
+                &TextAnnotations::default(),
                 highlights,
-                &cx.editor.config(),
+                &cx.editor.theme,
+                &mut decorations,
+                &mut [],
             );
-
-            // highlight the line
-            if let Some((start, end)) = range {
-                let offset = start.saturating_sub(first_line) as u16;
-                surface.set_style(
-                    Rect::new(
-                        inner.x,
-                        inner.y + offset,
-                        inner.width,
-                        (end.saturating_sub(start) as u16 + 1)
-                            .min(inner.height.saturating_sub(offset)),
-                    ),
-                    cx.editor
-                        .theme
-                        .try_get("ui.highlight")
-                        .unwrap_or_else(|| cx.editor.theme.get("ui.selection")),
-                );
-            }
         }
     }
 

helix-view/src/document.rs

@@ -1,7 +1,11 @@
 use anyhow::{anyhow, bail, Context, Error};
+use arc_swap::access::DynAccess;
 use futures_util::future::BoxFuture;
 use futures_util::FutureExt;
 use helix_core::auto_pairs::AutoPairs;
+use helix_core::doc_formatter::TextFormat;
+use helix_core::syntax::Highlight;
+use helix_core::text_annotations::TextAnnotations;
 use helix_core::Range;
 use helix_vcs::{DiffHandle, DiffProviderRegistry};
 
@@ -26,8 +30,8 @@ use helix_core::{
     DEFAULT_LINE_ENDING,
 };
 
-use crate::editor::RedrawHandle;
-use crate::{DocumentId, Editor, View, ViewId};
+use crate::editor::{Config, RedrawHandle};
+use crate::{DocumentId, Editor, Theme, View, ViewId};
 
 /// 8kB of buffer space for encoding and decoding `Rope`s.
 const BUF_SIZE: usize = 8192;
@@ -127,6 +131,7 @@ pub struct Document {
     // it back as it separated from the edits. We could split out the parts manually but that will
     // be more troublesome.
     pub history: Cell<History>,
+    pub config: Arc<dyn DynAccess<Config>>,
 
     pub savepoint: Option<Transaction>,
 
@@ -351,7 +356,11 @@ use helix_lsp::lsp;
 use url::Url;
 
 impl Document {
-    pub fn from(text: Rope, encoding: Option<&'static encoding::Encoding>) -> Self {
+    pub fn from(
+        text: Rope,
+        encoding: Option<&'static encoding::Encoding>,
+        config: Arc<dyn DynAccess<Config>>,
+    ) -> Self {
         let encoding = encoding.unwrap_or(encoding::UTF_8);
         let changes = ChangeSet::new(&text);
         let old_state = None;
@@ -377,9 +386,13 @@ impl Document {
             modified_since_accessed: false,
             language_server: None,
             diff_handle: None,
+            config,
         }
     }
-
+    pub fn default(config: Arc<dyn DynAccess<Config>>) -> Self {
+        let text = Rope::from(DEFAULT_LINE_ENDING.as_str());
+        Self::from(text, None, config)
+    }
     // TODO: async fn?
     /// Create a new document from `path`. Encoding is auto-detected, but it can be manually
     /// overwritten with the `encoding` parameter.
@@ -387,6 +400,7 @@ impl Document {
         path: &Path,
         encoding: Option<&'static encoding::Encoding>,
         config_loader: Option<Arc<syntax::Loader>>,
+        config: Arc<dyn DynAccess<Config>>,
     ) -> Result<Self, Error> {
         // Open the file if it exists, otherwise assume it is a new file (and thus empty).
         let (rope, encoding) = if path.exists() {
@@ -398,7 +412,7 @@ impl Document {
             (Rope::from(DEFAULT_LINE_ENDING.as_str()), encoding)
         };
 
-        let mut doc = Self::from(rope, Some(encoding));
+        let mut doc = Self::from(rope, Some(encoding), config);
 
         // set the path and try detecting the language
         doc.set_path(Some(path))?;
@@ -1192,12 +1206,34 @@ impl Document {
             None => global_config,
         }
     }
-}
 
-impl Default for Document {
-    fn default() -> Self {
-        let text = Rope::from(DEFAULT_LINE_ENDING.as_str());
-        Self::from(text, None)
+    pub fn text_format(&self, mut viewport_width: u16, theme: Option<&Theme>) -> TextFormat {
+        if let Some(max_line_len) = self
+            .language_config()
+            .and_then(|config| config.max_line_length)
+        {
+            viewport_width = viewport_width.min(max_line_len as u16)
+        }
+        let config = self.config.load();
+        let soft_wrap = &config.soft_wrap;
+        let tab_width = self.tab_width() as u16;
+        TextFormat {
+            soft_wrap: soft_wrap.enable && viewport_width > 10,
+            tab_width,
+            max_wrap: soft_wrap.max_wrap.min(viewport_width / 4),
+            max_indent_retain: soft_wrap.max_indent_retain.min(viewport_width * 2 / 5),
+            // avoid spinning forever when the window manager
+            // sets the size to something tiny
+            viewport_width,
+            wrap_indicator: soft_wrap.wrap_indicator.clone().into_boxed_str(),
+            wrap_indicator_highlight: theme
+                .and_then(|theme| theme.find_scope_index("ui.virtual.wrap"))
+                .map(Highlight),
+        }
+    }
+
+    pub fn text_annotations(&self, _theme: Option<&Theme>) -> TextAnnotations {
+        TextAnnotations::default()
     }
 }
 
@@ -1236,13 +1272,19 @@ impl Display for FormatterError {
 
 #[cfg(test)]
 mod test {
+    use arc_swap::ArcSwap;
+
     use super::*;
 
     #[test]
     fn changeset_to_changes_ignore_line_endings() {
         use helix_lsp::{lsp, Client, OffsetEncoding};
         let text = Rope::from("hello\r\nworld");
-        let mut doc = Document::from(text, None);
+        let mut doc = Document::from(
+            text,
+            None,
+            Arc::new(ArcSwap::new(Arc::new(Config::default()))),
+        );
         let view = ViewId::default();
         doc.set_selection(view, Selection::single(0, 0));
 
@@ -1276,7 +1318,11 @@ mod test {
     fn changeset_to_changes() {
         use helix_lsp::{lsp, Client, OffsetEncoding};
         let text = Rope::from("hello");
-        let mut doc = Document::from(text, None);
+        let mut doc = Document::from(
+            text,
+            None,
+            Arc::new(ArcSwap::new(Arc::new(Config::default()))),
+        );
         let view = ViewId::default();
         doc.set_selection(view, Selection::single(5, 5));
 
@@ -1389,7 +1435,9 @@ mod test {
     #[test]
     fn test_line_ending() {
         assert_eq!(
-            Document::default().text().to_string(),
+            Document::default(Arc::new(ArcSwap::new(Arc::new(Config::default()))))
+                .text()
+                .to_string(),
             DEFAULT_LINE_ENDING.as_str()
         );
     }

helix-view/src/editor.rs

@@ -7,6 +7,7 @@ use crate::{
     input::KeyEvent,
     theme::{self, Theme},
     tree::{self, Tree},
+    view::ViewPosition,
     Align, Document, DocumentId, View, ViewId,
 };
 use helix_vcs::DiffProviderRegistry;
@@ -18,6 +19,7 @@ use tokio_stream::wrappers::UnboundedReceiverStream;
 
 use std::{
     borrow::Cow,
+    cell::Cell,
     collections::{BTreeMap, HashMap},
     io::stdin,
     num::NonZeroUsize,
@@ -268,6 +270,44 @@ pub struct Config {
     pub indent_guides: IndentGuidesConfig,
     /// Whether to color modes with different colors. Defaults to `false`.
     pub color_modes: bool,
+    pub soft_wrap: SoftWrap,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+#[serde(default, rename_all = "kebab-case", deny_unknown_fields)]
+pub struct SoftWrap {
+    /// Soft wrap lines that exceed viewport width. Default to off
+    pub enable: bool,
+    /// Maximum space left free at the end of the line.
+    /// This space is used to wrap text at word boundaries. If that is not possible within this limit
+    /// the word is simply split at the end of the line.
+    ///
+    /// This is automatically hard-limited to a quarter of the viewport to ensure correct display on small views.
+    ///
+    /// Default to 20
+    pub max_wrap: u16,
+    /// Maximum number of indentation that can be carried over from the previous line when softwrapping.
+    /// If a line is indented further then this limit it is rendered at the start of the viewport instead.
+    ///
+    /// This is automatically hard-limited to a quarter of the viewport to ensure correct display on small views.
+    ///
+    /// Default to 40
+    pub max_indent_retain: u16,
+    /// Indicator placed at the beginning of softwrapped lines
+    ///
+    /// Defaults to ↪
+    pub wrap_indicator: String,
+}
+
+impl Default for SoftWrap {
+    fn default() -> Self {
+        SoftWrap {
+            enable: false,
+            max_wrap: 20,
+            max_indent_retain: 40,
+            wrap_indicator: "↪ ".into(),
+        }
+    }
 }
 
 #[derive(Debug, Default, Clone, PartialEq, Eq, Serialize, Deserialize)]
@@ -717,6 +757,7 @@ impl Default for Config {
             bufferline: BufferLine::default(),
             indent_guides: IndentGuidesConfig::default(),
             color_modes: false,
+            soft_wrap: SoftWrap::default(),
         }
     }
 }
@@ -797,7 +838,7 @@ pub struct Editor {
     pub status_msg: Option<(Cow<'static, str>, Severity)>,
     pub autoinfo: Option<Info>,
 
-    pub config: Box<dyn DynAccess<Config>>,
+    pub config: Arc<dyn DynAccess<Config>>,
     pub auto_pairs: Option<AutoPairs>,
 
     pub idle_timer: Pin<Box<Sleep>>,
@@ -813,6 +854,19 @@ pub struct Editor {
     /// The `RwLock` blocks the editor from performing the render until an exclusive lock can be aquired
     pub redraw_handle: RedrawHandle,
     pub needs_redraw: bool,
+    /// Cached position of the cursor calculated during rendering.
+    /// The content of `cursor_cache` is returned by `Editor::cursor` if
+    /// set to `Some(_)`. The value will be cleared after it's used.
+    /// If `cursor_cache` is `None` then the `Editor::cursor` function will
+    /// calculate the cursor position.
+    ///
+    /// `Some(None)` represents a cursor position outside of the visible area.
+    /// This will just cause `Editor::cursor` to return `None`.
+    ///
+    /// This cache is only a performance optimization to
+    /// avoid calculating the cursor position multiple
+    /// times during rendering and should not be set by other functions.
+    pub cursor_cache: Cell<Option<Option<Position>>>,
 }
 
 pub type RedrawHandle = (Arc<Notify>, Arc<RwLock<()>>);
@@ -866,7 +920,7 @@ impl Editor {
         mut area: Rect,
         theme_loader: Arc<theme::Loader>,
         syn_loader: Arc<syntax::Loader>,
-        config: Box<dyn DynAccess<Config>>,
+        config: Arc<dyn DynAccess<Config>>,
     ) -> Self {
         let conf = config.load();
         let auto_pairs = (&conf.auto_pairs).into();
@@ -910,6 +964,7 @@ impl Editor {
             config_events: unbounded_channel(),
             redraw_handle: Default::default(),
             needs_redraw: false,
+            cursor_cache: Cell::new(None),
         }
     }
 
@@ -994,7 +1049,7 @@ impl Editor {
 
     fn set_theme_impl(&mut self, theme: Theme, preview: ThemeAction) {
         // `ui.selection` is the only scope required to be able to render a theme.
-        if theme.find_scope_index("ui.selection").is_none() {
+        if theme.find_scope_index_exact("ui.selection").is_none() {
             self.set_error("Invalid theme: `ui.selection` required");
             return;
         }
@@ -1077,7 +1132,7 @@ impl Editor {
     fn replace_document_in_view(&mut self, current_view: ViewId, doc_id: DocumentId) {
         let view = self.tree.get_mut(current_view);
         view.doc = doc_id;
-        view.offset = Position::default();
+        view.offset = ViewPosition::default();
 
         let doc = doc_mut!(self, &doc_id);
         doc.ensure_view_init(view.id);
@@ -1204,12 +1259,15 @@ impl Editor {
     }
 
     pub fn new_file(&mut self, action: Action) -> DocumentId {
-        self.new_file_from_document(action, Document::default())
+        self.new_file_from_document(action, Document::default(self.config.clone()))
     }
 
     pub fn new_file_from_stdin(&mut self, action: Action) -> Result<DocumentId, Error> {
         let (rope, encoding) = crate::document::from_reader(&mut stdin(), None)?;
-        Ok(self.new_file_from_document(action, Document::from(rope, Some(encoding))))
+        Ok(self.new_file_from_document(
+            action,
+            Document::from(rope, Some(encoding), self.config.clone()),
+        ))
     }
 
     // ??? possible use for integration tests
@@ -1220,7 +1278,12 @@ impl Editor {
         let id = if let Some(id) = id {
             id
         } else {
-            let mut doc = Document::open(&path, None, Some(self.syn_loader.clone()))?;
+            let mut doc = Document::open(
+                &path,
+                None,
+                Some(self.syn_loader.clone()),
+                self.config.clone(),
+            )?;
 
             let _ = Self::launch_language_server(&mut self.language_servers, &mut doc);
             if let Some(diff_base) = self.diff_providers.get_diff_base(&path) {
@@ -1306,7 +1369,7 @@ impl Editor {
                 .iter()
                 .map(|(&doc_id, _)| doc_id)
                 .next()
-                .unwrap_or_else(|| self.new_document(Document::default()));
+                .unwrap_or_else(|| self.new_document(Document::default(self.config.clone())));
             let view = View::new(doc_id, self.config().gutters.clone());
             let view_id = self.tree.insert(view);
             let doc = doc_mut!(self, &doc_id);
@@ -1440,7 +1503,11 @@ impl Editor {
             .selection(view.id)
             .primary()
             .cursor(doc.text().slice(..));
-        if let Some(mut pos) = view.screen_coords_at_pos(doc, doc.text().slice(..), cursor) {
+        let pos = self
+            .cursor_cache
+            .get()
+            .unwrap_or_else(|| view.screen_coords_at_pos(doc, doc.text().slice(..), cursor));
+        if let Some(mut pos) = pos {
             let inner = view.inner_area(doc);
             pos.col += inner.x as usize;
             pos.row += inner.y as usize;

helix-view/src/gutter.rs

@@ -12,7 +12,7 @@ fn count_digits(n: usize) -> usize {
     std::iter::successors(Some(n), |&n| (n >= 10).then(|| n / 10)).count()
 }
 
-pub type GutterFn<'doc> = Box<dyn FnMut(usize, bool, &mut String) -> Option<Style> + 'doc>;
+pub type GutterFn<'doc> = Box<dyn FnMut(usize, bool, bool, &mut String) -> Option<Style> + 'doc>;
 pub type Gutter =
     for<'doc> fn(&'doc Editor, &'doc Document, &View, &Theme, bool, usize) -> GutterFn<'doc>;
 
@@ -58,31 +58,36 @@ pub fn diagnostic<'doc>(
     let hint = theme.get("hint");
     let diagnostics = doc.diagnostics();
 
-    Box::new(move |line: usize, _selected: bool, out: &mut String| {
-        use helix_core::diagnostic::Severity;
-        if let Ok(index) = diagnostics.binary_search_by_key(&line, |d| d.line) {
-            let after = diagnostics[index..].iter().take_while(|d| d.line == line);
+    Box::new(
+        move |line: usize, _selected: bool, first_visual_line: bool, out: &mut String| {
+            if !first_visual_line {
+                return None;
+            }
+            use helix_core::diagnostic::Severity;
+            if let Ok(index) = diagnostics.binary_search_by_key(&line, |d| d.line) {
+                let after = diagnostics[index..].iter().take_while(|d| d.line == line);
 
-            let before = diagnostics[..index]
-                .iter()
-                .rev()
-                .take_while(|d| d.line == line);
+                let before = diagnostics[..index]
+                    .iter()
+                    .rev()
+                    .take_while(|d| d.line == line);
 
-            let diagnostics_on_line = after.chain(before);
+                let diagnostics_on_line = after.chain(before);
 
-            // This unwrap is safe because the iterator cannot be empty as it contains at least the item found by the binary search.
-            let diagnostic = diagnostics_on_line.max_by_key(|d| d.severity).unwrap();
+                // This unwrap is safe because the iterator cannot be empty as it contains at least the item found by the binary search.
+                let diagnostic = diagnostics_on_line.max_by_key(|d| d.severity).unwrap();
 
-            write!(out, "●").unwrap();
-            return Some(match diagnostic.severity {
-                Some(Severity::Error) => error,
-                Some(Severity::Warning) | None => warning,
-                Some(Severity::Info) => info,
-                Some(Severity::Hint) => hint,
-            });
-        }
-        None
-    })
+                write!(out, "●").unwrap();
+                return Some(match diagnostic.severity {
+                    Some(Severity::Error) => error,
+                    Some(Severity::Warning) | None => warning,
+                    Some(Severity::Info) => info,
+                    Some(Severity::Hint) => hint,
+                });
+            }
+            None
+        },
+    )
 }
 
 pub fn diff<'doc>(
@@ -99,36 +104,41 @@ pub fn diff<'doc>(
         let hunks = diff_handle.hunks();
         let mut hunk_i = 0;
         let mut hunk = hunks.nth_hunk(hunk_i);
-        Box::new(move |line: usize, _selected: bool, out: &mut String| {
-            // truncating the line is fine here because we don't compute diffs
-            // for files with more lines than i32::MAX anyways
-            // we need to special case removals here
-            // these technically do not have a range of lines to highlight (`hunk.after.start == hunk.after.end`).
-            // However we still want to display these hunks correctly we must not yet skip to the next hunk here
-            while hunk.after.end < line as u32
-                || !hunk.is_pure_removal() && line as u32 == hunk.after.end
-            {
-                hunk_i += 1;
-                hunk = hunks.nth_hunk(hunk_i);
-            }
+        Box::new(
+            move |line: usize, _selected: bool, first_visual_line: bool, out: &mut String| {
+                // truncating the line is fine here because we don't compute diffs
+                // for files with more lines than i32::MAX anyways
+                // we need to special case removals here
+                // these technically do not have a range of lines to highlight (`hunk.after.start == hunk.after.end`).
+                // However we still want to display these hunks correctly we must not yet skip to the next hunk here
+                while hunk.after.end < line as u32
+                    || !hunk.is_pure_removal() && line as u32 == hunk.after.end
+                {
+                    hunk_i += 1;
+                    hunk = hunks.nth_hunk(hunk_i);
+                }
 
-            if hunk.after.start > line as u32 {
-                return None;
-            }
+                if hunk.after.start > line as u32 {
+                    return None;
+                }
 
-            let (icon, style) = if hunk.is_pure_insertion() {
-                ("▍", added)
-            } else if hunk.is_pure_removal() {
-                ("▔", deleted)
-            } else {
-                ("▍", modified)
-            };
+                let (icon, style) = if hunk.is_pure_insertion() {
+                    ("▍", added)
+                } else if hunk.is_pure_removal() {
+                    if !first_visual_line {
+                        return None;
+                    }
+                    ("▔", deleted)
+                } else {
+                    ("▍", modified)
+                };
 
-            write!(out, "{}", icon).unwrap();
-            Some(style)
-        })
+                write!(out, "{}", icon).unwrap();
+                Some(style)
+            },
+        )
     } else {
-        Box::new(move |_, _, _| None)
+        Box::new(move |_, _, _, _| None)
     }
 }
 
@@ -142,7 +152,7 @@ pub fn line_numbers<'doc>(
     let text = doc.text().slice(..);
     let width = line_numbers_width(view, doc);
 
-    let last_line_in_view = view.last_line(doc);
+    let last_line_in_view = view.estimate_last_doc_line(doc);
 
     // Whether to draw the line number for the last line of the
     // document or not.  We only draw it if it's not an empty line.
@@ -158,34 +168,42 @@ pub fn line_numbers<'doc>(
     let line_number = editor.config().line_number;
     let mode = editor.mode;
 
-    Box::new(move |line: usize, selected: bool, out: &mut String| {
-        if line == last_line_in_view && !draw_last {
-            write!(out, "{:>1$}", '~', width).unwrap();
-            Some(linenr)
-        } else {
-            use crate::{document::Mode, editor::LineNumber};
-
-            let relative = line_number == LineNumber::Relative
-                && mode != Mode::Insert
-                && is_focused
-                && current_line != line;
-
-            let display_num = if relative {
-                abs_diff(current_line, line)
+    Box::new(
+        move |line: usize, selected: bool, first_visual_line: bool, out: &mut String| {
+            if line == last_line_in_view && !draw_last {
+                write!(out, "{:>1$}", '~', width).unwrap();
+                Some(linenr)
             } else {
-                line + 1
-            };
+                use crate::{document::Mode, editor::LineNumber};
 
-            let style = if selected && is_focused {
-                linenr_select
-            } else {
-                linenr
-            };
+                let relative = line_number == LineNumber::Relative
+                    && mode != Mode::Insert
+                    && is_focused
+                    && current_line != line;
 
-            write!(out, "{:>1$}", display_num, width).unwrap();
-            Some(style)
-        }
-    })
+                let display_num = if relative {
+                    abs_diff(current_line, line)
+                } else {
+                    line + 1
+                };
+
+                let style = if selected && is_focused {
+                    linenr_select
+                } else {
+                    linenr
+                };
+
+                if first_visual_line {
+                    write!(out, "{:>1$}", display_num, width).unwrap();
+                } else {
+                    write!(out, "{:>1$}", " ", width).unwrap();
+                }
+
+                // TODO: Use then_some when MSRV reaches 1.62
+                first_visual_line.then(|| style)
+            }
+        },
+    )
 }
 
 /// The width of a "line-numbers" gutter
@@ -210,7 +228,7 @@ pub fn padding<'doc>(
     _theme: &Theme,
     _is_focused: bool,
 ) -> GutterFn<'doc> {
-    Box::new(|_line: usize, _selected: bool, _out: &mut String| None)
+    Box::new(|_line: usize, _selected: bool, _first_visual_line: bool, _out: &mut String| None)
 }
 
 #[inline(always)]
@@ -237,41 +255,46 @@ pub fn breakpoints<'doc>(
 
     let breakpoints = match breakpoints {
         Some(breakpoints) => breakpoints,
-        None => return Box::new(move |_, _, _| None),
+        None => return Box::new(move |_, _, _, _| None),
     };
 
-    Box::new(move |line: usize, _selected: bool, out: &mut String| {
-        let breakpoint = breakpoints
-            .iter()
-            .find(|breakpoint| breakpoint.line == line)?;
-
-        let mut style = if breakpoint.condition.is_some() && breakpoint.log_message.is_some() {
-            error.underline_style(UnderlineStyle::Line)
-        } else if breakpoint.condition.is_some() {
-            error
-        } else if breakpoint.log_message.is_some() {
-            info
-        } else {
-            warning
-        };
-
-        if !breakpoint.verified {
-            // Faded colors
-            style = if let Some(Color::Rgb(r, g, b)) = style.fg {
-                style.fg(Color::Rgb(
-                    ((r as f32) * 0.4).floor() as u8,
-                    ((g as f32) * 0.4).floor() as u8,
-                    ((b as f32) * 0.4).floor() as u8,
-                ))
-            } else {
-                style.fg(Color::Gray)
+    Box::new(
+        move |line: usize, _selected: bool, first_visual_line: bool, out: &mut String| {
+            if !first_visual_line {
+                return None;
             }
-        };
+            let breakpoint = breakpoints
+                .iter()
+                .find(|breakpoint| breakpoint.line == line)?;
 
-        let sym = if breakpoint.verified { "▲" } else { "⊚" };
-        write!(out, "{}", sym).unwrap();
-        Some(style)
-    })
+            let mut style = if breakpoint.condition.is_some() && breakpoint.log_message.is_some() {
+                error.underline_style(UnderlineStyle::Line)
+            } else if breakpoint.condition.is_some() {
+                error
+            } else if breakpoint.log_message.is_some() {
+                info
+            } else {
+                warning
+            };
+
+            if !breakpoint.verified {
+                // Faded colors
+                style = if let Some(Color::Rgb(r, g, b)) = style.fg {
+                    style.fg(Color::Rgb(
+                        ((r as f32) * 0.4).floor() as u8,
+                        ((g as f32) * 0.4).floor() as u8,
+                        ((b as f32) * 0.4).floor() as u8,
+                    ))
+                } else {
+                    style.fg(Color::Gray)
+                }
+            };
+
+            let sym = if breakpoint.verified { "▲" } else { "⊚" };
+            write!(out, "{}", sym).unwrap();
+            Some(style)
+        },
+    )
 }
 
 pub fn diagnostics_or_breakpoints<'doc>(
@@ -284,18 +307,22 @@ pub fn diagnostics_or_breakpoints<'doc>(
     let mut diagnostics = diagnostic(editor, doc, view, theme, is_focused);
     let mut breakpoints = breakpoints(editor, doc, view, theme, is_focused);
 
-    Box::new(move |line, selected, out| {
-        breakpoints(line, selected, out).or_else(|| diagnostics(line, selected, out))
+    Box::new(move |line, selected, first_visual_line: bool, out| {
+        breakpoints(line, selected, first_visual_line, out)
+            .or_else(|| diagnostics(line, selected, first_visual_line, out))
     })
 }
 
 #[cfg(test)]
 mod tests {
+    use std::sync::Arc;
+
     use super::*;
     use crate::document::Document;
-    use crate::editor::{GutterConfig, GutterLineNumbersConfig};
+    use crate::editor::{Config, GutterConfig, GutterLineNumbersConfig};
     use crate::graphics::Rect;
     use crate::DocumentId;
+    use arc_swap::ArcSwap;
     use helix_core::Rope;
 
     #[test]
@@ -304,7 +331,11 @@ mod tests {
         view.area = Rect::new(40, 40, 40, 40);
 
         let rope = Rope::from_str("abc\n\tdef");
-        let doc = Document::from(rope, None);
+        let doc = Document::from(
+            rope,
+            None,
+            Arc::new(ArcSwap::new(Arc::new(Config::default()))),
+        );
 
         assert_eq!(view.gutters.layout.len(), 5);
         assert_eq!(view.gutters.layout[0].width(&view, &doc), 1);
@@ -325,7 +356,11 @@ mod tests {
         view.area = Rect::new(40, 40, 40, 40);
 
         let rope = Rope::from_str("abc\n\tdef");
-        let doc = Document::from(rope, None);
+        let doc = Document::from(
+            rope,
+            None,
+            Arc::new(ArcSwap::new(Arc::new(Config::default()))),
+        );
 
         assert_eq!(view.gutters.layout.len(), 1);
         assert_eq!(view.gutters.layout[0].width(&view, &doc), 1);
@@ -339,7 +374,11 @@ mod tests {
         view.area = Rect::new(40, 40, 40, 40);
 
         let rope = Rope::from_str("abc\n\tdef");
-        let doc = Document::from(rope, None);
+        let doc = Document::from(
+            rope,
+            None,
+            Arc::new(ArcSwap::new(Arc::new(Config::default()))),
+        );
 
         assert_eq!(view.gutters.layout.len(), 2);
         assert_eq!(view.gutters.layout[0].width(&view, &doc), 1);
@@ -357,10 +396,18 @@ mod tests {
         view.area = Rect::new(40, 40, 40, 40);
 
         let rope = Rope::from_str("a\nb");
-        let doc_short = Document::from(rope, None);
+        let doc_short = Document::from(
+            rope,
+            None,
+            Arc::new(ArcSwap::new(Arc::new(Config::default()))),
+        );
 
         let rope = Rope::from_str("a\nb\nc\nd\ne\nf\ng\nh\ni\nj\nk\nl\nm\nn\no\np");
-        let doc_long = Document::from(rope, None);
+        let doc_long = Document::from(
+            rope,
+            None,
+            Arc::new(ArcSwap::new(Arc::new(Config::default()))),
+        );
 
         assert_eq!(view.gutters.layout.len(), 2);
         assert_eq!(view.gutters.layout[1].width(&view, &doc_short), 1);

helix-view/src/lib.rs

@@ -49,13 +49,10 @@ pub enum Align {
 }
 
 pub fn align_view(doc: &Document, view: &mut View, align: Align) {
-    let pos = doc
-        .selection(view.id)
-        .primary()
-        .cursor(doc.text().slice(..));
-    let line = doc.text().char_to_line(pos);
-
-    let last_line_height = view.inner_height().saturating_sub(1);
+    let doc_text = doc.text().slice(..);
+    let cursor = doc.selection(view.id).primary().cursor(doc_text);
+    let viewport = view.inner_area(doc);
+    let last_line_height = viewport.height.saturating_sub(1);
 
     let relative = match align {
         Align::Center => last_line_height / 2,
@@ -63,10 +60,20 @@ pub fn align_view(doc: &Document, view: &mut View, align: Align) {
         Align::Bottom => last_line_height,
     };
 
-    view.offset.row = line.saturating_sub(relative);
+    let text_fmt = doc.text_format(viewport.width, None);
+    let annotations = view.text_annotations(doc, None);
+    (view.offset.anchor, view.offset.vertical_offset) = char_idx_at_visual_offset(
+        doc_text,
+        cursor,
+        -(relative as isize),
+        0,
+        &text_fmt,
+        &annotations,
+    );
 }
 
 pub use document::Document;
 pub use editor::Editor;
+use helix_core::char_idx_at_visual_offset;
 pub use theme::Theme;
 pub use view::View;

helix-view/src/theme.rs

@@ -314,10 +314,23 @@ impl Theme {
         &self.scopes
     }
 
-    pub fn find_scope_index(&self, scope: &str) -> Option<usize> {
+    pub fn find_scope_index_exact(&self, scope: &str) -> Option<usize> {
         self.scopes().iter().position(|s| s == scope)
     }
 
+    pub fn find_scope_index(&self, mut scope: &str) -> Option<usize> {
+        loop {
+            if let Some(highlight) = self.find_scope_index_exact(scope) {
+                return Some(highlight);
+            }
+            if let Some(new_end) = scope.rfind('.') {
+                scope = &scope[..new_end];
+            } else {
+                return None;
+            }
+        }
+    }
+
     pub fn is_16_color(&self) -> bool {
         self.styles.iter().all(|(_, style)| {
             [style.fg, style.bg]

helix-view/src/view.rs

@@ -2,11 +2,13 @@ use crate::{
     align_view,
     editor::{GutterConfig, GutterType},
     graphics::Rect,
-    Align, Document, DocumentId, ViewId,
+    Align, Document, DocumentId, Theme, ViewId,
 };
 
 use helix_core::{
-    pos_at_visual_coords, visual_coords_at_pos, Position, RopeSlice, Selection, Transaction,
+    char_idx_at_visual_offset, doc_formatter::TextFormat, text_annotations::TextAnnotations,
+    visual_offset_from_anchor, visual_offset_from_block, Position, RopeSlice, Selection,
+    Transaction,
 };
 
 use std::{
@@ -93,10 +95,17 @@ impl JumpList {
     }
 }
 
+#[derive(Clone, Debug, PartialEq, Eq, Copy, Default)]
+pub struct ViewPosition {
+    pub anchor: usize,
+    pub horizontal_offset: usize,
+    pub vertical_offset: usize,
+}
+
 #[derive(Clone)]
 pub struct View {
     pub id: ViewId,
-    pub offset: Position,
+    pub offset: ViewPosition,
     pub area: Rect,
     pub doc: DocumentId,
     pub jumps: JumpList,
@@ -133,7 +142,11 @@ impl View {
         Self {
             id: ViewId::default(),
             doc,
-            offset: Position::new(0, 0),
+            offset: ViewPosition {
+                anchor: 0,
+                horizontal_offset: 0,
+                vertical_offset: 0,
+            },
             area: Rect::default(), // will get calculated upon inserting into tree
             jumps: JumpList::new((doc, Selection::point(0))), // TODO: use actual sel
             docs_access_history: Vec::new(),
@@ -159,6 +172,10 @@ impl View {
         self.area.clip_bottom(1).height.into() // -1 for statusline
     }
 
+    pub fn inner_width(&self, doc: &Document) -> u16 {
+        self.area.clip_left(self.gutter_offset(doc)).width
+    }
+
     pub fn gutters(&self) -> &[GutterType] {
         &self.gutters.layout
     }
@@ -176,84 +193,120 @@ impl View {
         &self,
         doc: &Document,
         scrolloff: usize,
-    ) -> Option<(usize, usize)> {
-        self.offset_coords_to_in_view_center(doc, scrolloff, false)
+    ) -> Option<ViewPosition> {
+        self.offset_coords_to_in_view_center::<false>(doc, scrolloff)
     }
 
-    pub fn offset_coords_to_in_view_center(
+    pub fn offset_coords_to_in_view_center<const CENTERING: bool>(
         &self,
         doc: &Document,
         scrolloff: usize,
-        centering: bool,
-    ) -> Option<(usize, usize)> {
-        let cursor = doc
-            .selection(self.id)
-            .primary()
-            .cursor(doc.text().slice(..));
+    ) -> Option<ViewPosition> {
+        let doc_text = doc.text().slice(..);
+        let viewport = self.inner_area(doc);
+        let vertical_viewport_end = self.offset.vertical_offset + viewport.height as usize;
+        let text_fmt = doc.text_format(viewport.width, None);
+        let annotations = self.text_annotations(doc, None);
 
-        let Position { col, row: line } =
-            visual_coords_at_pos(doc.text().slice(..), cursor, doc.tab_width());
+        // - 1 so we have at least one gap in the middle.
+        // a height of 6 with padding of 3 on each side will keep shifting the view back and forth
+        // as we type
+        let scrolloff = scrolloff.min(viewport.height.saturating_sub(1) as usize / 2);
 
-        let inner_area = self.inner_area(doc);
-        let last_line = (self.offset.row + inner_area.height as usize).saturating_sub(1);
-        let last_col = self.offset.col + inner_area.width.saturating_sub(1) as usize;
+        let cursor = doc.selection(self.id).primary().cursor(doc_text);
+        let mut offset = self.offset;
 
-        let new_offset = |scrolloff: usize| {
-            // - 1 so we have at least one gap in the middle.
-            // a height of 6 with padding of 3 on each side will keep shifting the view back and forth
-            // as we type
-            let scrolloff = scrolloff.min(inner_area.height.saturating_sub(1) as usize / 2);
-
-            let row = if line > last_line.saturating_sub(scrolloff) {
-                // scroll down
-                self.offset.row + line - (last_line.saturating_sub(scrolloff))
-            } else if line < self.offset.row + scrolloff {
-                // scroll up
-                line.saturating_sub(scrolloff)
-            } else {
-                self.offset.row
-            };
-
-            let col = if col > last_col.saturating_sub(scrolloff) {
-                // scroll right
-                self.offset.col + col - (last_col.saturating_sub(scrolloff))
-            } else if col < self.offset.col + scrolloff {
-                // scroll left
-                col.saturating_sub(scrolloff)
-            } else {
-                self.offset.col
-            };
-            (row, col)
-        };
-        let current_offset = (self.offset.row, self.offset.col);
-        if centering {
-            // return None if cursor is out of view
-            let offset = new_offset(0);
-            (offset == current_offset).then(|| {
-                if scrolloff == 0 {
-                    offset
-                } else {
-                    new_offset(scrolloff)
-                }
-            })
+        let (visual_off, mut at_top) = if cursor >= offset.anchor {
+            let off = visual_offset_from_anchor(
+                doc_text,
+                offset.anchor,
+                cursor,
+                &text_fmt,
+                &annotations,
+                vertical_viewport_end,
+            );
+            (off, false)
+        } else if CENTERING {
+            // cursor out of view
+            return None;
         } else {
-            // return None if cursor is in (view - scrolloff)
-            let offset = new_offset(scrolloff);
-            (offset != current_offset).then(|| offset) // TODO: use 'then_some' when 1.62 <= MSRV
+            (None, true)
+        };
+
+        let new_anchor = match visual_off {
+            Some((visual_pos, _)) if visual_pos.row < scrolloff + offset.vertical_offset => {
+                if CENTERING && visual_pos.row < offset.vertical_offset {
+                    // cursor out of view
+                    return None;
+                }
+                at_top = true;
+                true
+            }
+            Some((visual_pos, _)) if visual_pos.row >= vertical_viewport_end - scrolloff => {
+                if CENTERING && visual_pos.row >= vertical_viewport_end as usize {
+                    // cursor out of view
+                    return None;
+                }
+                true
+            }
+            Some(_) => false,
+            None => true,
+        };
+
+        if new_anchor {
+            let v_off = if at_top {
+                scrolloff as isize
+            } else {
+                viewport.height as isize - scrolloff as isize
+            };
+            (offset.anchor, offset.vertical_offset) =
+                char_idx_at_visual_offset(doc_text, cursor, -v_off, 0, &text_fmt, &annotations);
         }
+
+        if text_fmt.soft_wrap {
+            offset.horizontal_offset = 0;
+        } else {
+            // determine the current visual column of the text
+            let col = visual_off
+                .unwrap_or_else(|| {
+                    visual_offset_from_block(
+                        doc_text,
+                        offset.anchor,
+                        cursor,
+                        &text_fmt,
+                        &annotations,
+                    )
+                })
+                .0
+                .col;
+
+            let last_col = offset.horizontal_offset + viewport.width.saturating_sub(1) as usize;
+            if col > last_col.saturating_sub(scrolloff) {
+                // scroll right
+                offset.horizontal_offset += col - (last_col.saturating_sub(scrolloff))
+            } else if col < offset.horizontal_offset + scrolloff {
+                // scroll left
+                offset.horizontal_offset = col.saturating_sub(scrolloff)
+            };
+        }
+
+        // if we are not centering return None if view position is unchanged
+        if !CENTERING && offset == self.offset {
+            return None;
+        }
+
+        Some(offset)
     }
 
     pub fn ensure_cursor_in_view(&mut self, doc: &Document, scrolloff: usize) {
-        if let Some((row, col)) = self.offset_coords_to_in_view_center(doc, scrolloff, false) {
-            self.offset.row = row;
-            self.offset.col = col;
+        if let Some(offset) = self.offset_coords_to_in_view_center::<false>(doc, scrolloff) {
+            self.offset = offset;
         }
     }
 
     pub fn ensure_cursor_in_view_center(&mut self, doc: &Document, scrolloff: usize) {
-        if let Some((row, col)) = self.offset_coords_to_in_view_center(doc, scrolloff, true) {
-            self.offset.row = row;
-            self.offset.col = col;
+        if let Some(offset) = self.offset_coords_to_in_view_center::<true>(doc, scrolloff) {
+            self.offset = offset;
         } else {
             align_view(doc, self, Align::Center);
         }
@@ -263,14 +316,51 @@ impl View {
         self.offset_coords_to_in_view(doc, scrolloff).is_none()
     }
 
-    /// Calculates the last visible line on screen
+    /// Estimates the last visible document line on screen.
+    /// This estimate is an upper bound obtained by calculating the first
+    /// visible line and adding the viewport height.
+    /// The actual last visible line may be smaller if softwrapping occurs
+    /// or virtual text lines are visible
     #[inline]
-    pub fn last_line(&self, doc: &Document) -> usize {
-        std::cmp::min(
-            // Saturating subs to make it inclusive zero indexing.
-            (self.offset.row + self.inner_height()).saturating_sub(1),
-            doc.text().len_lines().saturating_sub(1),
-        )
+    pub fn estimate_last_doc_line(&self, doc: &Document) -> usize {
+        let doc_text = doc.text().slice(..);
+        let line = doc_text.char_to_line(self.offset.anchor.min(doc_text.len_chars()));
+        // Saturating subs to make it inclusive zero indexing.
+        (line + self.inner_height())
+            .min(doc_text.len_lines())
+            .saturating_sub(1)
+    }
+
+    /// Calculates the last non-empty visual line on screen
+    #[inline]
+    pub fn last_visual_line(&self, doc: &Document) -> usize {
+        let doc_text = doc.text().slice(..);
+        let viewport = self.inner_area(doc);
+        let text_fmt = doc.text_format(viewport.width, None);
+        let annotations = self.text_annotations(doc, None);
+
+        // last visual line in view is trivial to compute
+        let visual_height = self.offset.vertical_offset + viewport.height as usize;
+
+        // fast path when the EOF is not visible on the screen,
+        if self.estimate_last_doc_line(doc) < doc_text.len_lines() - 1 {
+            return visual_height.saturating_sub(1);
+        }
+
+        // translate to document line
+        let pos = visual_offset_from_anchor(
+            doc_text,
+            self.offset.anchor,
+            usize::MAX,
+            &text_fmt,
+            &annotations,
+            visual_height,
+        );
+
+        match pos {
+            Some((Position { row, .. }, _)) => row.saturating_sub(self.offset.vertical_offset),
+            None => visual_height.saturating_sub(1),
+        }
     }
 
     /// Translates a document position to an absolute position in the terminal.
@@ -282,22 +372,39 @@ impl View {
         text: RopeSlice,
         pos: usize,
     ) -> Option<Position> {
-        let line = text.char_to_line(pos);
-
-        if line < self.offset.row || line > self.last_line(doc) {
+        if pos < self.offset.anchor {
             // Line is not visible on screen
             return None;
         }
 
-        let tab_width = doc.tab_width();
-        // TODO: visual_coords_at_pos also does char_to_line which we ignore, can we reuse the call?
-        let Position { col, .. } = visual_coords_at_pos(text, pos, tab_width);
+        let viewport = self.inner_area(doc);
+        let text_fmt = doc.text_format(viewport.width, None);
+        let annotations = self.text_annotations(doc, None);
 
-        // It is possible for underflow to occur if the buffer length is larger than the terminal width.
-        let row = line.saturating_sub(self.offset.row);
-        let col = col.saturating_sub(self.offset.col);
+        let mut pos = visual_offset_from_anchor(
+            text,
+            self.offset.anchor,
+            pos,
+            &text_fmt,
+            &annotations,
+            viewport.height as usize,
+        )?
+        .0;
+        if pos.row < self.offset.vertical_offset {
+            return None;
+        }
+        pos.row -= self.offset.vertical_offset;
+        if pos.row >= viewport.height as usize {
+            return None;
+        }
+        pos.col = pos.col.saturating_sub(self.offset.horizontal_offset);
 
-        Some(Position::new(row, col))
+        Some(pos)
+    }
+
+    pub fn text_annotations(&self, doc: &Document, theme: Option<&Theme>) -> TextAnnotations {
+        // TODO custom annotations for custom views like side by side diffs
+        doc.text_annotations(theme)
     }
 
     pub fn text_pos_at_screen_coords(
@@ -305,9 +412,10 @@ impl View {
         doc: &Document,
         row: u16,
         column: u16,
-        tab_width: usize,
+        fmt: TextFormat,
+        annotations: &TextAnnotations,
+        ignore_virtual_text: bool,
     ) -> Option<usize> {
-        let text = doc.text().slice(..);
         let inner = self.inner_area(doc);
         // 1 for status
         if row < inner.top() || row >= inner.bottom() {
@@ -318,27 +426,80 @@ impl View {
             return None;
         }
 
-        let text_row = (row - inner.y) as usize + self.offset.row;
-        if text_row > text.len_lines() - 1 {
-            return Some(text.len_chars());
-        }
+        self.text_pos_at_visual_coords(
+            doc,
+            row - inner.y,
+            column - inner.x,
+            fmt,
+            annotations,
+            ignore_virtual_text,
+        )
+    }
 
-        let text_col = (column - inner.x) as usize + self.offset.col;
+    pub fn text_pos_at_visual_coords(
+        &self,
+        doc: &Document,
+        row: u16,
+        column: u16,
+        text_fmt: TextFormat,
+        annotations: &TextAnnotations,
+        ignore_virtual_text: bool,
+    ) -> Option<usize> {
+        let text = doc.text().slice(..);
 
-        Some(pos_at_visual_coords(
+        let text_row = row as usize + self.offset.vertical_offset;
+        let text_col = column as usize + self.offset.horizontal_offset;
+
+        let (char_idx, virt_lines) = char_idx_at_visual_offset(
             text,
-            Position {
-                row: text_row,
-                col: text_col,
-            },
-            tab_width,
-        ))
+            self.offset.anchor,
+            text_row as isize,
+            text_col,
+            &text_fmt,
+            annotations,
+        );
+
+        // if the cursor is on a line with only virtual text return None
+        if virt_lines != 0 && ignore_virtual_text {
+            return None;
+        }
+        Some(char_idx)
     }
 
     /// Translates a screen position to position in the text document.
     /// Returns a usize typed position in bounds of the text if found in this view, None if out of view.
-    pub fn pos_at_screen_coords(&self, doc: &Document, row: u16, column: u16) -> Option<usize> {
-        self.text_pos_at_screen_coords(doc, row, column, doc.tab_width())
+    pub fn pos_at_screen_coords(
+        &self,
+        doc: &Document,
+        row: u16,
+        column: u16,
+        ignore_virtual_text: bool,
+    ) -> Option<usize> {
+        self.text_pos_at_screen_coords(
+            doc,
+            row,
+            column,
+            doc.text_format(self.inner_width(doc), None),
+            &self.text_annotations(doc, None),
+            ignore_virtual_text,
+        )
+    }
+
+    pub fn pos_at_visual_coords(
+        &self,
+        doc: &Document,
+        row: u16,
+        column: u16,
+        ignore_virtual_text: bool,
+    ) -> Option<usize> {
+        self.text_pos_at_visual_coords(
+            doc,
+            row,
+            column,
+            doc.text_format(self.inner_width(doc), None),
+            &self.text_annotations(doc, None),
+            ignore_virtual_text,
+        )
     }
 
     /// Translates screen coordinates into coordinates on the gutter of the view.
@@ -419,7 +580,10 @@ impl View {
 
 #[cfg(test)]
 mod tests {
+    use std::sync::Arc;
+
     use super::*;
+    use arc_swap::ArcSwap;
     use helix_core::Rope;
 
     // 1 diagnostic + 1 spacer + 3 linenr (< 1000 lines) + 1 spacer + 1 diff
@@ -429,52 +593,174 @@ mod tests {
     const DEFAULT_GUTTER_OFFSET_ONLY_DIAGNOSTICS: u16 = 3;
 
     use crate::document::Document;
-    use crate::editor::{GutterConfig, GutterLineNumbersConfig, GutterType};
+    use crate::editor::{Config, GutterConfig, GutterLineNumbersConfig, GutterType};
 
     #[test]
     fn test_text_pos_at_screen_coords() {
         let mut view = View::new(DocumentId::default(), GutterConfig::default());
         view.area = Rect::new(40, 40, 40, 40);
         let rope = Rope::from_str("abc\n\tdef");
-        let doc = Document::from(rope, None);
-
-        assert_eq!(view.text_pos_at_screen_coords(&doc, 40, 2, 4), None);
-
-        assert_eq!(view.text_pos_at_screen_coords(&doc, 40, 41, 4), None);
-
-        assert_eq!(view.text_pos_at_screen_coords(&doc, 0, 2, 4), None);
-
-        assert_eq!(view.text_pos_at_screen_coords(&doc, 0, 49, 4), None);
-
-        assert_eq!(view.text_pos_at_screen_coords(&doc, 0, 41, 4), None);
-
-        assert_eq!(view.text_pos_at_screen_coords(&doc, 40, 81, 4), None);
-
-        assert_eq!(view.text_pos_at_screen_coords(&doc, 78, 41, 4), None);
+        let doc = Document::from(
+            rope,
+            None,
+            Arc::new(ArcSwap::new(Arc::new(Config::default()))),
+        );
 
         assert_eq!(
-            view.text_pos_at_screen_coords(&doc, 40, 40 + DEFAULT_GUTTER_OFFSET + 3, 4),
+            view.text_pos_at_screen_coords(
+                &doc,
+                40,
+                2,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
+            None
+        );
+
+        assert_eq!(
+            view.text_pos_at_screen_coords(
+                &doc,
+                40,
+                41,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
+            None
+        );
+
+        assert_eq!(
+            view.text_pos_at_screen_coords(
+                &doc,
+                0,
+                2,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
+            None
+        );
+
+        assert_eq!(
+            view.text_pos_at_screen_coords(
+                &doc,
+                0,
+                49,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
+            None
+        );
+
+        assert_eq!(
+            view.text_pos_at_screen_coords(
+                &doc,
+                0,
+                41,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
+            None
+        );
+
+        assert_eq!(
+            view.text_pos_at_screen_coords(
+                &doc,
+                40,
+                81,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
+            None
+        );
+
+        assert_eq!(
+            view.text_pos_at_screen_coords(
+                &doc,
+                78,
+                41,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
+            None
+        );
+
+        assert_eq!(
+            view.text_pos_at_screen_coords(
+                &doc,
+                40,
+                40 + DEFAULT_GUTTER_OFFSET + 3,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
             Some(3)
         );
 
-        assert_eq!(view.text_pos_at_screen_coords(&doc, 40, 80, 4), Some(3));
+        assert_eq!(
+            view.text_pos_at_screen_coords(
+                &doc,
+                40,
+                80,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
+            Some(3)
+        );
 
         assert_eq!(
-            view.text_pos_at_screen_coords(&doc, 41, 40 + DEFAULT_GUTTER_OFFSET + 1, 4),
+            view.text_pos_at_screen_coords(
+                &doc,
+                41,
+                40 + DEFAULT_GUTTER_OFFSET + 1,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
             Some(4)
         );
 
         assert_eq!(
-            view.text_pos_at_screen_coords(&doc, 41, 40 + DEFAULT_GUTTER_OFFSET + 4, 4),
+            view.text_pos_at_screen_coords(
+                &doc,
+                41,
+                40 + DEFAULT_GUTTER_OFFSET + 4,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
             Some(5)
         );
 
         assert_eq!(
-            view.text_pos_at_screen_coords(&doc, 41, 40 + DEFAULT_GUTTER_OFFSET + 7, 4),
+            view.text_pos_at_screen_coords(
+                &doc,
+                41,
+                40 + DEFAULT_GUTTER_OFFSET + 7,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
             Some(8)
         );
 
-        assert_eq!(view.text_pos_at_screen_coords(&doc, 41, 80, 4), Some(8));
+        assert_eq!(
+            view.text_pos_at_screen_coords(
+                &doc,
+                41,
+                80,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
+            Some(8)
+        );
     }
 
     #[test]
@@ -488,13 +774,19 @@ mod tests {
         );
         view.area = Rect::new(40, 40, 40, 40);
         let rope = Rope::from_str("abc\n\tdef");
-        let doc = Document::from(rope, None);
+        let doc = Document::from(
+            rope,
+            None,
+            Arc::new(ArcSwap::new(Arc::new(Config::default()))),
+        );
         assert_eq!(
             view.text_pos_at_screen_coords(
                 &doc,
                 41,
                 40 + DEFAULT_GUTTER_OFFSET_ONLY_DIAGNOSTICS + 1,
-                4
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
             ),
             Some(4)
         );
@@ -511,8 +803,22 @@ mod tests {
         );
         view.area = Rect::new(40, 40, 40, 40);
         let rope = Rope::from_str("abc\n\tdef");
-        let doc = Document::from(rope, None);
-        assert_eq!(view.text_pos_at_screen_coords(&doc, 41, 40 + 1, 4), Some(4));
+        let doc = Document::from(
+            rope,
+            None,
+            Arc::new(ArcSwap::new(Arc::new(Config::default()))),
+        );
+        assert_eq!(
+            view.text_pos_at_screen_coords(
+                &doc,
+                41,
+                40 + 1,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
+            Some(4)
+        );
     }
 
     #[test]
@@ -520,34 +826,80 @@ mod tests {
         let mut view = View::new(DocumentId::default(), GutterConfig::default());
         view.area = Rect::new(40, 40, 40, 40);
         let rope = Rope::from_str("Hi! こんにちは皆さん");
-        let doc = Document::from(rope, None);
+        let doc = Document::from(
+            rope,
+            None,
+            Arc::new(ArcSwap::new(Arc::new(Config::default()))),
+        );
 
         assert_eq!(
-            view.text_pos_at_screen_coords(&doc, 40, 40 + DEFAULT_GUTTER_OFFSET, 4),
+            view.text_pos_at_screen_coords(
+                &doc,
+                40,
+                40 + DEFAULT_GUTTER_OFFSET,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
             Some(0)
         );
 
         assert_eq!(
-            view.text_pos_at_screen_coords(&doc, 40, 40 + DEFAULT_GUTTER_OFFSET + 4, 4),
+            view.text_pos_at_screen_coords(
+                &doc,
+                40,
+                40 + DEFAULT_GUTTER_OFFSET + 4,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
             Some(4)
         );
         assert_eq!(
-            view.text_pos_at_screen_coords(&doc, 40, 40 + DEFAULT_GUTTER_OFFSET + 5, 4),
+            view.text_pos_at_screen_coords(
+                &doc,
+                40,
+                40 + DEFAULT_GUTTER_OFFSET + 5,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
             Some(4)
         );
 
         assert_eq!(
-            view.text_pos_at_screen_coords(&doc, 40, 40 + DEFAULT_GUTTER_OFFSET + 6, 4),
+            view.text_pos_at_screen_coords(
+                &doc,
+                40,
+                40 + DEFAULT_GUTTER_OFFSET + 6,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
             Some(5)
         );
 
         assert_eq!(
-            view.text_pos_at_screen_coords(&doc, 40, 40 + DEFAULT_GUTTER_OFFSET + 7, 4),
+            view.text_pos_at_screen_coords(
+                &doc,
+                40,
+                40 + DEFAULT_GUTTER_OFFSET + 7,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
             Some(5)
         );
 
         assert_eq!(
-            view.text_pos_at_screen_coords(&doc, 40, 40 + DEFAULT_GUTTER_OFFSET + 8, 4),
+            view.text_pos_at_screen_coords(
+                &doc,
+                40,
+                40 + DEFAULT_GUTTER_OFFSET + 8,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
             Some(6)
         );
     }
@@ -557,30 +909,69 @@ mod tests {
         let mut view = View::new(DocumentId::default(), GutterConfig::default());
         view.area = Rect::new(40, 40, 40, 40);
         let rope = Rope::from_str("Hèl̀l̀ò world!");
-        let doc = Document::from(rope, None);
+        let doc = Document::from(
+            rope,
+            None,
+            Arc::new(ArcSwap::new(Arc::new(Config::default()))),
+        );
 
         assert_eq!(
-            view.text_pos_at_screen_coords(&doc, 40, 40 + DEFAULT_GUTTER_OFFSET, 4),
+            view.text_pos_at_screen_coords(
+                &doc,
+                40,
+                40 + DEFAULT_GUTTER_OFFSET,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
             Some(0)
         );
 
         assert_eq!(
-            view.text_pos_at_screen_coords(&doc, 40, 40 + DEFAULT_GUTTER_OFFSET + 1, 4),
+            view.text_pos_at_screen_coords(
+                &doc,
+                40,
+                40 + DEFAULT_GUTTER_OFFSET + 1,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
             Some(1)
         );
 
         assert_eq!(
-            view.text_pos_at_screen_coords(&doc, 40, 40 + DEFAULT_GUTTER_OFFSET + 2, 4),
+            view.text_pos_at_screen_coords(
+                &doc,
+                40,
+                40 + DEFAULT_GUTTER_OFFSET + 2,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
             Some(3)
         );
 
         assert_eq!(
-            view.text_pos_at_screen_coords(&doc, 40, 40 + DEFAULT_GUTTER_OFFSET + 3, 4),
+            view.text_pos_at_screen_coords(
+                &doc,
+                40,
+                40 + DEFAULT_GUTTER_OFFSET + 3,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
             Some(5)
         );
 
         assert_eq!(
-            view.text_pos_at_screen_coords(&doc, 40, 40 + DEFAULT_GUTTER_OFFSET + 4, 4),
+            view.text_pos_at_screen_coords(
+                &doc,
+                40,
+                40 + DEFAULT_GUTTER_OFFSET + 4,
+                TextFormat::default(),
+                &TextAnnotations::default(),
+                true
+            ),
             Some(7)
         );
     }