on master help sign in
Generalised to multiple runtime directories with priorities (#5411)

* Generalised to multiple runtime directories with priorities This is an implementation for #3346. Previously, one of the following runtime directories were used: 1. `$HELIX_RUNTIME` 2. sibling directory to `$CARGO_MANIFEST_DIR` 3. subdirectory of user config directory 4. subdirectory of path to helix executable The first directory provided / found to exist in this order was used as a root for all runtime file searches (grammars, themes, queries). This change lowers the priority of `$HELIX_RUNTIME` so that the user config runtime has higher priority. More significantly, all of these directories are now searched for runtime files, enabling a user to override default or system-level runtime files. If the same file name appears in multiple runtime directories, the following priority is now used: 1. sibling directory to `$CARGO_MANIFEST_DIR` 2. subdirectory of user config directory 3. `$HELIX_RUNTIME` 4. subdirectory of path to helix executable One exception to this rule is that a user can have a `themes` directory directly in the user config directory that has higher piority to `themes` directories in runtime directories. That behaviour has been preserved. As part of implementing this feature `theme::Loader` was simplified and the cycle detection logic of the theme inheritance was improved to cover more cases and to be more explicit. * Removed AsRef usage to avoid binary growth * Health displaying ;-separated runtime dirs * Changed HELIX_RUNTIME build from src instructions * Updated doc for more detail on runtime directories * Improved health symlink printing and theme cycle errors The health display of runtime symlinks now prints both ends of the link. Separate errors are given when theme file is not found and when the only theme file found would form an inheritence cycle. * Satisfied clippy on passing Path * Clarified highest priority runtime directory purpose * Further clarified multiple runtime details in book Also gave markdown headings to subsections. Fixed a error with table indentation not building table that also appears present on master. --------- Co-authored-by: Paul Scott <paul.scott@anu.edu.au> Co-authored-by: Blaž Hrastnik <blaz@mxxn.io>

commitce1fb9e64c189fb7476b4c72c6774a5bf6cbfd0f authorpaul-scott <paul-scott@users.noreply.github.com> dateFri 10 Mar 2023, 01:50 (4 years ago) parent2cf4ce2
8 files changed+197−98
book/src/install.md+27−11 helix-loader/src/grammar.rs+14−9 helix-loader/src/lib.rs+67−14 helix-term/src/application.rs+5−5 helix-term/src/commands/typed.rs+1−1 helix-term/src/health.rs+26−12 helix-term/src/ui/mod.rs+4−4 helix-view/src/theme.rs+53−42
book/src/install.md+27−11
@@ -137,8 +137,8 @@ cargo install --path helix-term --locked
137137```
138138
139139This command will create the `hx` executable and construct the tree-sitter
140−grammars either in the `runtime` folder, or in the folder specified in `HELIX_RUNTIME`
141−(as described below). To build the tree-sitter grammars requires a c++ compiler to be installed, for example `gcc-c++`.
140+grammars in the local `runtime` folder. To build the tree-sitter grammars requires
141+a c++ compiler to be installed, for example `gcc-c++`.
142142
143143> 💡 If you are using the musl-libc instead of glibc the following environment variable must be set during the build
144144> to ensure tree-sitter grammars can be loaded correctly:
@@ -149,11 +149,13 @@ grammars either in the `runtime` folder, or in the folder specified in `HELIX_RU
149149
150150> 💡 Tree-sitter grammars can be fetched and compiled if not pre-packaged. Fetch
151151> grammars with `hx --grammar fetch` (requires `git`) and compile them with
152−> `hx --grammar build` (requires a C++ compiler).
152+> `hx --grammar build` (requires a C++ compiler). This will install them in
153+> the `runtime` directory within the user's helix config directory (more
154+> [details below](#multiple-runtime-directories)).
153155
154156### Configuring Helix's runtime files
155157
156−- **Linux and macOS**
158+#### Linux and macOS
157159
158160Either set the `HELIX_RUNTIME` environment variable to point to the runtime files and add it to your `~/.bashrc` or equivalent:
159161
@@ -167,7 +169,7 @@ Or, create a symlink in `~/.config/helix` that links to the source code director
167169ln -s $PWD/runtime ~/.config/helix/runtime
168170```
169171
170−- **Windows**
172+#### Windows
171173
172174Either set the `HELIX_RUNTIME` environment variable to point to the runtime files using the Windows setting (search for
173175`Edit environment variables for your account`) or use the `setx` command in
@@ -182,13 +184,27 @@ setx HELIX_RUNTIME "%userprofile%\source\repos\helix\runtime"
182184
183185Or, create a symlink in `%appdata%\helix\` that links to the source code directory:
184186
185− | Method | Command |
186− | ---------- | -------------------------------------------------------------------------------------- |
187− | PowerShell | `New-Item -ItemType Junction -Target "runtime" -Path "$Env:AppData\helix\runtime"` |
188− | Cmd | `cd %appdata%\helix` <br/> `mklink /D runtime "%userprofile%\src\helix\runtime"` |
187+| Method | Command |
188+| ---------- | -------------------------------------------------------------------------------------- |
189+| PowerShell | `New-Item -ItemType Junction -Target "runtime" -Path "$Env:AppData\helix\runtime"` |
190+| Cmd | `cd %appdata%\helix` <br/> `mklink /D runtime "%userprofile%\src\helix\runtime"` |
189191
190− > 💡 On Windows, creating a symbolic link may require running PowerShell or
191− > Cmd as an administrator.
192+> 💡 On Windows, creating a symbolic link may require running PowerShell or
193+> Cmd as an administrator.
194+
195+#### Multiple runtime directories
196+
197+When Helix finds multiple runtime directories it will search through them for files in the
198+following order:
199+
200+1. `runtime/` sibling directory to `$CARGO_MANIFEST_DIR` directory (this is intended for
201+ developing and testing helix only).
202+2. `runtime/` subdirectory of OS-dependent helix user config directory.
203+3. `$HELIX_RUNTIME`.
204+4. `runtime/` subdirectory of path to Helix executable.
205+
206+This order also sets the priority for selecting which file will be used if multiple runtime
207+directories have files with the same name.
192208
193209### Validating the installation
194210
helix-loader/src/grammar.rs+14−9
@@ -67,8 +67,9 @@ pub fn get_language(name: &str) -> Result<Language> {
6767#[cfg(not(target_arch = "wasm32"))]
6868pub fn get_language(name: &str) -> Result<Language> {
6969 use libloading::{Library, Symbol};
70− let mut library_path = crate::runtime_dir().join("grammars").join(name);
71− library_path.set_extension(DYLIB_EXTENSION);
70+ let mut rel_library_path = PathBuf::new().join("grammars").join(name);
71+ rel_library_path.set_extension(DYLIB_EXTENSION);
72+ let library_path = crate::runtime_file(&rel_library_path);
7273
7374 let library = unsafe { Library::new(&library_path) }
7475 .with_context(|| format!("Error opening dynamic library {:?}", library_path))?;
@@ -252,7 +253,9 @@ fn fetch_grammar(grammar: GrammarConfiguration) -> Result<FetchStatus> {
252253 remote, revision, ..
253254 } = grammar.source
254255 {
255− let grammar_dir = crate::runtime_dir()
256+ let grammar_dir = crate::runtime_dirs()
257+ .first()
258+ .expect("No runtime directories provided") // guaranteed by post-condition
256259 .join("grammars")
257260 .join("sources")
258261 .join(&grammar.grammar_id);
@@ -350,7 +353,9 @@ fn build_grammar(grammar: GrammarConfiguration, target: Option<&str>) -> Result<
350353 let grammar_dir = if let GrammarSource::Local { path } = &grammar.source {
351354 PathBuf::from(&path)
352355 } else {
353− crate::runtime_dir()
356+ crate::runtime_dirs()
357+ .first()
358+ .expect("No runtime directories provided") // guaranteed by post-condition
354359 .join("grammars")
355360 .join("sources")
356361 .join(&grammar.grammar_id)
@@ -401,7 +406,10 @@ fn build_tree_sitter_library(
401406 None
402407 }
403408 };
404− let parser_lib_path = crate::runtime_dir().join("grammars");
409+ let parser_lib_path = crate::runtime_dirs()
410+ .first()
411+ .expect("No runtime directories provided") // guaranteed by post-condition
412+ .join("grammars");
405413 let mut library_path = parser_lib_path.join(&grammar.grammar_id);
406414 library_path.set_extension(DYLIB_EXTENSION);
407415
@@ -511,9 +519,6 @@ fn mtime(path: &Path) -> Result<SystemTime> {
511519/// Gives the contents of a file from a language's `runtime/queries/<lang>`
512520/// directory
513521pub fn load_runtime_file(language: &str, filename: &str) -> Result<String, std::io::Error> {
514− let path = crate::RUNTIME_DIR
515− .join("queries")
516− .join(language)
517− .join(filename);
522+ let path = crate::runtime_file(&PathBuf::new().join("queries").join(language).join(filename));
518523 std::fs::read_to_string(path)
519524}
helix-loader/src/lib.rs+67−14
@@ -2,11 +2,12 @@ pub mod config;
22pub mod grammar;
33
44use etcetera::base_strategy::{choose_base_strategy, BaseStrategy};
5−use std::path::PathBuf;
5+use std::path::{Path, PathBuf};
66
77pub const VERSION_AND_GIT_HASH: &str = env!("VERSION_AND_GIT_HASH");
88
9−pub static RUNTIME_DIR: once_cell::sync::Lazy<PathBuf> = once_cell::sync::Lazy::new(runtime_dir);
9+static RUNTIME_DIRS: once_cell::sync::Lazy<Vec<PathBuf>> =
10+ once_cell::sync::Lazy::new(prioritize_runtime_dirs);
1011
1112static CONFIG_FILE: once_cell::sync::OnceCell<PathBuf> = once_cell::sync::OnceCell::new();
1213
@@ -25,31 +26,83 @@ pub fn initialize_config_file(specified_file: Option<PathBuf>) {
2526 CONFIG_FILE.set(config_file).ok();
2627}
2728
28−pub fn runtime_dir() -> PathBuf {
29− if let Ok(dir) = std::env::var("HELIX_RUNTIME") {
30− return dir.into();
31− }
32−
29+/// A list of runtime directories from highest to lowest priority
30+///
31+/// The priority is:
32+///
33+/// 1. sibling directory to `CARGO_MANIFEST_DIR` (if environment variable is set)
34+/// 2. subdirectory of user config directory (always included)
35+/// 3. `HELIX_RUNTIME` (if environment variable is set)
36+/// 4. subdirectory of path to helix executable (always included)
37+///
38+/// Postcondition: returns at least two paths (they might not exist).
39+fn prioritize_runtime_dirs() -> Vec<PathBuf> {
40+ const RT_DIR: &str = "runtime";
41+ // Adding higher priority first
42+ let mut rt_dirs = Vec::new();
3343 if let Ok(dir) = std::env::var("CARGO_MANIFEST_DIR") {
3444 // this is the directory of the crate being run by cargo, we need the workspace path so we take the parent
3545 let path = std::path::PathBuf::from(dir).parent().unwrap().join(RT_DIR);
3646 log::debug!("runtime dir: {}", path.to_string_lossy());
37− return path;
47+ rt_dirs.push(path);
3848 }
3949
40− const RT_DIR: &str = "runtime";
41− let conf_dir = config_dir().join(RT_DIR);
42− if conf_dir.exists() {
43− return conf_dir;
50+ let conf_rt_dir = config_dir().join(RT_DIR);
51+ rt_dirs.push(conf_rt_dir);
52+
53+ if let Ok(dir) = std::env::var("HELIX_RUNTIME") {
54+ rt_dirs.push(dir.into());
4455 }
4556
4657 // fallback to location of the executable being run
4758 // canonicalize the path in case the executable is symlinked
48− std::env::current_exe()
59+ let exe_rt_dir = std::env::current_exe()
4960 .ok()
5061 .and_then(|path| std::fs::canonicalize(path).ok())
5162 .and_then(|path| path.parent().map(|path| path.to_path_buf().join(RT_DIR)))
52− .unwrap()
63+ .unwrap();
64+ rt_dirs.push(exe_rt_dir);
65+ rt_dirs
66+}
67+
68+/// Runtime directories ordered from highest to lowest priority
69+///
70+/// All directories should be checked when looking for files.
71+///
72+/// Postcondition: returns at least one path (it might not exist).
73+pub fn runtime_dirs() -> &'static [PathBuf] {
74+ &RUNTIME_DIRS
75+}
76+
77+/// Find file with path relative to runtime directory
78+///
79+/// `rel_path` should be the relative path from within the `runtime/` directory.
80+/// The valid runtime directories are searched in priority order and the first
81+/// file found to exist is returned, otherwise None.
82+fn find_runtime_file(rel_path: &Path) -> Option<PathBuf> {
83+ RUNTIME_DIRS.iter().find_map(|rt_dir| {
84+ let path = rt_dir.join(rel_path);
85+ if path.exists() {
86+ Some(path)
87+ } else {
88+ None
89+ }
90+ })
91+}
92+
93+/// Find file with path relative to runtime directory
94+///
95+/// `rel_path` should be the relative path from within the `runtime/` directory.
96+/// The valid runtime directories are searched in priority order and the first
97+/// file found to exist is returned, otherwise the path to the final attempt
98+/// that failed.
99+pub fn runtime_file(rel_path: &Path) -> PathBuf {
100+ find_runtime_file(rel_path).unwrap_or_else(|| {
101+ RUNTIME_DIRS
102+ .last()
103+ .map(|dir| dir.join(rel_path))
104+ .unwrap_or_default()
105+ })
53106}
54107
55108pub fn config_dir() -> PathBuf {
helix-term/src/application.rs+5−5
@@ -31,6 +31,7 @@ use crate::{
3131use log::{debug, error, warn};
3232use std::{
3333 io::{stdin, stdout},
34+ path::Path,
3435 sync::Arc,
3536 time::{Duration, Instant},
3637};
@@ -113,10 +114,9 @@ impl Application {
113114
114115 use helix_view::editor::Action;
115116
116− let theme_loader = std::sync::Arc::new(theme::Loader::new(
117− &helix_loader::config_dir(),
118− &helix_loader::runtime_dir(),
119− ));
117+ let mut theme_parent_dirs = vec![helix_loader::config_dir()];
118+ theme_parent_dirs.extend(helix_loader::runtime_dirs().iter().cloned());
119+ let theme_loader = std::sync::Arc::new(theme::Loader::new(&theme_parent_dirs));
120120
121121 let true_color = config.editor.true_color || crate::true_color();
122122 let theme = config
@@ -162,7 +162,7 @@ impl Application {
162162 compositor.push(editor_view);
163163
164164 if args.load_tutor {
165− let path = helix_loader::runtime_dir().join("tutor");
165+ let path = helix_loader::runtime_file(Path::new("tutor"));
166166 editor.open(&path, Action::VerticalSplit)?;
167167 // Unset path to prevent accidentally saving to the original tutor file.
168168 doc_mut!(editor).set_path(None)?;
helix-term/src/commands/typed.rs+1−1
@@ -1565,7 +1565,7 @@ fn tutor(
15651565 return Ok(());
15661566 }
15671567
1568− let path = helix_loader::runtime_dir().join("tutor");
1568+ let path = helix_loader::runtime_file(Path::new("tutor"));
15691569 cx.editor.open(&path, Action::Replace)?;
15701570 // Unset path to prevent accidentally saving to the original tutor file.
15711571 doc_mut!(cx.editor).set_path(None)?;
helix-term/src/health.rs+26−12
@@ -52,7 +52,7 @@ pub fn general() -> std::io::Result<()> {
5252 let config_file = helix_loader::config_file();
5353 let lang_file = helix_loader::lang_config_file();
5454 let log_file = helix_loader::log_file();
55− let rt_dir = helix_loader::runtime_dir();
55+ let rt_dirs = helix_loader::runtime_dirs();
5656 let clipboard_provider = get_clipboard_provider();
5757
5858 if config_file.exists() {
@@ -66,17 +66,31 @@ pub fn general() -> std::io::Result<()> {
6666 writeln!(stdout, "Language file: default")?;
6767 }
6868 writeln!(stdout, "Log file: {}", log_file.display())?;
69− writeln!(stdout, "Runtime directory: {}", rt_dir.display())?;
70−
71− if let Ok(path) = std::fs::read_link(&rt_dir) {
72− let msg = format!("Runtime directory is symlinked to {}", path.display());
73− writeln!(stdout, "{}", msg.yellow())?;
74− }
75− if !rt_dir.exists() {
76− writeln!(stdout, "{}", "Runtime directory does not exist.".red())?;
77− }
78− if rt_dir.read_dir().ok().map(|it| it.count()) == Some(0) {
79− writeln!(stdout, "{}", "Runtime directory is empty.".red())?;
69+ writeln!(
70+ stdout,
71+ "Runtime directories: {}",
72+ rt_dirs
73+ .iter()
74+ .map(|d| d.to_string_lossy())
75+ .collect::<Vec<_>>()
76+ .join(";")
77+ )?;
78+ for rt_dir in rt_dirs.iter() {
79+ if let Ok(path) = std::fs::read_link(rt_dir) {
80+ let msg = format!(
81+ "Runtime directory {} is symlinked to: {}",
82+ rt_dir.display(),
83+ path.display()
84+ );
85+ writeln!(stdout, "{}", msg.yellow())?;
86+ }
87+ if !rt_dir.exists() {
88+ let msg = format!("Runtime directory does not exist: {}", rt_dir.display());
89+ writeln!(stdout, "{}", msg.yellow())?;
90+ } else if rt_dir.read_dir().ok().map(|it| it.count()) == Some(0) {
91+ let msg = format!("Runtime directory is empty: {}", rt_dir.display());
92+ writeln!(stdout, "{}", msg.yellow())?;
93+ }
8094 }
8195 writeln!(stdout, "Clipboard provider: {}", clipboard_provider.name())?;
8296
helix-term/src/ui/mod.rs+4−4
@@ -280,10 +280,10 @@ pub mod completers {
280280 }
281281
282282 pub fn theme(_editor: &Editor, input: &str) -> Vec<Completion> {
283− let mut names = theme::Loader::read_names(&helix_loader::runtime_dir().join("themes"));
284− names.extend(theme::Loader::read_names(
285− &helix_loader::config_dir().join("themes"),
286− ));
283+ let mut names = theme::Loader::read_names(&helix_loader::config_dir().join("themes"));
284+ for rt_dir in helix_loader::runtime_dirs() {
285+ names.extend(theme::Loader::read_names(&rt_dir.join("themes")));
286+ }
287287 names.push("default".into());
288288 names.push("base16_default".into());
289289 names.sort();
helix-view/src/theme.rs+53−42
@@ -1,5 +1,5 @@
11use std::{
2− collections::HashMap,
2+ collections::{HashMap, HashSet},
33 path::{Path, PathBuf},
44 str,
55};
@@ -37,19 +37,21 @@ pub static BASE16_DEFAULT_THEME: Lazy<Theme> = Lazy::new(|| Theme {
3737
3838#[derive(Clone, Debug)]
3939pub struct Loader {
40− user_dir: PathBuf,
41− default_dir: PathBuf,
40+ /// Theme directories to search from highest to lowest priority
41+ theme_dirs: Vec<PathBuf>,
4242}
4343impl Loader {
44− /// Creates a new loader that can load themes from two directories.
45− pub fn new<P: AsRef<Path>>(user_dir: P, default_dir: P) -> Self {
44+ /// Creates a new loader that can load themes from multiple directories.
45+ ///
46+ /// The provided directories should be ordered from highest to lowest priority.
47+ /// The directories will have their "themes" subdirectory searched.
48+ pub fn new(dirs: &[PathBuf]) -> Self {
4649 Self {
47− user_dir: user_dir.as_ref().join("themes"),
48− default_dir: default_dir.as_ref().join("themes"),
50+ theme_dirs: dirs.iter().map(|p| p.join("themes")).collect(),
4951 }
5052 }
5153
52− /// Loads a theme first looking in the `user_dir` then in `default_dir`
54+ /// Loads a theme searching directories in priority order.
5355 pub fn load(&self, name: &str) -> Result<Theme> {
5456 if name == "default" {
5557 return Ok(self.default());
@@ -58,7 +60,8 @@ impl Loader {
5860 return Ok(self.base16_default());
5961 }
6062
61− let theme = self.load_theme(name, name, false).map(Theme::from)?;
63+ let mut visited_paths = HashSet::new();
64+ let theme = self.load_theme(name, &mut visited_paths).map(Theme::from)?;
6265
6366 Ok(Theme {
6467 name: name.into(),
@@ -66,16 +69,18 @@ impl Loader {
6669 })
6770 }
6871
69− // load the theme and its parent recursively and merge them
70− // `base_theme_name` is the theme from the config.toml,
71− // used to prevent some circular loading scenarios
72− fn load_theme(
73− &self,
74− name: &str,
75− base_theme_name: &str,
76− only_default_dir: bool,
77− ) -> Result<Value> {
78− let path = self.path(name, only_default_dir);
72+ /// Recursively load a theme, merging with any inherited parent themes.
73+ ///
74+ /// The paths that have been visited in the inheritance hierarchy are tracked
75+ /// to detect and avoid cycling.
76+ ///
77+ /// It is possible for one file to inherit from another file with the same name
78+ /// so long as the second file is in a themes directory with lower priority.
79+ /// However, it is not recommended that users do this as it will make tracing
80+ /// errors more difficult.
81+ fn load_theme(&self, name: &str, visited_paths: &mut HashSet<PathBuf>) -> Result<Value> {
82+ let path = self.path(name, visited_paths)?;
83+
7984 let theme_toml = self.load_toml(path)?;
8085
8186 let inherits = theme_toml.get("inherits");
@@ -92,11 +97,7 @@ impl Loader {
9297 // load default themes's toml from const.
9398 "default" => DEFAULT_THEME_DATA.clone(),
9499 "base16_default" => BASE16_DEFAULT_THEME_DATA.clone(),
95− _ => self.load_theme(
96− parent_theme_name,
97− base_theme_name,
98− base_theme_name == parent_theme_name,
99− )?,
100+ _ => self.load_theme(parent_theme_name, visited_paths)?,
100101 };
101102
102103 self.merge_themes(parent_theme_toml, theme_toml)
@@ -148,7 +149,7 @@ impl Loader {
148149 merge_toml_values(theme, palette.into(), 1)
149150 }
150151
151− // Loads the theme data as `toml::Value` first from the user_dir then in default_dir
152+ // Loads the theme data as `toml::Value`
152153 fn load_toml(&self, path: PathBuf) -> Result<Value> {
153154 let data = std::fs::read_to_string(path)?;
154155 let value = toml::from_str(&data)?;
@@ -156,25 +157,35 @@ impl Loader {
156157 Ok(value)
157158 }
158159
159− // Returns the path to the theme with the name
160− // With `only_default_dir` as false the path will first search for the user path
161− // disabled it ignores the user path and returns only the default path
162− fn path(&self, name: &str, only_default_dir: bool) -> PathBuf {
160+ /// Returns the path to the theme with the given name
161+ ///
162+ /// Ignores paths already visited and follows directory priority order.
163+ fn path(&self, name: &str, visited_paths: &mut HashSet<PathBuf>) -> Result<PathBuf> {
163164 let filename = format!("{}.toml", name);
164165
165− let user_path = self.user_dir.join(&filename);
166− if !only_default_dir && user_path.exists() {
167− user_path
168− } else {
169− self.default_dir.join(filename)
170− }
171− }
172−
173− /// Lists all theme names available in default and user directory
174− pub fn names(&self) -> Vec<String> {
175− let mut names = Self::read_names(&self.user_dir);
176− names.extend(Self::read_names(&self.default_dir));
177− names
166+ let mut cycle_found = false; // track if there was a path, but it was in a cycle
167+ self.theme_dirs
168+ .iter()
169+ .find_map(|dir| {
170+ let path = dir.join(&filename);
171+ if !path.exists() {
172+ None
173+ } else if visited_paths.contains(&path) {
174+ // Avoiding cycle, continuing to look in lower priority directories
175+ cycle_found = true;
176+ None
177+ } else {
178+ visited_paths.insert(path.clone());
179+ Some(path)
180+ }
181+ })
182+ .ok_or_else(|| {
183+ if cycle_found {
184+ anyhow!("Theme: cycle found in inheriting: {}", name)
185+ } else {
186+ anyhow!("Theme: file not found for: {}", name)
187+ }
188+ })
178189 }
179190
180191 pub fn default_theme(&self, true_color: bool) -> Theme {