2021-02-13 23:41:48 +03:00
|
|
|
# Architecture
|
|
|
|
|
|
|
|
This document covers the internal architecture of the Logfile Navigator (lnav),
|
|
|
|
a terminal-based tool for viewing and analyzing log files.
|
|
|
|
|
|
|
|
## Goals
|
|
|
|
|
|
|
|
The following goals drive the design and implementation of lnav:
|
|
|
|
|
|
|
|
- Don't make the user do something that can be done automatically.
|
|
|
|
|
|
|
|
Example: Automatically detect log formats for files instead of making them
|
|
|
|
specify the format for each file.
|
|
|
|
|
|
|
|
- Be performant on low-spec hardware.
|
|
|
|
|
|
|
|
Example: Prefer single-threaded optimizations over trying to parallelize
|
|
|
|
|
|
|
|
- Operations should be "live" and not block the user from continuing to work.
|
|
|
|
|
|
|
|
Example: Searches are run in the background.
|
|
|
|
|
|
|
|
- Provide context-sensitive help.
|
|
|
|
|
|
|
|
Example: When the cursor is over a SQL keyword/function, the help text for
|
|
|
|
that is shown above.
|
|
|
|
|
|
|
|
- Show a preview of operations so the user knows what is going to happen.
|
|
|
|
|
|
|
|
Example: When entering a `:filter-out` command, the matched parts of the
|
|
|
|
lines are highlighted in red.
|
|
|
|
|
|
|
|
## Overview
|
|
|
|
|
|
|
|
The whole of lnav consists of a
|
|
|
|
[log file parser](https://lnav.readthedocs.io/en/latest/formats.html),
|
|
|
|
[text UI](https://lnav.readthedocs.io/en/latest/ui.html),
|
|
|
|
[integrations with SQLite](https://lnav.readthedocs.io/en/latest/sqlext.html),
|
|
|
|
[command-line interface](https://lnav.readthedocs.io/en/latest/cli.html),
|
|
|
|
and
|
|
|
|
[commands for operating on logs](https://lnav.readthedocs.io/en/latest/commands.html).
|
|
|
|
Since the majority of lnav's operations center around logs, the core
|
|
|
|
data-structure is the combined log message index. The message index
|
|
|
|
is populated when new messages are read from log files. The text UI
|
|
|
|
displays a subset of messages from the index. The SQLite virtual-tables
|
|
|
|
allow for programmatic access to the messages and lnav's internal state.
|
|
|
|
|
|
|
|
[![lnav architecture](docs/lnav-architecture.png)](https://whimsical.com/lnav-architecture-UM594Qo4G3nt2XWaSZA1mh)
|
|
|
|
|
|
|
|
## File Monitoring
|
|
|
|
|
|
|
|
Each file being monitored by lnav has an associated [`logfile`](src/logfile.hh)
|
|
|
|
object, be they plaintext files or files with a recognized format. These
|
|
|
|
objects are periodically polled by the main event loop to check if the file
|
|
|
|
was deleted, truncated, or new lines added. While reading new lines, if no
|
|
|
|
log format has matched yet, each line will be passed through the log format
|
|
|
|
regular expressions to try and find a match. Each line that is read is added
|
|
|
|
to an index
|
|
|
|
|
2021-02-14 10:08:40 +03:00
|
|
|
#### Why is `mmap()` not used?
|
2021-02-13 23:41:48 +03:00
|
|
|
|
|
|
|
Note that file contents are consumed using `pread(2)`/`read(2)` and not
|
|
|
|
`mmap(2)` since `mmap(2)` does not react well to files changing out from
|
|
|
|
underneath it. For example, a truncated file would likely result in a
|
|
|
|
`SIGBUS`.
|
|
|
|
|
|
|
|
## Log Messages
|
|
|
|
|
|
|
|
As files are being indexed, if a matching format is found, the file is
|
|
|
|
"promoted" from a plaintext file to a log file. When the file is promoted,
|
|
|
|
it is added to the [logfile_sub_source](src/logfile_sub_source.hh), which
|
|
|
|
collates all log messages together into a single index.
|
|
|
|
|
2021-02-14 10:08:40 +03:00
|
|
|
## Log Formats
|
|
|
|
|
|
|
|
[log_format](src/log_format.hh) instances are used to parse lines from files
|
|
|
|
into `logline` objects. The majority of log formats are
|
|
|
|
[external_log_format](src/log_format_ext.hh) objects that are create from
|
|
|
|
[JSON format definitions](https://lnav.readthedocs.io/en/latest/formats.html).
|
|
|
|
The built-in definitions are located in the [formats](src/formats) directory.
|
|
|
|
Log formats that cannot be handled through a simple regular expression are
|
|
|
|
implemented in the [log_format_impls.cc](src/log_format_impls.cc) file.
|
|
|
|
|
2021-02-13 23:41:48 +03:00
|
|
|
## User Interface
|
|
|
|
|
|
|
|
[![lnav TUI](docs/lnav-tui.png)](https://whimsical.com/lnav-tui-MQjXc7Vx23BxQTHrnuNp5F)
|
|
|
|
|
|
|
|
The lnav text-user-interface is built on top of the basic drawing functionality
|
2021-02-14 10:08:40 +03:00
|
|
|
provided by [ncurses](https://invisible-island.net/ncurses/announce.html).
|
|
|
|
However, the higher-level functionality of panels, widgets, and such is not
|
|
|
|
used.
|