52 KiB
lnav v0.12.0
Features:
- Added the
:sh
command and-e
option to execute a shell command-line and display its output within lnav. The captured output will be displayed in the TEXT view. The lines from stdout and stderr are recorded separately so that the lines from stderr can be shown in the theme's "error" highlight. The time that the lines were received are also recorded internally so that the "time-offset" display (enabled by pressingShift
+T
) can be shown and the "jump to slow-down" hotkeys (s
/Shift
+S
) work. Since the line-by-line timestamps are recorded internally, they will not interfere with timestamps that are in the commands output. - Added a
:cd
command to change lnav's current directory. - Added support for automatically converting files that are
in a format not natively supported by lnav. The new
converter
section in a log format definition allows you to specify how a file type can be detected and converted. The built-in PCAP support in lnav is implemented using this mechanism. - Added a
shell_exec()
SQLite function that executes a command-line with the user's$SHELL
and returns the output. - Added support for custom URL schemes that are handled by an
lnav script. Schemes can be defined under
/tuning/url-schemes
. See the main docs for more details. - Added a
docker://
URL scheme that can be used to tail the logs for a container (e.g.docker://my-container
) or files within a container (e.g.docker://my-serv/var/log/dpkg.log
). - Added the SQLite JSON functions to the online help.
- Added
config get
andconfig blame
management CLI commands to get the current configuration and the file locations where the configuration options came from. - When piping data into lnav's stdin, the input used to
only be written to a single file without any rotation.
Now, the input is written to a directory of rotating files.
The same is true for the command-lines executed through the
new
:sh
command. The piped data can be managed using the newpiper
commands in the management CLI. - The
$LNAV_HOME_DIR
and$LNAV_WORK_DIR
environment variables are now defined inside lnav and refer to the location of the user's configuration directory and the directory where cached data is stored, respectively.
Bug Fixes:
- Binary data piped into stdin should now be treated the same as if it was in a file that was passed on the command-line.
- The
-I
option is now recognized in the management CLI (i.e. when you run lnav with the-m
flag). - Fields in the bro and w3c log formats that were hidden are now saved in the session and restored.
Interface changes:
- The DB view now uses the "alt-text" theme style to draw alternating rows instead of being hard-coded to bold. The alternation is also now done in groups of two rows instead of only a single row. Numbers are also rendered using the "number" theme style as well.
Breaking changes:
- Removed the
-w
command-line option. This option was useful when stdin was not automatically preserved. Since the data is now stored (and cleaned up) as well as being spread across multiple files, this option doesn't make sense anymore. - Removed the
-t
command-line flag. Text data fed in on stdin and captured from a:sh
execution is automatically timestamped. - Data piped into lnav is now stored in the work
directory instead of the
stdin-captures
dot-lnav directory.
lnav v0.11.2
Features:
- A "cursor" mode has been added to the main view that can be toggled by pressing CTRL-X. While in cursor mode, any operations that would normally work on the "top" line will now operate on the focused line instead.
- Added CTRL-D and CTRL-U hotkeys to move down/up by half a page.
- Added an
auto-width
flag to the elements of theline-format
array that indicates that the width of the field should automatically be determined by the observed values. - Added bunyan log format from Tobias Gruetzmacher.
- Added cloudflare log format from @minusf.
- Number fields used in a JSON log format
line-format
array now default to being right-aligned. Also, addedprefix
andsuffix
toline-format
elements so a string can optionally be prepended/appended if the value is not empty. - JSON log format detection has been improved to not rely
on matching the file name. All possible formats are
tried and the one with the most available fields for a
given
line-format
is used. For example, if the first log message has 8 fields and format A contains 5 of those fields in itsline-format
while format B only contains 2 of those fields in itsline-format
, format A will be used for the file.
Changes:
- For JSON-lines logs, line-feeds at the end of a value are automatically stripped.
Bug Fixes:
- Hidden values in JSON logs are now hidden by default.
- Text with ANSI-escapes is now filtered properly.
lnav v0.11.1
Features:
- Additional validation checks for log formats have been
added and will result in warnings. Pass
-W
on the command-line to view the warnings. The following new check have been added:- Each regex must have a corresponding sample log message that it matches.
- Each sample must be matched by only one regex.
- Added built-in support for anonymizing content. The
:write-*
commands now accept an--anonymize
option and there is ananonymize()
SQL function. The anonymization process will try to replace identifying information with random data. For example, IPv4 addresses are replaced with addresses in the 10.0.0.0/8 range. (This feature is mainly intended to help with providing information to lnav support that does not have sensitive values.) - Added
parse_url()
andunparse_url()
SQL functions for parsing URLs into a JSON object and then back again. Note that the implementation relies on libcurl which has some limitations, like not supporting all types of schemes (e.g.mailto:
). - Added the
subsecond-field
andsubsecond-units
log format properties to allow for specifying a separate field for the sub-second portion of a timestamp. - Added a keymap for Swedish keyboards.
Breaking changes:
- The
regexp_capture()
table-valued-function now returns NULL instead of an empty string for thecapture_name
column if the capture is not named.
Fixes:
- Reduce the "no patterns have a capture" error to a warning so that it doesn't block lnav from starting up.
- Some ANSI escape sequences will now be removed before testing regexes against a log message.
- If a line in a JSON-lines log file does not start with a
{
, it will now be shown as-is and will not have the JSON parse error.
Cost of Doing Business:
- Migrated from pcre to pcre2.
lnav v0.11.0
Features:
- Redesigned the top status area to allow for user-specified
messages and added a second line that displays an interactive
breadcrumb bar. The top status line now shows the clock and
the remaining area displays whatever messages are inserted
into the lnav_user_notifications table. The information that
was originally on top is now in a second line and organized
as breadcrumbs. Pressing
ENTER
will activate the breadcrumb bar and the left/right cursor keys can be used to select a particular crumb while the up/down keys can select a value to switch to. While a crumb is selected, you can also type in some text to do a fuzzy search on the possibilities or, if the crumb represents an array of values, enter the index to jump to. - The pretty-print view will now show breadcrumbs that indicate the location of the top line in the view with the prettified structure.
- Markdown files (those with a .md extension) are now rendered in the TEXT view. The breadcrumb bar at the top will also be updated depending on the section of the document that you are in and you can use it to jump to different parts of the doc.
- The
:goto
command will now accept anchor links (i.e.#section-id
) as an argument when the text file being viewed has sections. You can also specify an anchor when opening a file by appending#<link-name>
. For example,README.md#screenshot
. - Log message comments are now treated as markdown and rendered accordingly in the overlay. Multi-line comments are now supported as well.
- Metadata embedded in files can now be accessed by the
lnav_file_metadata
table. Currently, only the front-matter in Markdown files is supported. - Added an integration with regex101.com to make it easier to edit
log message regular expressions. Using the new "management CLI"
(activated by the
-m
option), a log format can be created from a regular expression entry on regex101.com and existing patterns can be edited. - In the spectrogram view, the selected value range is now shown by an overlay that includes a summary of the range and the number of values that fall in that range. There is also a detail panel at the bottom that shows the log-messages/DB-rows whose values are in that range. You can then press TAB to focus on the detail view and scroll around.
- Add initial support for pcap(3) files using tshark(1).
- SQL statement execution can now be canceled by pressing
CTRL+]
(same as canceling out of a prompt). - To make it possible to automate some operations, there is now an
lnav_events
table that is updated when internal events occur within lnav (e.g. opening a file, format is detected). You can then add SQLiteTRIGGER
s to this table that can perform a task by updating other tables. - Tags can automatically be added to messages by defining a pattern in a log format. Under a format definition, add the tag name into the "tags" object in a format definition. The "pattern" property specifies the regular expression to match against a line in a file that matches the format. If a match is found, the tag will be applied to the log message. To restrict matches to certain files, you can add a "paths" array whose object elements contain a "glob" property that will be matched against file names.
- Log messages can now be detected automatically via "watch
expressions". These are SQL expressions that are executed for
each log message. If the expressions evaluates to true, an
event is published to the
lnav_events
table that includes the message contents. - Added the
regexp_capture_into_json()
table-valued-function that is similar toregexp_capture()
, but returns a single row with a JSON value for each match instead of a row for each capture. - Added a
top_meta
column to the lnav_views table that contains metadata related to the top line in the view. - Added a
log_opid
hidden column to all log tables that contains the "operation ID" as specified in the log format. - Moved the
log_format
column from the all_logs table to a hidden column on all tables. - Add format for UniFi gateway.
- Added a
glob
property to search tables defined in log formats to constrain searches to log messages from files that have a matching log_path value. - Initial indexing of large files should be faster. Decompression and searching for line-endings are now pipelined, so they happen in a thread that is separate from the regular expression matcher.
- Writing to the clipboard now falls back to OSC 52 escape sequence if none of the clipboard commands could be detected. Your terminal software will need to support the sequence and you may need to explicitly enable it in the terminal.
- Added the
:export-session-to <path>
command that writes the current session state to a file as a list of commands/SQL statements. This script file can be executed to restore the majority of the current state. - Added the
echoln()
SQL function that behaves similarly to the:echo
command, writing its first argument to the current output. - Added
encode()
anddecode()
SQL functions for transcoding blobs or text values using one of the following algorithms: base64, hex, or uri. - In regular expressions, capture group names are now semantically
highlighted (e.g. in the capture,
(?<name>\w+)
, "name" would have a unique color). Also, operations or previews that use that regular expression will highlight the matched data with the same color. - Added an lnav_views_echo table that is a real SQLite table that you can create TRIGGERs on in order to perform actions when scrolling in a view.
- Added a
yaml_to_json()
SQL function that converts a YAML document to the equivalent JSON.
Breaking Changes:
- Formats definitions are now checked to ensure that values have a corresponding capture in at least one pattern.
- Added a 'language' column to the lnav_view_filters table that specifies the language of the 'pattern' column, either 'regex' or 'sql'.
- Timestamps that do not have a day or month are rewritten to a full timestamp like YYYY-MM-DD HH:MM:SS.
- Removed the summary overlay at the bottom of the log view that displayed things like "Error rate" and the time span. It doesn't seem like anyone used it.
- Removed the
log_msg_instance
column from the logline and search tables since it causes problems with performance. - Search tables now search for multiple matches within a message
instead of stopping at the first hit. Each additional match is
returned as a separate row. A
match_index
column has been added to capture the index of the match within the message. The table regex is also compiled with the "multiline" flag enabled so the meaning of the^
and$
metacharacters are changed to match the start/end of a line instead of the start/end of the entire message string. - Search tables defined in formats are now constrained to only match log messages that are in that log format instead of all log messages. As a benefit, the search table now includes the columns that are defined as part of the format.
- The lnav_view_filters table will treats the tuple of
(view_name, type, language, pattern) as a
UNIQUE
index and will raise a conflict error on anINSERT
. UseREPLACE INTO
instead ofINSERT INTO
to ignore conflict error. - The types of SQL values stored as local variables in scripts is now preserved when used as bound variables at a later point in the script.
Fixes:
- Toggling enabled/disabled filters when there is a SQL expression no longer causes a crash.
- Fix a crash related to long lines that are word wrapped.
- Multiple SQL statements in a SQL block of a script are now executed instead of just the first one.
- In cases where there were many colors on screen, some text would be colored incorrectly.
- The pretty-print view now handles ANSI escape sequences.
- The "overstrike" convention for doing bold and underline is now supported. (Overstrike is a character followed by a backspace and then the same character for bold or an underscore for underline.)
- The
:eval
command now works with searching (using the '/' prefix).
lnav v0.10.1
Features:
- Added
:show-only-this-file
command that hides all files except the one for the top line in the view. - The
:write-raw-to
command now accepts a--view
flag that specifies the source view for the data to write. For example, to write the results of a SQL query, you would pass--view=db
to the command. - The commands used to access the clipboard are now configured through the "tuning" section of the configuration.
- Added an
lnav_version()
SQL function that returns the current version string. - Added basic support for the logfmt file format. Currently, only files
whose lines are entirely logfmt-encoded are supported. The lines
must also contain either a field named
time
orts
that contains the timestamp. - Added the
logfmt2json()
SQL function to convert a string containing a logfmt-encoded message into a JSON object that can be operated on more easily. - Added the
gzip()
andgunzip()
SQL functions to compress values into a blob and decompress a blob into a string. Interface changes: - The xclip implementation for accessing the system clipboard now writes to the "clipboard" selection instead of the "primary" selection.
- The 'query' bookmark type and
y
/Y
hotkeys have been removed due to performance issues and the functionality is probably rarely used. Bug Fixes: - The text "send-input" would show up on some terminals instead of ignoring the escape sequence. This control sequence was only intended to be used in the test suite.
- Remote file synchronization has been optimized a bit.
- Configuration values loaded from the
~/.lnav/configs
directory are now included in the default configuration, so they won't be saved into the~/.lnav/config.json
user configuration file. - Key handling in the visual filter editor will no longer swallow certain key-presses when editing a filter.
- Scrolling performance restored in the SQL view.
- The
:redirect-to
command now works with/dev/clipboard
- The field overlay (opened by pressing 'p') now shows
log_time
for the timestamp field instead of the name defined in the format. - The search term in the bottom status bar will now update properly when switching views.
- The "Out-Of-Time-Order Message" overlay will be shown again.
- The tab for the "Files" panel will be highlighted in red if there is an issue opening a file.
- Overwritten files should be reloaded again.
- The
jget()
SQL function now returns numbers with the correct type. - The
json_contains()
SQL function now returns false if the first argument is NULL instead of NULL. - The local copies of remote files are now cleaned up after a couple days of the host not being accessed.
- The initial loading and indexing phase has been optimized.
lnav v0.10.0
Features:
- Files on remote machines can be viewed/tailed if they are accessible
via SSH. The syntax for specifying the host and path is similar to
scp. For example, to view the files in the /var/log directory on the
machine
host1.example.org
:
Note that you must be able to log into the machine without any interaction.$ lnav user@host1.example.org:/var/log
- Added the
:filter-expr
command to filter log messages based on an SQL expression. This command allows much greater control over filtering. - Added the
:mark-expr
command to mark log messages based on an SQL expression. This command makes it easier to programmatically mark log messages compared to using SQL. - Added support for archive files, like zip, and other compression formats, like xz, when compiled with libarchive. When one of these types of files is detected, they are unpacked into a temporary directory and all the files are loaded into lnav.
- Added an
xpath()
table-valued function for extracting values from strings containing XML snippets. - Added the
:prompt
command to allow for more customization of prompts. Combined with a custom keymapping, you can now open a prompt and prefill it with a given value. For example, a key could be bound to the following command to open the command prompt with:filter-in
already filled in::prompt command : 'filter-in '
- Added support for the W3C Extended Log File Format with the name
w3c_log
. Similarly to the bro log format, the header is used to determine the columns in a particular file. However, since the columns can be different between files, the SQL table only has a well-known set of columns and the remainder are accessible through JSON-objects stored in columns likecs_headers
andsc_headers
. - Added support for the S3 Access File Format.
- To jump to the first search hit above the top line in a view, you can
press
CTRL+J
instead ofENTER
in the search prompt. PressingENTER
will jump to the first hit below the current window. - Filtering, as a whole, can be now disabled/enabled without affecting
the state of individual filters. This includes text and time-filters
(i.e.
:hide-lines-before
). You can enable/disable filtering by: pressingf
in the filter editor UI; executing the:toggle-filtering
command; or by doing anUPDATE
on the "filtering" column of thelnav_views
SQLite table. - Themes can now include definitions for text highlights under:
/ui/theme-defs/<theme_name>/highlights
- Added a "grayscale" theme that isn't so colorful.
- Added the
humanize_file_size()
SQL function that converts a numeric size to a human-friendly string. - Added the
sparkline()
SQL function that returns a "sparkline" bar made out of unicode characters. It can be used with a single value or as an aggregator. - Added a
log_time_msecs
hidden column to the log tables that returns the timestamp as the number of milliseconds from the epoch. - Added an
lnav_top_file()
SQL function that can be used to get the name of the top line in the top view or NULL if the line did not come from a file. - Added a
mimetype
column to the lnav_file table that returns a guess as to the MIME type of the file contents. - Added a
content
hidden column to the lnav_file table that can be used to read the contents of the file. The contents can then be passed to functions that operate on XML/JSON data, likexpath()
orjson_tree()
. - Added an
lnav_top_view
SQL VIEW that returns the row for the top view in the lnav_views table. - The
generate_series()
SQLite extension is now included by default. One change from the standard implementation is that both the start and stop are required parameters. - Added the
;.read
SQL command for executing a plain SQL file. - Added the
-N
flag so that lnav will run without opening the default syslog file.
Interface Changes:
- When copying log lines, the file name and time offset will be included in the copy if they are enabled.
- Log messages that cannot be parsed properly will be given an "invalid" log level and the invalid portions colored yellow.
- The range_start and range_stop values of the
regexp_capture()
results now start at 1 instead of zero to match with what the other SQL string functions expect. - The
:write-cols-to
command has been renamed to:write-table-to
. - The DB view will limit the maximum column width to 120 characters.
- The
:echo
command now evaluates its message to do variable substitution. - The
:write-raw-to
command has been changed to write the original log file content of marked lines. For example, when viewing a JSON log, the JSON-Line values from the log file will be written to the output file. The:write-view-to
command has been added to perform the previous work of:write-raw-to
where the raw content of the view is written to the file.
Fixes:
- Unicode text can now be entered in prompts.
- The
replicate()
SQL function would cause a crash if the number of replications was zero. - Many internal improvements.
lnav v0.9.0
Features:
-
Added support for themes and included a few as well: default, eldar, monocai, night-owl, solarized-light, and solarized-dark. The theme can be changed using the
:config
command, like so::config /ui/theme night-owl
Consult the online documentation for defining a new theme at: https://lnav.readthedocs.io/en/latest/config.html#theme-definitions
-
Added support for custom keymaps and included the following: de, fr, uk, us. The keymap can be changed using the
:config
command, like so::config /ui/keymap uk
Consult the online documentation for defining a new keymap at: https://lnav.readthedocs.io/en/latest/config.html#keymap-definitions
-
The following JSON-Schemas have been published for the log format and configuration JSON files:
Formats should be updated to reference the schema using the
$schema
property. -
Indexing of new data in log files can now be paused by pressing
=
and unpaused by pressing it again. The bottom status bar will display 'Paused' in the right corner while paused. -
CMake is now a supported way to build.
-
When viewing data from the standard-input, a symbolic name can be used to preserve session state. The name can be changed using the
|rename-stdin
lnav script or by doing anUPDATE
to the filepath column of the lnav_file table. For example, to assign the name "journald", the following SQL statement can be executed in lnav:;UPDATE lnav_file SET filepath='journald' WHERE filepath='stdin'
-
The size of the terminal can be accessed in SQL using the
$LINES
and$COLS
variables. -
The
raise_error(msg)
SQL function has been added to make it easier to raise an error in an lnav script to stop execution and notify the user. -
Added the
json_concat()
function to make it easier to append/concatenate values onto arrays. -
Added the
:write-jsonlines-to
command that writes the result of a SQL query to a file in the JSON Lines format.
Interface Changes:
- Data piped into lnav is no longer dumped to the console after exit.
Instead, a file containing the data is left in
.lnav/stdin-captures/
and a message is printed to the console indicating the file name. - In time-offset mode, the deltas for messages before the first mark are now negative instead of relative to the start of the log.
- The $XDG_CONFIG_HOME environment variable (or
~/.config
directory) are now respected for storing lnav's configuration. If you have an existing~/.lnav
directory, that will continue to be used until you move it to$XDG_CONFIG_HOME/lnav
or~/.config/lnav
. - Removed the
:save-config
command. Changes to the configuration are now immediately saved.
Fixes:
- Added 'notice' log level.
- If a
timestamp-format
is used in an element of aline-format
, the field name is ignored and a formatted timestamp is always used. - Ignore stdin when it is connected to
/dev/null
.
lnav v0.8.5
Features:
- Added a visual filter editor to make it easier to update existing
filters. The editor can be opened by pressing
TAB
. Once the editor is opened, you can create/delete, enable/disable, and edit the patterns with hotkeys. - Added an
lnav_view_filters
SQL table that can be used to programmatically manipulate filters. - Added an
lnav_view_filter_stats
SQL table that contains the number of times a given filter matched a line in the view. - Added a
log_filters
column to log tables that can be used to see what filters matched the log message. - A history of locations in a view is now kept so that you can jump back
to where you were previously using the
{
and}
keys. The location history can also be accessed through the:prev-location
and:next-location
commands. - The
:write-*
commands will now accept/dev/clipboard
as a file name that writes to the system clipboard. - The
:write-to
and:write-raw-to
commands will now print out comments and tags attached to the lines. - Added a
:redirect-to <path>
command to redirect command output to the given file. This command is mostly useful in scripts where one might want to redirect all output from commands like:echo
and:write-to -
to a single file. - If a log file format has multiple patterns for matching log messages, each pattern is now tried to match a message in a file. Previously, only one pattern was ever used for an entire file.
- Added haproxy log format from Peter Hoffmann.
- Added
spooky_hash()
andgroup_spooky_hash()
SQL functions to generate a hash of their parameters. - Added
time_offset
to thelnav_file
table so that the timestamps in a file can be adjusted programmatically.
Interface Changes:
- The auto-complete behavior in the prompt has been modified to fall back
to a fuzzy search if the prefix search finds no matches. For example,
typing in
:fin
and pressing TAB would previously not do anything. Now, the:fin
will be completed to:filter-in
since that is a strong fuzzy match. If there are multiple matches, as would happen with:dfil
, readline's menu-complete behavior will be engaged and you can pressTAB
cycle through the options. - Added
CTRL+F
to toggle the enabled/disabled state of all filters for the current view. - The
-r
flag is now for recursively loading files. The functionality for loading rotated files is now under the-R
flag. - The current search term is now shown in the bottom status bar.
- Some initial help text is now shown for the search and SQL prompts to refresh the memory.
- When entering the
:comment
command for a line with a comment, the command prompt will be filled in with the existing comment to make editing easier. - Hidden fields now show up as a unicode vertical ellipsis (⋮) instead of three-dot ellipsis to save space.
- Pressing 7/8 will now move to the previous/next minute.
- The
:write-raw-to
command has been changed to write the entire contents of the current view and a:write-screen-to
command has been added to write only the current screen contents. - Disabled filters are now saved in sessions.
- The
:adjust-log-time
command now accepts relative times as input.
Fixes:
- The
:write-json-to
command will now pass through JSON cells as their JSON values instead of a JSON-encoded string.
lnav v0.8.4
Features:
- Added the
:comment
command that can be used to attach a comment to a log line. The comment will be displayed below the line, like so:
The2017-01-01T15:30:00 error: computer is on fire + This is where it all went wrong
:clear-comment
command will remove the attached comment. Comments are searchable with the standard search mechanism and they are available in SQL through thelog_comment
column. - Added the
:tag
,:untag
, and:delete-tags
commands that can be used to attach/detach tags on the top log line and delete all instances of a tag. Tags are also searchable and are available in SQL as a JSON array in thelog_tags
column. - Pressing left-arrow while viewing log messages will reveal the source file name for each line and the unique parts of the source path. Pressing again will reveal the full path.
- The file name section of the top status line will show only the unique parts of the log file path if there is not enough room to show the full path.
- Added the
:hide-unmarked-lines
and:show-unmarked-lines
commands that hide/show lines based on whether they are bookmarked. - Added the
json_contains()
SQL function to check if a JSON value contains a number of a string. - The relative time parser recognizes "next" at the beginning of the
input, for example, "next hour" or "next day". Handy for use in the
:goto
command. - Added a "text-transform" option for formatting JSON log messages. The supported options are: none, uppercase, lowercase, and capitalize.
- Added a special
__level__
field name for formatting JSON messages so that the lnav level name can be used instead of the internal value in the JSON object. - Added a log format for journald JSON logs.
Interface Changes:
- When typing in a search, instead of moving the view to the first match that was found, the first ten matches will be displayed in the preview window.
- The pretty-print view maintains highlighting from the log view.
- The pretty-print view no longer tries to reverse lookup IP addresses.
- The online help for commands and SQL functions now includes a 'See Also' section that lists related commands/functions.
Fixes:
- The HOME key should now work in the command-prompt and move the cursor to the beginning of the line.
- The
:delete-filter
command should now tab-complete existing filters. - Milliseconds can now be used in relative times (e.g. 10:00:00.123)
- The
J
/K
hotkeys were not marking lines correctly when the bottom of the view was reached. - The level field in JSON logs should now be recognized by the level patterns in the format.
lnav v0.8.3
Features:
- Support for the Bro Network Security Monitor (https://www.bro.org) log file format.
- Added an
fstat()
table-valued function for querying the local filesystem. - Added
readlink()
andrealpath()
SQL functions. - Highlights specified in log formats can now specify the colors to use for the highlighted parts of the log message.
- Added a
:quit
command. - Added a
/ui/default-colors
configuration option to specify that the terminal's default background and foreground colors should be used instead of black and white.
Interface Changes:
- Pressing delete at a command-prompt will exit the prompt if there is no other input.
Fixes:
- The help view now includes all the command-help that would pop up as you entered commands and SQL queries.
- Hidden fields and lines hidden before/after times are now saved in the current session and restored.
- Unicode characters should now be displayed correctly (make sure you have LANG set to a UTF-8 locale).
lnav v0.8.2
Features:
- The timestamp format for JSON log files can be specified with the
timestamp-format
option in theline-format
array. - Added "min-width", "max-width", "align", and "overflow" options to the "line-format" in format definitions for JSON log files. These options give you more control over how the displayed line looks.
- Added a "hidden" option to log format values so that you can hide JSON log fields from being displayed if they are not in the line format.
- Added a
rewriter
field to log format value definitions that is a command used to rewrite the field in the pretty-printed version of a log message. For example, the HTTP access log format will rewrite the status code field to include the textual version (e.g. 200 (OK)). - Log message fields can now be hidden using the
:hide-fields
command or by setting the 'hidden' property in the log format. When hidden, the fields will be replaced with a yellow ellipsis when displayed. Hiding large fields that contain extra details can make the log easier to read. Thex
hotkey can be used to quickly toggle whether these fields are displayed or not. - Added a
:mark
command to bookmark the top line in the current view. - Added an
:alt-msg
command that can be used to set the text to be displayed in the bottom right of the command line. This command is mostly intended for use by hotkey maps to set the help text. - In lnav scripts, the first row of a SQL query result will now be turned
into local variables that can be referenced in other commands or
queries. For example, the following script will print the number one:
;SELECT 1 as foobar :eval :echo ${foobar}
- Added an
lnav_view_stack
SQL table that gives access to the view stack. - Added a
top_time
column to the lnav_views table so that you can get the timestamp for the top line in views that are time-based as well as allowing you to move the view to a given time with an UPDATE statement. - Added a 'search' column to the lnav_views table so that you can perform a text search programmatically.
- Added a
regexp_capture(<string>, <pattern>)
table-valued function for getting detailed results from matching a regular expression against a string. - Added a
timediff(<time1>, <time2>)
SQL function for computing the difference between two relative or absolute timestamps. - Log formats can now define a default set of highlights with the "highlights" property.
- Added a
|search-for <pattern>
built-in script that can be used to start a search from the command-line. - Log format definitions can now specify the expected log level for a sample line. This check should make it easier to validate the definition.
Interface Changes:
- Command and SQL documentation is now displayed in a section at the
bottom of the screen when a command or query is being entered. Some
commands will also display a preview of the command results. For
example, the
:open
command will display the first ten lines of the file to be opened and the:filter-out
command will highlight text that matches in the current view. The preview pane can be shown/hidden by pressingCTRL-P
. - The color used for text colored via
:highlight
is now based on the the regex instead of randomly picked so that colors are consistent across invocations. - The "graph" view has been removed since it's functionality has been
obsoleted by other features, like
:create-search-table
. - When doing a search, if a hit is found within a second after hitting
<ENTER>
, the view will move to the matched line. The previous behavior was to stay on the current line, which tended to be a surprise to new users. - Pressing
n
/N
to move through the next/previous search hit will now skip adjacent lines, up to the vertical size of the view. This should make scanning through clusters of hits much faster. Repeatedly pressing these keys within a short time will also accelerate scanning by moving the view at least a full page at a time.
Breaking Changes:
- The captured timestamp text in log files must fully match a known format or an error will be reported. The previous behavior was to ignore any text at the end of the line.
Fixes:
- You can now execute commands from the standard input by using a dash (
-
) with the-f
command-line argument. Reading commands from a file descriptor should also work, for example, with the following bash syntax:$ lnav -f <(echo :open the-file-to-open)
- Programming language syntax highlighting should now only be applied to source code files instead of everywhere.
lnav v0.8.1
Features:
-
Added a spectrogram command and view that displays the values of a numeric field over time. The view works for log message fields or for database result columns.
-
Log formats can now create SQL views and execute other statements by adding
.sql
files to their format directories. The SQL scripts will be executed on startup. -
Added
json_group_object
andjson_group_array
aggregate SQL functions that collects values from a GROUP BY query into a JSON object or array, respectively. -
The SQL view will now graph values found in JSON objects/arrays in addition to the regular columns in the result.
-
Added an
regexp_match(<re>, <str>)
SQL function that can be used to extract values from a string using a regular expression. -
Added an
extract(<str>)
SQL function that extracts values using the same data discover/extraction parser used in thelogline
table. -
Added a "summary" overlay line to the bottom of the log view that displays how long ago the last message was received, along with the total number of files and the error rate over the past five minutes.
-
Pressing
V
in the DB view will now check for a column with a timestamp and move to the corresponding time in the log view. -
Added
a
/A
hotkeys to restore a view previously popped withq
/Q
. -
Added
:hide-lines-before
,:hide-lines-after
, and:show-lines-before-and-after
commands so that you can filter out log lines based on time. -
Scripts containing lnav commands/queries can now be executed using the pipe (
|
) hotkey. See the documentation for more information. -
Added an
:eval
command that can be used to execute a command or query after performing environment variable substitution. -
Added an
:echo
command that can be useful for scripts to message the user. -
The
log_part
column can now be set with an SQLUPDATE
statement. -
Added a
log_body
hidden column that returns the body of the log message. -
Added
:config
,:reset-config
, and:save-config
commands to change configuration options, reset to default, and save them for future executions. -
Added a
/ui/clock-format
configuration option that controls the time format in the top-left corner. -
Added a
/ui/dim-text
configuration option that controls the brightness of text in the UI. -
Added support for TAI64 timestamps (http://cr.yp.to/libtai/tai64.html).
-
Added a safe execution mode. If the
LNAVSECURE
environment variable is set before executing lnav, the following commands are disabled::open
:pipe-to
:pipe-line-to
:write-*-to
This makes it easier to run lnav with escalated privileges in restricted environments, without the risk of users being able to use the above mentioned commands to gain privileged access.
Interface Changes:
- The
o
/O
hotkeys have been reassigned to navigate through log messages that have a matching "opid" field. The old action of moving forward and backward by 60 minutes can be simulated by using the:goto
command with a relative time and ther
/R
hotkeys. - Log messages with timestamps that pre-date previous log messages will
have the timestamp highlighted in yellow and underlined. These out-
of-time-order messages will be assigned the time of the previous
message for sorting purposes. You can press the 'p' hotkey to examine
the 'Received Time' of the message as well as the time parsed from the
original message. A
log_actual_time
hidden field has also been added to the SQLite virtual table so you can operate on the original message time from the file. - The
A
/B
hotkeys for moving forward/backward by 10% line increments have been reassigned to[
and]
. Thea
andA
hotkeys are now used to return to the previously popped view while trying to preserve the time range. For example, after leaving the spectrogram view with 'q', you can press 'A' return to the view with the top time in the spectrogram matching the top time in the log view. - The 'Q' hotkey now pops the current view off of the stack while maintaining the top time between views.
Fixes:
- Issues with tailing JSON logs have been fixed.
- The
jget()
SQL function should now work for objects nested in arrays.
lnav v0.8.0
Features:
- Integration with "papertrailapp.com" for querying and tailing server log and syslog messages. See the Papertrail section in the online help for more details.
- Remote files can be opened when lnav is built with libcurl v7.23.0+
- SQL queries can now be done on lines that match a regular expression
using the
log_search
table or by creating custom tables with the:create-search-table
command. - Log formats that are "containers" for other log formats, like syslog, are now supported. See the online help for more information.
- Formats can be installed from git repositories using the
-i
option. A standard set of extra formats can be installed by doinglnav -i extra
. (You must have git installed for this to work.) - Added support for 'VMware vSphere Auto Deploy' log format.
- Added a 'sudo' log format.
- Added hotkeys to move left/right by a smaller increment (H/L or Shift+Left/Shift+Right).
- A color-coded bar has been added to the left side to show where messages from one file stop and messages from another file start.
- The
-C
option will now try to check any specified log files to make sure the format(s) match all of the lines. - Added an
all_logs
SQLite table that contains the message format extracted from each log line. Also added a;.msgformat
SQL command that executes a query that returns the counts for each format and the first line where the format was seen. - Added an
lnav_views
SQLite table that can be used to query and change the lnav view state. - When typing in a command, the status bar will display a short summary of the currently entered command.
- Added a
:delete-filter
command. - Added a
log_msg_instance
column to the logline and log_search tables to make it easier to join tables that are matching log messages that are ordered. - Added a
timeslice()
function to SQLite so that it is easier to group log messages by time buckets. - The
:goto
command now supports relative time values likea minute ago
,an hour later
, and many more.
Interface Changes:
- The
r
/R
hotkeys have been reassigned to navigate through the log messages by the relative time value that was last used with the:goto
command.
Fixes:
- The pretty-print view should now work for text files.
- Nested fields in JSON logs are now supported for levels, bodies, etc...
- Tab-completion should work for quoted SQL identifiers.
- 'lo-fi' mode key shortcut changed to
CTRL+L
. - 'redraw' shortcut removed. Relegated to just a command.
- Fixed lnav hang in pretty-print mode while doing a dns lookup.
- The generic log message parser used to extract data has been optimized and should be a bit faster.
lnav v0.7.3
Features:
- Add
:pipe-to
and:pipe-line-to
commands that pipe the currently marked lines or the current log message to a shell command, respectively. - Added a "pretty-print" view (P hotkey) that tries to reformat log messages so that they are easier to read.
- Added a
:redraw
command (CTRL+L hotkey) to redraw the window in case it has been corrupted. - Added a
:relative-goto
command to move the current view relative to its current position. - Experimental support for linking with jemalloc.
- The plain text view now supports filtering.
- Added
:next-mark
and:prev-mark
commands to jump to the next or previous bookmarked line (e.g. error, warning, ...) - Added a
:zoom-to
command to change the zoom level of the histogram view. - Log formats can now define their own timestamp formats with the
timestamp-format
field.
Fixes:
- Autotools scripts overhaul.
- Added a configure option to disable linking with libtinfo. The newer
versions of ncurses don't require it, however the build silently pulls
it in as a dependency, if it is available on the system. This can be
explicitly disabled using the
--disable-tinfo
option during configure. - Fixed the configure script behavior to ignore the values specified using
the CFLAGS and LDFLAGS environment variables while searching for sqlite3
when
--with-sqlite3
switch was specified without the prefix. - The configure script now recognizes libeditline symlink'ed to masquerade
as libreadline. This previously used to cause problems at compile time,
specially on OS X. If you come across this error, use the
--with-readline=prefix
switch to specify the path to the correct location of libreadline. - The order that log formats are tried against a log file is now automatically determined so that more specific formats are tested before more general ones. The order is determined on startup based on how each format matches each other formats sample lines.
- Command files (i.e. those executed via the
-f
flag) now support commands/queries that span more than one line. - Added more log levels: stats, debug2 - debug5.
lnav v0.7.2
- Added log formats for vdsm, openstack, and the vmkernel.
- Added a "lo-fi" mode (L hotkey) that dumps the displayed log lines
to the terminal without any decorations. The
:write-to
,:write-json-to
, and:write-csv-to
commands will also write their output to the terminal when passed-
as the file name. This mode can be useful for copying plain text lines to the clipboard. - (OS X) Text search strings are copied to the system's "find" clipboard. Also, when starting a new search, the current value in the "find" clipboard can be tab-completed.
lnav v0.7.1
Features:
- Added an
environ
SQL table that reflects lnav's environment variables. The table can be read and written to using SQLSELECT
,INSERT
,UPDATE
, andDELETE
statements. Setting variables can be a way to use SQL query results in lnav commands. - Added a
jget()
SQLite function that can extract fields from a JSON- encoded value. - Added log formats for the OpenAM identity provider.
- Added a
:clear-highlight
command to clear previous calls to the:highlight
command. - Fixed some performance bugs in indexing JSON log formats. Loading times should be at least five times faster.
- Filtering performance should be improved so that enabling/disabling filters should be almost instantaneous.
- The
:filter-in
,:filter-out
, and:highlight
commands now support tab-completion of text in the current view. - Add a
-i
flag that installs format files in:~/.lnav/formats/installed
lnav v0.7.0
Features:
- Add the '.schema' SQL command to open a view that displays the schema for the internal tables and any attached databases. If lnav was only executed with a SQLite database and no text files, this view will open by default.
- The scroll bar now indicates the location of errors/warnings, search hits, and bookmarks.
- The xterm title is update to reflect the file name for the top line in the view.
- Added a "headless" mode so that you can execute commands and run SQL queries from the command-line without having to do it from the curses UI.
- When doing a search or SQL query, any text that is currently being displayed can be tab-completed.
- The
-H
option was added so you can view the internal help text. - Added the 'g/G' hotkeys to move to the top/bottom of the file.
- Added a
log_mark
column to the log tables that indicates whether or not a log message is bookmarked. The field is writable, so you can bookmark lines using an SQL UPDATE query. - Added syntax-highlighting when editing SQL queries or search regexes.
- Added a
:write-json-to
command that writes the result of a SQL query to a JSON-formatted file. - The "elapsed time" column now uses red/green coloring to indicate sharp changes in the message rate.
- Added a
:set-min-log-level
command to filter out log messages that are below a given level.
Fixes:
- Performance improvements.
- Multi-line filtering has been fixed.
- A collator has been added to the log_level column in the log tables
so that you can write expressions like
log_level > 'warning'
. - The log_time datetime format now matches what is returned by
datetime('now')
so that collating works correctly. - If a search string is not valid PCRE syntax, a search is done for the exact string instead of just returning an error.
- Static-linking has been cleaned up.
- OpenSSL is no longer a requirement.
- Alpha support for Windows/cygwin.
- Environment variables can now be accessed in SQL queries using
the syntax:
$VAR_NAME
- An internal log is kept and written out on a crash.
- Partition bookmarks are now tracked separately from regular user bookmarks. You can start a partition with the 'partition-name' command and remove it with the 'clear-partition' command.
- Improved display of possible matches during tab-completion in the command-prompt. The matches are now shown in a separate view and pressing tab repeatedly will scroll through the view.
- The "open" command now does shell word expansion for file names.
- More config directory paths have been added:
/etc/lnav
,$prefix/etc/lnav
, and directories passed on the command-line with-I
.
lnav v0.6.2
Features:
- Word-wrap support.
Fixes:
- Fix some OS X Mavericks build/runtime issues.
lnav v0.6.1
Features:
- Support for JSON-encoded log files.
Fixes:
- Some minor fixes and performance improvements.
lnav v0.6.0
Features:
- Custom log formats and more builtin formats
- Automatic extraction of data from logs
- UI improvements, support for 256 color terminals
lnav v0.5.1
Features:
- Added the
-t
and-w
options which can be used to prepend a timestamp to any data piped in on stdin and to specify a file to write the contents of stdin to.
Fixes:
- Cleanup for packaging.
lnav v0.5.0
Features:
- Files can be specified on the command-line using wildcards so that new files are automatically loaded. Directories can also be passed as command-line arguments to read all of the files in the directory.
- Builds on cygwin again.
- Added the
C
hotkey to clear any existing user bookmarks. - Added experimental support for accepting input from mice.
Fixes:
- Internal cleanup.
- Copying to the clipboard on OS X is now supported.
- Many bug fixes.
lnav v0.4.0
Features:
- Files that are not recognized as containing log messages have been
broken out to a separate text files view. You can flip between the
log view and the text file view with the
t
hotkey. When viewing text files, thef
hotkey will switch between files. - Files compressed with bzip2 are recognized and decompressed on the fly.
- Added a "session" file and command for storing commands that should be executed on startup. For example, if you always want some highlighting to be done, you can add that command to the session file.
Fixes:
- Add some more log file formats for generic log files.
- Performance improvements for compressed files.
- Works on OS X now.
lnav v0.3.0
Changes:
- The hotkey for the SQL view was changed to
v
andV
from '.'.
Features:
- You can now switch between the SQL result view and the log view while
keeping the top of the views in sync with the
log_line
column.
Fixes:
- The
log_line
column is no longer included in the SQL result view's stacked bar graph. - Added a "warnings" count to the histogram view.