Phil Hord 5513deeade Add a gzip indexing class for faster gz navigation ... The gzread function is slow. Every time you seek to a new location, the whole file up to that position has to be decompressed again. This causes massive lags when trying to do simple things in lnav on a large .gz file. Use the zlib inflate* functions instead and record the dictionary periodically while processing the file the first time. Then use inflateSetDictionary to restore the dictionary to a convenient location when trying to seek into the file again in the future. Use a default period of 1MB of compressed data for syncpoints. Each syncpoint uses 32KB. This is a ratio of 3.2%. For example, a 1GB .gz file (compressed size) will require us to keep 32MB of index data in memory. A better method may be to use a fixed number of syncpoints and divide the file appropriately. This would keep the memory bounded at the cost of slower file navigation on large .gz files. Use pread to read the data for the stream decompressor and remove the lock_hack previously employed. NB. The documentation on these zlib functions is sparse. I followed the example in zlib/examples/zran.c, but I used the z_stream total_in and total_out variables instead of keeping my own separately as zran.c does. Maybe this is incompatible with some very old zlib versions. I haven't looked.		2019-08-17 18:40:55 -07:00
..
bad-config/formats	[highlights] allow the highlight color to be specified in log formats	2017-12-29 06:51:09 -08:00
bad-config2/formats/invalid-config	[build] add fmt lib	2019-05-04 07:07:39 -07:00
bad-config-json/formats	[time-offset] add days to offset	2017-04-25 05:41:14 -07:00
formats	[log_format] ignore line-format field if timestamp-format is set	2019-07-26 01:16:37 -07:00
log-samples	[data-parser] fix some issues with parsing hierarchical data and improve the debugger	2016-04-28 23:34:13 -07:00
aftest.cc	[release] bump to 0.5.1	2013-05-02 23:02:03 -07:00
ansi-colors.0.in	[textview] cleanup the ansi scrubber	2013-06-08 07:57:40 -07:00
CMakeLists.txt	[stdin] Keep the stdin piper file in .lnav so that it can be reopened	2019-07-29 22:18:32 -07:00
datafile_ipaddr.0	[docs] more docs and some simple bug fixes	2013-08-25 07:12:41 -07:00
datafile_simple.0	[data-parser] fix some issues with parsing hierarchical data and improve the debugger	2016-04-28 23:34:13 -07:00
datafile_simple.1	[data-parser] fix some issues with parsing hierarchical data and improve the debugger	2016-04-28 23:34:13 -07:00
datafile_simple.2	[data-parser] fix some issues with parsing hierarchical data and improve the debugger	2016-04-28 23:34:13 -07:00
datafile_simple.3	[data-parser] fix some issues with parsing hierarchical data and improve the debugger	2016-04-28 23:34:13 -07:00
datafile_simple.4	[data-parser] fix some issues with parsing hierarchical data and improve the debugger	2016-04-28 23:34:13 -07:00
datafile_simple.5	[data-parser] fix some issues with parsing hierarchical data and improve the debugger	2016-04-28 23:34:13 -07:00
datafile_simple.6	[data-parser] fix some issues with parsing hierarchical data and improve the debugger	2016-04-28 23:34:13 -07:00
datafile_simple.7	[data-parser] fix some issues with parsing hierarchical data and improve the debugger	2016-04-28 23:34:13 -07:00
datafile_simple.8	[data-parser] fix some issues with parsing hierarchical data and improve the debugger	2016-04-28 23:34:13 -07:00
datafile_simple.9	[data-parser] fix some issues with parsing hierarchical data and improve the debugger	2016-04-28 23:34:13 -07:00
datafile_simple.10	[data-parser] fix some issues with parsing hierarchical data and improve the debugger	2016-04-28 23:34:13 -07:00
datafile_simple.11	[data-parser] fix some issues with parsing hierarchical data and improve the debugger	2016-04-28 23:34:13 -07:00
datafile_simple.12	[data-parser] fix some issues with parsing hierarchical data and improve the debugger	2016-04-28 23:34:13 -07:00
datafile_simple.13	[data-parser] fix some issues with parsing hierarchical data and improve the debugger	2016-04-28 23:34:13 -07:00
datafile_simple.14	[data-parser] fix some issues with parsing hierarchical data and improve the debugger	2016-04-28 23:34:13 -07:00
datafile_simple.15	[data-parser] fix some issues with parsing hierarchical data and improve the debugger	2016-04-28 23:34:13 -07:00
datafile_simple.16	[data-parser] fix some issues with parsing hierarchical data and improve the debugger	2016-04-28 23:34:13 -07:00
datafile_simple.17	[data-parser] fix some issues with parsing hierarchical data and improve the debugger	2016-04-28 23:34:13 -07:00
datafile_simple.18	[data-parser] fix some issues with parsing hierarchical data and improve the debugger	2016-04-28 23:34:13 -07:00
datafile_simple.19	[data-parser] fix some issues with parsing hierarchical data and improve the debugger	2016-04-28 23:34:13 -07:00
datafile_simple.20	[data-parser] fix some issues with parsing hierarchical data and improve the debugger	2016-04-28 23:34:13 -07:00
datafile_syslog.0	[data parser] hex dump values and add the uwsgi log format	2013-07-17 21:24:33 -07:00
datafile_syslog.1	[sqlite] Major improvements to the sqlite integration.	2013-05-24 07:55:56 -07:00
datafile_xml.0	[data-parser] fix some issues with parsing hierarchical data and improve the debugger	2016-04-28 23:34:13 -07:00
drive_data_scanner.cc	[stdin] Keep the stdin piper file in .lnav so that it can be reopened	2019-07-29 22:18:32 -07:00
drive_grep_proc.cc	Some refactoring for the line buffer and other cleanups	2019-06-15 06:32:02 -07:00
drive_line_buffer.cc	Add a gzip indexing class for faster gz navigation	2019-08-17 18:40:55 -07:00
drive_listview.cc	[sql] do some minimal parsing/annotation of SQL statements	2017-03-26 06:02:53 -07:00
drive_logfile.cc	[stdin] Keep the stdin piper file in .lnav so that it can be reopened	2019-07-29 22:18:32 -07:00
drive_mvwattrline.cc	[tests] update drive_mvwattrline	2018-04-04 08:00:25 -07:00
drive_readline_curses.cc	[pretty] highlight log messages in the pretty view as in the normal log view	2018-05-10 06:44:03 -07:00
drive_sequencer.cc	Some refactoring for the line buffer and other cleanups	2019-06-15 06:32:02 -07:00
drive_shlexer.cc	[test] Fix printf format specifier for size_t.	2017-10-18 12:52:49 -07:00
drive_sql_anno.cc	[sql] do some minimal parsing/annotation of SQL statements	2017-03-26 06:02:53 -07:00
drive_sql.cc	[build] some refactoring of files	2019-05-08 05:30:59 -07:00
drive_view_colors.cc	[build] try to fix a build issue	2018-08-18 23:11:02 -07:00
drive_vt52_curses.cc	[filter] add panel that shows the set of filters	2018-11-09 09:45:19 -08:00
expected_help.txt	[textview] pause loading of files by pressing =	2019-07-30 06:28:42 -07:00
gp_test.cc	[release] bump to 0.5.1	2013-05-02 23:02:03 -07:00
lb_test.cc	[release] bump to 0.5.1	2013-05-02 23:02:03 -07:00
listview_output.0	[ui] rework the default colors change	2017-12-02 22:11:34 -08:00
listview_output.1	[ui] rework the default colors change	2017-12-02 22:11:34 -08:00
listview_output.2	[ui] rework the default colors change	2017-12-02 22:11:34 -08:00
listview_output.3	[ui] rework the default colors change	2017-12-02 22:11:34 -08:00
listview_output.4	[ui] rework the default colors change	2017-12-02 22:11:34 -08:00
listview_output.5	[ui] rework the default colors change	2017-12-02 22:11:34 -08:00
listview_output.6	[ui] rework the default colors change	2017-12-02 22:11:34 -08:00
lnav_doctests.cc	[logfile] the index rebuild didn't check the mtime if the file size was the same	2019-07-25 07:31:46 -07:00
log.clog	[log-format] fix line-format where a variable can span lines	2016-08-23 06:34:18 -07:00
logfile_access_log.0	remove soci	2009-10-06 14:14:49 -07:00
logfile_access_log.1	[log_format] the access_log format wasn't picking up the timestamp range correctly	2013-06-22 07:00:59 -07:00
logfile_bad_access_log.0	[format] try to validate formats against a given file	2015-07-11 16:32:48 -07:00
logfile_bad_syslog.0	[format] try to validate formats against a given file	2015-07-11 16:32:48 -07:00
logfile_blued.0	[format] order the formats based on sample matches	2015-04-05 07:12:20 -07:00
logfile_bro_conn.log.0	[formats] support for CSV/TSV log formats	2017-04-23 07:11:21 -07:00
logfile_bro_http.log.0	[formats] support for CSV/TSV log formats	2017-04-23 07:11:21 -07:00
logfile_crlf.0	[line_buffer] fix crlf handling in log files	2018-10-12 07:46:18 -07:00
logfile_cxx.0	[pretty-print] check if lines are XML before indenting them as such	2018-04-02 13:25:07 -07:00
logfile_empty.0	first commit	2009-09-13 18:07:32 -07:00
logfile_epoch.0	[fmt] allow timestamp to be specified in the log format	2015-04-10 22:55:57 -07:00
logfile_epoch.1	[ptime] make sure the full time stamp is matched	2016-11-22 08:38:45 -08:00
logfile_filter.0	missing test file	2014-12-10 07:19:31 -08:00
logfile_for_join.0	[filter] some improvements to the core filtering code	2014-10-19 22:16:40 -07:00
logfile_generic_with_header.0	[logfile] fix issue when files have headers	2019-05-17 07:42:05 -07:00
logfile_generic.0	add a test for change that fixes #110	2014-10-29 20:05:33 -07:00
logfile_generic.1	[generic_log] fix log levels for generic_log	2015-05-01 20:52:00 -07:00
logfile_generic.2	[generic_log] add time format that includes zone	2015-05-17 05:15:41 -07:00
logfile_glog.0	[fmt] Fix glog_log format	2016-06-30 18:36:45 -07:00
logfile_haproxy.0	[format] allow multiple log format patterns per log file	2019-03-13 16:37:41 -07:00
logfile_journald.json	add monotonic timestamp back to the journald format, map priority 1 to fatal as well	2018-08-21 11:39:47 -07:00
logfile_json2.json	[log-format] abbreviate long fields when formatting JSON logs	2016-08-22 06:45:27 -07:00
logfile_json3.json	[format] hidden object/array fields add sub-lines when they should not	2018-11-26 10:30:27 -08:00
logfile_json.json	[json-format] fix json file recognition for crlf line-endings	2018-08-13 05:30:47 -07:00
logfile_leveltest.0	[loader] the loader was not converting the level name to the enum value correctly	2016-06-30 21:59:51 -07:00
logfile_multiline.0	[headless] report output from commands/queries that were executed	2014-03-02 16:52:18 -08:00
logfile_nested_json.json	[json-log] handle arrays in json log message fields	2019-04-17 06:17:11 -07:00
logfile_openam.0	[format] add OpenAM log formats	2014-06-17 21:29:42 -07:00
logfile_plain.0	[text] add filtering support to the plain text view	2015-03-28 06:30:30 -07:00
logfile_pretty.0	[format] add more log levels and fix some more pretty-printer glitches	2015-04-10 19:50:05 -07:00
logfile_rollover.0	[time] fix time rollover for days	2016-05-03 06:43:00 -07:00
logfile_rollover.1	[log] do a full sort if there is a year rollover	2017-04-24 07:03:46 -07:00
logfile_strace_log.0	clean up some auto conf and add an strace test	2010-01-24 12:25:34 -08:00
logfile_syslog_fr.0	[locale] some tweaks for recognizing non-english logs	2018-08-21 21:43:32 -07:00
logfile_syslog_with_access_log.0	[fmt] highlight identifiers in module formats	2015-07-19 02:40:02 -07:00
logfile_syslog_with_header.0	[logfile] fix issue when files have headers	2019-05-17 07:42:05 -07:00
logfile_syslog_with_mixed_times.0	[cmd] add support for times in goto	2015-09-14 08:56:42 -07:00
logfile_syslog.0	first commit	2009-09-13 18:07:32 -07:00
logfile_syslog.1	first commit	2009-09-13 18:07:32 -07:00
logfile_syslog.2	Some more sequence matching stuff.	2011-06-05 19:05:46 -07:00
logfile_tai64n.0	[time] add support for TAI64N timestamps	2016-04-02 20:58:20 -07:00
logfile_tcf.0	[format] add TCF file format and some fixes to make it work	2014-04-06 22:11:04 -07:00
logfile_tcf.1	[format] add TCF file format and some fixes to make it work	2014-04-06 22:11:04 -07:00
logfile_tcsh_history.0	first commit	2009-09-13 18:07:32 -07:00
logfile_uwsgi.0	[pretty-print] allow formats to do transforms before pretty-printing	2016-12-05 16:34:30 -08:00
logfile_vami.0	[test] use doc ip range	2015-04-08 20:01:20 -07:00
logfile_vdsm.0	[vdsm_log] add new format	2014-11-10 08:56:35 -08:00
logfile_with_a_really_long_name_to_test_a_bug_with_long_names.0	first commit	2009-09-13 18:07:32 -07:00
Makefile.am	[build] missed cleaning a dir up	2019-08-10 21:59:43 -07:00
multiline.lnav	[build] bump version to 0.7.3 since we will release soon	2015-04-08 02:26:25 -07:00
mvwattrline_output.0	[view_curses] fix highlighting when a line contains unicode	2018-04-04 07:41:09 -07:00
nested.lnav	[exec] add a prompt to execute lnav scripts	2015-12-18 22:39:27 -08:00
parser_debugger.py	[codacy] fix a few issues turned up	2019-07-11 14:59:54 -07:00
rltest.cc	[release] bump to 0.5.1	2013-05-02 23:02:03 -07:00
scripty.cc	[stdin] Keep the stdin piper file in .lnav so that it can be reopened	2019-07-29 22:18:32 -07:00
si_test.cc	[release] bump to 0.5.1	2013-05-02 23:02:03 -07:00
slicer.cc	Some refactoring for the line buffer and other cleanups	2019-06-15 06:32:02 -07:00
sql.0.in	[cleanup] start a tracer/debugger for the data parser	2013-06-08 06:10:18 -07:00
sql.0.out	[cleanup] start a tracer/debugger for the data parser	2013-06-08 06:10:18 -07:00
test_abbrev.cc	[log-format] abbreviate long fields when formatting JSON logs	2016-08-22 06:45:27 -07:00
test_ansi_scrubber.cc	[build] some refactoring of files	2019-05-08 05:30:59 -07:00
test_auto_fd.cc	[release] bump to 0.5.1	2013-05-02 23:02:03 -07:00
test_auto_mem.cc	[test] fix test_auto_mem test	2013-05-31 21:20:58 -07:00
test_bookmarks.cc	[cleanup] prune some includes and some other cleaning	2018-10-11 05:50:28 -07:00
test_cli.sh	[stdin] Keep the stdin piper file in .lnav so that it can be reopened	2019-07-29 22:18:32 -07:00
test_cmds.sh	[config] initial support for themes	2019-05-03 13:50:19 -07:00
test_config.sh	[view_curses] colors	2019-08-07 09:46:19 -07:00
test_curl.sh	[curl] add a curl looper to handle url requests	2015-08-02 06:43:57 -07:00
test_data_parser.sh	[pretty-print] first pass at a pretty-printer	2015-03-16 23:10:34 -07:00
test_date_time_scanner.cc	[locale] some tweaks for recognizing non-english logs	2018-08-21 21:43:32 -07:00
test_format_installer.sh	[fmt] install/update formats from git repos	2015-08-09 21:03:23 -07:00
test_format_loader.sh	[filters] add a log_filters column to log tables that contains the matched filters	2019-02-14 22:42:44 -08:00
test_grep_proc2.cc	[bookmarks] add support for tags and comments for log lines	2018-05-17 07:06:50 -07:00
test_grep_proc.sh	[headless] report output from commands/queries that were executed	2014-03-02 16:52:18 -08:00
test_json_format.sh	[json] newlines in the line format cause problems	2019-05-13 10:44:33 -07:00
test_line_buffer2.cc	[line_buffer] refactor left a bug with non-blocking files	2019-06-27 10:19:54 -07:00
test_line_buffer.sh	Add a gzip indexing class for faster gz navigation	2019-08-17 18:40:55 -07:00
test_listview.sh	Freebsd build changes	2009-12-24 10:36:01 -08:00
test_log_accel.cc	[elapsed-time] add a visual indicator for sharp changes in message rate	2014-03-16 15:07:08 -07:00
test_logfile.sh	[logfile] fix issue when files have headers	2019-05-17 07:42:05 -07:00
test_meta.sh	[config] pay attention to XDG_CONFIG_HOME	2019-08-01 07:14:47 -07:00
test_mvwattrline.sh	[view_curses] add some tests for mvwattrline	2014-03-08 05:26:49 -08:00
test_ncurses_unicode.cc	[tests] fix an issue with the last commit	2018-03-29 13:45:59 -07:00
test_pcrepp.cc	[build] some refactoring of files	2019-05-08 05:30:59 -07:00
test_pretty_print.sh	[pretty] highlight log messages in the pretty view as in the normal log view	2018-05-10 06:44:03 -07:00
test_reltime.cc	[codacy] fix a few issues turned up	2019-07-11 14:59:54 -07:00
test_scripts.sh	[cmds] add :redirect-to command for redirecting the output of commands in scripts	2018-10-12 07:12:35 -07:00
test_sessions.sh	[sql] fix some issues found while trying to query some logs	2019-03-21 07:46:51 -07:00
test_shlexer.sh	[cmd] replace wordexp with shlex	2015-12-25 12:03:44 -08:00
test_sql_coll_func.sh	[misc] some more miscellaneous bugs	2017-04-12 15:54:54 -07:00
test_sql_fs_func.sh	[help] show help text for SQL functions	2017-03-31 07:01:11 -07:00
test_sql_json_func.sh	[bookmarks] add support for tags and comments for log lines	2018-05-17 07:06:50 -07:00
test_sql_str_func.sh	[sql] do some minimal parsing/annotation of SQL statements	2017-03-26 06:02:53 -07:00
test_sql_time_func.sh	[help] show help text for SQL functions	2017-03-31 07:01:11 -07:00
test_sql.sh	[tests] fix a test	2019-03-15 22:38:10 -07:00
test_top_status.cc	[stdin] Keep the stdin piper file in .lnav so that it can be reopened	2019-07-29 22:18:32 -07:00
test_view_colors.sh	[test] some cleanup for tests and other things	2012-06-05 13:18:59 -07:00
test_vt52_curses.sh	[filter] add panel that shows the set of filters	2018-11-09 09:45:19 -08:00
textfile_json_indented.0	[text] pretty-printing should work in the text view	2015-08-18 20:58:20 -07:00
textfile_json_one_line.0	[text] pretty-printing should work in the text view	2015-08-18 20:58:20 -07:00
textfile_quoted_json.0	[pretty] pretty-print the contents of strings	2015-08-26 21:28:30 -07:00
toplevel.lnav	[cmd] replace wordexp with shlex	2015-12-25 12:03:44 -08:00
update_parser_output.sh	[data-parser] fix some issues with parsing hierarchical data and improve the debugger	2016-04-28 23:34:13 -07:00
UTF-8-test.txt	[perf] improve initial indexing times	2018-10-17 07:03:33 -07:00
view_colors_output.0	[tests] disable test_view_colors for now	2018-08-18 23:20:12 -07:00
vt52_curses_input.0	first commit	2009-09-13 18:07:32 -07:00
vt52_curses_input.1	first commit	2009-09-13 18:07:32 -07:00
vt52_curses_output.0	first commit	2009-09-13 18:07:32 -07:00
vt52_curses_output.1	first commit	2009-09-13 18:07:32 -07:00