1
1
mirror of https://github.com/wader/fq.git synced 2024-12-24 22:05:31 +03:00
Commit Graph

1416 Commits

Author SHA1 Message Date
David McDonald
3198602d25 removed unused return type 2022-09-23 20:23:29 -05:00
David McDonald
369f4016c1 removed unneccessary type conversions 2022-09-23 20:23:29 -05:00
David McDonald
6b04f2de1c Documentation cleanup 2022-09-23 20:23:29 -05:00
David McDonald
368d183b73 Size check on nBits to save memory 2022-09-23 20:23:29 -05:00
David McDonald
a77cec921f Added documentation and tests, fixed bad date parsing 2022-09-23 20:23:23 -05:00
David McDonald
5711f29084 Code fixes from PR, still need to add tests and testdata 2022-09-23 20:21:32 -05:00
David McDonald
d784db69bc Adds support for Apple Binary Plist, version 00 2022-09-23 20:21:32 -05:00
Mattias Wadman
783b20ece5
Merge pull request #430 from wader/all-format-test-in-format
doc,help: Move help tests into each format
2022-09-22 23:14:40 +02:00
Mattias Wadman
a7a101ca67 doc,help: Nicer format help and move help tests into each format
Also add authors to avro and macho decoders

Generated with:
for i in $(go run . -r -n '_registry | [([([.files[][].name | split(".")[0]] | unique[]), (.formats[].name)] | count[] | select(.[1] == 2)[0]), (.formats[] | select(.decode_in_arg).name)] | unique[]'); do DIR=$(dirname $(find format -name $i.go)); echo "$ fq -h $i" > "$DIR/testdata/help_$i.fqtest" ; done
2022-09-22 23:07:58 +02:00
Mattias Wadman
2e3d71fdcc
Merge pull request #429 from wader/more-format-doc
doc,html,xml: Add more documentation and examples
2022-09-20 18:32:15 +02:00
Mattias Wadman
725ab1b17b doc,html,xml: Add more documentation and examples 2022-09-20 17:35:27 +02:00
Mattias Wadman
01be59e459
Merge pull request #428 from wader/ar-allow-empty-integerr
ar: Integer strings might be empty
2022-09-17 16:17:47 +02:00
Mattias Wadman
f08f44f12a ar: Integer strings might be empty 2022-09-17 15:36:34 +02:00
Mattias Wadman
ee7b9f4186
Merge pull request #426 from wader/csv-tsv-doc
csv: Add tsv and header example
2022-09-15 01:27:54 +02:00
Mattias Wadman
fe64530e11 csv: Add tsv and header example
Also fix string esacpe in fuzz options parser
2022-09-15 01:16:45 +02:00
Mattias Wadman
88009ee9a3
Merge pull request #425 from wader/wasm-doc
wasm: Add some documentation
2022-09-13 09:59:17 +02:00
Mattias Wadman
1385b5d0ba wasm: Add some documentation 2022-09-12 11:53:48 +02:00
Mattias Wadman
1a2cee94b7
Merge pull request #424 from wader/mp4-more-codec-names
mp4: More codec names (from mp4ra.org)
2022-09-12 11:42:10 +02:00
Mattias Wadman
2655ba09c5 mp4: More codec names (from mp4ra.org) 2022-09-12 11:25:43 +02:00
Mattias Wadman
ac93da62c2
Merge pull request #423 from wader/doc-formats-sections
doc: Fix format sections a bit
2022-09-11 10:08:24 +02:00
Mattias Wadman
abd19ed89b doc: Fix format sections a bit 2022-09-11 09:55:56 +02:00
Mattias Wadman
577ea7665c
Merge pull request #422 from wader/markdown
Move format documentation to markdown again
2022-09-11 00:59:49 +02:00
Mattias Wadman
87b2c6c10c help,doc: Use markdown for format documentation again
Markdown is used as is in online documentation and in cli the markdown decoder
is used to decode and the some jq code massages it into something cli friendly.

Was just too much of a mess to have doc in jq.
2022-09-11 00:53:29 +02:00
Mattias Wadman
00a50662ea markdown: Add decoder 2022-09-10 18:26:45 +02:00
Mattias Wadman
e98adfaa03
Merge pull request #421 from wader/elf-notes
elf: Basic program header notes decoding
2022-09-09 14:48:02 +02:00
Mattias Wadman
5382d46a9a elf: Basic program header notes decoding
Used in coredump to record various statees but also used in executables.

Next step would be to decode the note descriptions, like register info, file mappings etc.
Maybe a bit tricky as it is os/abi specific.
2022-09-09 14:35:35 +02:00
Mattias Wadman
2a5a53447c
Merge pull request #420 from piggynl/patch-1
test: Skip go test with -race by default
2022-09-09 09:40:38 +02:00
Piggy NL
808202fa34
test: Skip go test with -race by default 2022-09-09 15:15:18 +08:00
Mattias Wadman
ee6691728f
Merge pull request #419 from wader/bump-github-go-version-1.19.1
Update github-go-version from 1.19.0, 1.19.0, 1.19.0 to 1.19.1
2022-09-07 18:15:01 +02:00
Mattias Wadman
4096de99bd
Merge pull request #418 from wader/bump-docker-golang-1.19.1
Update docker-golang from 1.19.0 to 1.19.1
2022-09-07 18:11:40 +02:00
bump
c47c3866c4 Update github-go-version from 1.19.0, 1.19.0, 1.19.0 to 1.19.1 2022-09-07 16:03:48 +00:00
bump
84ca1010e3 Update docker-golang from 1.19.0 to 1.19.1 2022-09-07 16:03:45 +00:00
Mattias Wadman
f617f39a9a
Merge pull request #417 from wader/elf-more-robust
elf: Treat missing string tables as empty to be more robust
2022-09-06 11:40:38 +02:00
Mattias Wadman
12105d8c1d elf: Treat missing string tables as empty to be more robust
Makes it possible to decode shellcode snippets etc:

$ fq -n '"f0VMRgIBAQAAAAAAAAAAAAIAPgABAAAAeABAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAEAAOAABAAAAAAAAAAEAAAAFAAAAAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAIAAAAAAAsDxmvwYADwU=" | frombase64 | elf | d'
    |00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f|0123456789abcdef|.{}: (elf)
    |                                               |                |  error: elf: BitBufRange: failed at position 0 (read size 4294967296 seek pos 0): outside buffer
    |                                               |                |  header{}:
    |                                               |                |    ident{}:
0x00|7f 45 4c 46                                    |.ELF            |      magic: raw bits (valid)
0x00|            02                                 |    .           |      class: 64 (2)
0x00|               01                              |     .          |      data: "little_endian" (1)
0x00|                  01                           |      .         |      version: 1
0x00|                     00                        |       .        |      os_abi: "sysv" (0)
0x00|                        00                     |        .       |      abi_version: 0
0x00|                           00 00 00 00 00 00 00|         .......|      pad: raw bits (all zero)
0x10|02 00                                          |..              |    type: "exec" (0x2)
0x10|      3e 00                                    |  >.            |    machine: "x86_64" (0x3e) (AMD x86-64)
0x10|            01 00 00 00                        |    ....        |    version: 1
0x10|                        78 00 40 00 00 00 00 00|        x.@.....|    entry: 4194424
0x20|40 00 00 00 00 00 00 00                        |@.......        |    phoff: 64
0x20|                        00 00 00 00 00 00 00 00|        ........|    shoff: 0
0x30|00 00 00 00                                    |....            |    flags: 0
0x30|            40 00                              |    @.          |    ehsize: 64
0x30|                  38 00                        |      8.        |    phentsize: 56
0x30|                        01 00                  |        ..      |    phnum: 1
0x30|                              00 00            |          ..    |    shentsize: 0
0x30|                                    00 00      |            ..  |    shnum: 0
0x30|                                          00 00|              ..|    shstrndx: 0
    |                                               |                |  program_headers[0:1]:
    |                                               |                |    [0]{}: program_header
0x40|01 00 00 00                                    |....            |      type: "load" (1) (Loadable segment)
    |                                               |                |      flags{}:
0x40|            05                                 |    .           |        unused0: 0
0x40|            05                                 |    .           |        r: true
0x40|            05                                 |    .           |        w: false
0x40|            05                                 |    .           |        x: true
0x40|               00 00 00                        |     ...        |        unused1: 0
0x40|                        00 00 00 00 00 00 00 00|        ........|      offset: 0x0
0x50|00 00 40 00 00 00 00 00                        |..@.....        |      vaddr: 0x400000
0x50|                        00 00 40 00 00 00 00 00|        ..@.....|      paddr: 0x400000
0x60|00 00 00 00 01 00 00 00                        |........        |      filesz: 4294967296
0x60|                        00 00 00 00 01 00 00 00|        ........|      memsz: 4294967296
0x70|00 00 20 00 00 00 00 00                        |.. .....        |      align: 2097152
0x70|                        b0 3c 66 bf 06 00 0f 05|        .<f.....|  unknown0: raw bits
$ fq -n '"f0VMRgIBAQAAAAAAAAAAAAIAPgABAAAAeABAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAEAAOAABAAAAAAAAAAEAAAAFAAAAAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAIAAAAAAAsDxmvwYADwU=" | frombase64 | . as $b | elf | $b[.header.entry-.program_headers[0].vaddr:]' | ndisasm -b 64 -
00000000  B03C              mov al,0x3c
00000002  66BF0600          mov di,0x6
00000006  0F05              syscall
2022-09-06 11:31:48 +02:00
Mattias Wadman
0f853bda22
Merge pull request #414 from wader/html-array-doctype-etc
html: Handle html with leading doctype etc
2022-09-01 23:09:34 +02:00
Mattias Wadman
8016352b3b html: Handle html with leading doctype etc 2022-09-01 22:58:48 +02:00
Mattias Wadman
df1a81ed26
Merge pull request #413 from wader/decode-remove-rangesort
decode: Remove RangeSorted flag as we can decide on array/struct instead
2022-09-01 17:55:25 +02:00
Mattias Wadman
a6429ffe7a decode: Remove RangeSorted flag as we can decide on array/struct instead 2022-09-01 17:45:28 +02:00
Mattias Wadman
a1cade5390
Merge pull request #412 from wader/interp-decode-struct-byname
interp,decode: For struct use map to lookup field
2022-09-01 11:44:02 +02:00
Mattias Wadman
768df3012c interp,decode: For struct use map to lookup field
Will make it faster for struct with logs of fields and seems to
not cuase any significant difference for small structs.

All this really needs a rewrite somehow, maybe refactor into interfaces somehow? getting messy.
2022-09-01 11:35:19 +02:00
Mattias Wadman
16ebdc1f39
Merge pull request #411 from pnsafonov/perf2
perfomance: increase performance by map usage
2022-09-01 09:47:51 +02:00
Pavel Safonov
7cd43b4919 perfomance: increase performance by map usage 2022-09-01 09:36:50 +03:00
Mattias Wadman
6748b4dccf
Merge pull request #407 from wader/more-golang-x-exp
generics: Use more from x/exp
2022-08-31 11:03:30 +02:00
Mattias Wadman
226a9a3e08 generics: Use more from x/exp 2022-08-31 10:50:56 +02:00
Mattias Wadman
e14cb49ba9
Merge pull request #406 from wader/range-sort-speedup
range,decode: Use own range sort impl to speed up a bit
2022-08-30 19:49:09 +02:00
Mattias Wadman
6a6fec54f5 range,decode: Use own range sort impl to speed up a bit 2022-08-30 15:24:19 +02:00
Mattias Wadman
a87616763e
Merge pull request #405 from wader/sort-refactor
sortex: Package with type safe sort helpers
2022-08-30 11:11:10 +02:00
Mattias Wadman
7d86534348 sortex: Package with type safe sort helpers 2022-08-30 11:02:57 +02:00
Mattias Wadman
e0bd489668
Merge pull request #404 from wader/matroska-lacing
matroska: Add proper lacing support
2022-08-28 21:13:50 +02:00
Mattias Wadman
7749e1b5b9 matroska: Add proper lacing support 2022-08-28 21:03:26 +02:00