David McDonald
368d183b73
Size check on nBits to save memory
2022-09-23 20:23:29 -05:00
David McDonald
a77cec921f
Added documentation and tests, fixed bad date parsing
2022-09-23 20:23:23 -05:00
David McDonald
5711f29084
Code fixes from PR, still need to add tests and testdata
2022-09-23 20:21:32 -05:00
David McDonald
d784db69bc
Adds support for Apple Binary Plist, version 00
2022-09-23 20:21:32 -05:00
Mattias Wadman
783b20ece5
Merge pull request #430 from wader/all-format-test-in-format
...
doc,help: Move help tests into each format
2022-09-22 23:14:40 +02:00
Mattias Wadman
a7a101ca67
doc,help: Nicer format help and move help tests into each format
...
Also add authors to avro and macho decoders
Generated with:
for i in $(go run . -r -n '_registry | [([([.files[][].name | split(".")[0]] | unique[]), (.formats[].name)] | count[] | select(.[1] == 2)[0]), (.formats[] | select(.decode_in_arg).name)] | unique[]'); do DIR=$(dirname $(find format -name $i.go)); echo "$ fq -h $i" > "$DIR/testdata/help_$i.fqtest" ; done
2022-09-22 23:07:58 +02:00
Mattias Wadman
2e3d71fdcc
Merge pull request #429 from wader/more-format-doc
...
doc,html,xml: Add more documentation and examples
2022-09-20 18:32:15 +02:00
Mattias Wadman
725ab1b17b
doc,html,xml: Add more documentation and examples
2022-09-20 17:35:27 +02:00
Mattias Wadman
01be59e459
Merge pull request #428 from wader/ar-allow-empty-integerr
...
ar: Integer strings might be empty
2022-09-17 16:17:47 +02:00
Mattias Wadman
f08f44f12a
ar: Integer strings might be empty
2022-09-17 15:36:34 +02:00
Mattias Wadman
ee7b9f4186
Merge pull request #426 from wader/csv-tsv-doc
...
csv: Add tsv and header example
2022-09-15 01:27:54 +02:00
Mattias Wadman
fe64530e11
csv: Add tsv and header example
...
Also fix string esacpe in fuzz options parser
2022-09-15 01:16:45 +02:00
Mattias Wadman
88009ee9a3
Merge pull request #425 from wader/wasm-doc
...
wasm: Add some documentation
2022-09-13 09:59:17 +02:00
Mattias Wadman
1385b5d0ba
wasm: Add some documentation
2022-09-12 11:53:48 +02:00
Mattias Wadman
1a2cee94b7
Merge pull request #424 from wader/mp4-more-codec-names
...
mp4: More codec names (from mp4ra.org)
2022-09-12 11:42:10 +02:00
Mattias Wadman
2655ba09c5
mp4: More codec names (from mp4ra.org)
2022-09-12 11:25:43 +02:00
Mattias Wadman
ac93da62c2
Merge pull request #423 from wader/doc-formats-sections
...
doc: Fix format sections a bit
2022-09-11 10:08:24 +02:00
Mattias Wadman
abd19ed89b
doc: Fix format sections a bit
2022-09-11 09:55:56 +02:00
Mattias Wadman
577ea7665c
Merge pull request #422 from wader/markdown
...
Move format documentation to markdown again
2022-09-11 00:59:49 +02:00
Mattias Wadman
87b2c6c10c
help,doc: Use markdown for format documentation again
...
Markdown is used as is in online documentation and in cli the markdown decoder
is used to decode and the some jq code massages it into something cli friendly.
Was just too much of a mess to have doc in jq.
2022-09-11 00:53:29 +02:00
Mattias Wadman
00a50662ea
markdown: Add decoder
2022-09-10 18:26:45 +02:00
Mattias Wadman
e98adfaa03
Merge pull request #421 from wader/elf-notes
...
elf: Basic program header notes decoding
2022-09-09 14:48:02 +02:00
Mattias Wadman
5382d46a9a
elf: Basic program header notes decoding
...
Used in coredump to record various statees but also used in executables.
Next step would be to decode the note descriptions, like register info, file mappings etc.
Maybe a bit tricky as it is os/abi specific.
2022-09-09 14:35:35 +02:00
Mattias Wadman
2a5a53447c
Merge pull request #420 from piggynl/patch-1
...
test: Skip go test with -race by default
2022-09-09 09:40:38 +02:00
Piggy NL
808202fa34
test: Skip go test with -race by default
2022-09-09 15:15:18 +08:00
Mattias Wadman
ee6691728f
Merge pull request #419 from wader/bump-github-go-version-1.19.1
...
Update github-go-version from 1.19.0, 1.19.0, 1.19.0 to 1.19.1
2022-09-07 18:15:01 +02:00
Mattias Wadman
4096de99bd
Merge pull request #418 from wader/bump-docker-golang-1.19.1
...
Update docker-golang from 1.19.0 to 1.19.1
2022-09-07 18:11:40 +02:00
bump
c47c3866c4
Update github-go-version from 1.19.0, 1.19.0, 1.19.0 to 1.19.1
2022-09-07 16:03:48 +00:00
bump
84ca1010e3
Update docker-golang from 1.19.0 to 1.19.1
2022-09-07 16:03:45 +00:00
Mattias Wadman
f617f39a9a
Merge pull request #417 from wader/elf-more-robust
...
elf: Treat missing string tables as empty to be more robust
2022-09-06 11:40:38 +02:00
Mattias Wadman
12105d8c1d
elf: Treat missing string tables as empty to be more robust
...
Makes it possible to decode shellcode snippets etc:
$ fq -n '"f0VMRgIBAQAAAAAAAAAAAAIAPgABAAAAeABAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAEAAOAABAAAAAAAAAAEAAAAFAAAAAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAIAAAAAAAsDxmvwYADwU=" | frombase64 | elf | d'
|00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f|0123456789abcdef|.{}: (elf)
| | | error: elf: BitBufRange: failed at position 0 (read size 4294967296 seek pos 0): outside buffer
| | | header{}:
| | | ident{}:
0x00|7f 45 4c 46 |.ELF | magic: raw bits (valid)
0x00| 02 | . | class: 64 (2)
0x00| 01 | . | data: "little_endian" (1)
0x00| 01 | . | version: 1
0x00| 00 | . | os_abi: "sysv" (0)
0x00| 00 | . | abi_version: 0
0x00| 00 00 00 00 00 00 00| .......| pad: raw bits (all zero)
0x10|02 00 |.. | type: "exec" (0x2)
0x10| 3e 00 | >. | machine: "x86_64" (0x3e) (AMD x86-64)
0x10| 01 00 00 00 | .... | version: 1
0x10| 78 00 40 00 00 00 00 00| x.@.....| entry: 4194424
0x20|40 00 00 00 00 00 00 00 |@....... | phoff: 64
0x20| 00 00 00 00 00 00 00 00| ........| shoff: 0
0x30|00 00 00 00 |.... | flags: 0
0x30| 40 00 | @. | ehsize: 64
0x30| 38 00 | 8. | phentsize: 56
0x30| 01 00 | .. | phnum: 1
0x30| 00 00 | .. | shentsize: 0
0x30| 00 00 | .. | shnum: 0
0x30| 00 00| ..| shstrndx: 0
| | | program_headers[0:1]:
| | | [0]{}: program_header
0x40|01 00 00 00 |.... | type: "load" (1) (Loadable segment)
| | | flags{}:
0x40| 05 | . | unused0: 0
0x40| 05 | . | r: true
0x40| 05 | . | w: false
0x40| 05 | . | x: true
0x40| 00 00 00 | ... | unused1: 0
0x40| 00 00 00 00 00 00 00 00| ........| offset: 0x0
0x50|00 00 40 00 00 00 00 00 |..@..... | vaddr: 0x400000
0x50| 00 00 40 00 00 00 00 00| ..@.....| paddr: 0x400000
0x60|00 00 00 00 01 00 00 00 |........ | filesz: 4294967296
0x60| 00 00 00 00 01 00 00 00| ........| memsz: 4294967296
0x70|00 00 20 00 00 00 00 00 |.. ..... | align: 2097152
0x70| b0 3c 66 bf 06 00 0f 05| .<f.....| unknown0: raw bits
$ fq -n '"f0VMRgIBAQAAAAAAAAAAAAIAPgABAAAAeABAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAEAAOAABAAAAAAAAAAEAAAAFAAAAAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAIAAAAAAAsDxmvwYADwU=" | frombase64 | . as $b | elf | $b[.header.entry-.program_headers[0].vaddr:]' | ndisasm -b 64 -
00000000 B03C mov al,0x3c
00000002 66BF0600 mov di,0x6
00000006 0F05 syscall
2022-09-06 11:31:48 +02:00
Mattias Wadman
0f853bda22
Merge pull request #414 from wader/html-array-doctype-etc
...
html: Handle html with leading doctype etc
2022-09-01 23:09:34 +02:00
Mattias Wadman
8016352b3b
html: Handle html with leading doctype etc
2022-09-01 22:58:48 +02:00
Mattias Wadman
df1a81ed26
Merge pull request #413 from wader/decode-remove-rangesort
...
decode: Remove RangeSorted flag as we can decide on array/struct instead
2022-09-01 17:55:25 +02:00
Mattias Wadman
a6429ffe7a
decode: Remove RangeSorted flag as we can decide on array/struct instead
2022-09-01 17:45:28 +02:00
Mattias Wadman
a1cade5390
Merge pull request #412 from wader/interp-decode-struct-byname
...
interp,decode: For struct use map to lookup field
2022-09-01 11:44:02 +02:00
Mattias Wadman
768df3012c
interp,decode: For struct use map to lookup field
...
Will make it faster for struct with logs of fields and seems to
not cuase any significant difference for small structs.
All this really needs a rewrite somehow, maybe refactor into interfaces somehow? getting messy.
2022-09-01 11:35:19 +02:00
Mattias Wadman
16ebdc1f39
Merge pull request #411 from pnsafonov/perf2
...
perfomance: increase performance by map usage
2022-09-01 09:47:51 +02:00
Pavel Safonov
7cd43b4919
perfomance: increase performance by map usage
2022-09-01 09:36:50 +03:00
Mattias Wadman
6748b4dccf
Merge pull request #407 from wader/more-golang-x-exp
...
generics: Use more from x/exp
2022-08-31 11:03:30 +02:00
Mattias Wadman
226a9a3e08
generics: Use more from x/exp
2022-08-31 10:50:56 +02:00
Mattias Wadman
e14cb49ba9
Merge pull request #406 from wader/range-sort-speedup
...
range,decode: Use own range sort impl to speed up a bit
2022-08-30 19:49:09 +02:00
Mattias Wadman
6a6fec54f5
range,decode: Use own range sort impl to speed up a bit
2022-08-30 15:24:19 +02:00
Mattias Wadman
a87616763e
Merge pull request #405 from wader/sort-refactor
...
sortex: Package with type safe sort helpers
2022-08-30 11:11:10 +02:00
Mattias Wadman
7d86534348
sortex: Package with type safe sort helpers
2022-08-30 11:02:57 +02:00
Mattias Wadman
e0bd489668
Merge pull request #404 from wader/matroska-lacing
...
matroska: Add proper lacing support
2022-08-28 21:13:50 +02:00
Mattias Wadman
7749e1b5b9
matroska: Add proper lacing support
2022-08-28 21:03:26 +02:00
Mattias Wadman
32c16dd839
Merge pull request #402 from wader/bitcoin-blkdata-has-header
...
bitcoin_blkdat,bitcoin_block: Make sure there is a header if blkdat
2022-08-26 12:40:04 +02:00
Mattias Wadman
004406de65
bitcoin_blkdat,bitcoin_block: Make sure there is a header if blkdat
...
Makes bitcoin_blkdat fails fast as it is part of probe group.
Speeds up reading a big JSON file etc.
2022-08-26 12:07:53 +02:00
Mattias Wadman
6bbfc66051
Merge pull request #388 from wader/release-0.0.9
...
fq: Release 0.0.9
2022-08-25 17:40:00 +02:00
