1
1
mirror of https://github.com/wader/fq.git synced 2024-11-29 04:10:01 +03:00
Commit Graph

41 Commits

Author SHA1 Message Date
Mattias Wadman
c8f9cdc9d7 interp: Add string_truncate option
Was hardcoded to 50, now confiugurable also used affected by dd, dv etc
2024-04-08 17:45:10 +02:00
Mattias Wadman
051a70bd4b interp: Change bit ranges to use exclusive end
All other ranges and slicing uses exclusive end so i think it make sense
to make it consistent.

Update docs and add additional example for non-byte-aligned field.

Also fixes issue showing zero bit ranges as start-NA.
2023-10-20 15:37:26 +02:00
Mattias Wadman
23ae4d978b decode,interp: Make synthetic values more visible and not act as decode values 2023-10-07 17:40:16 +02:00
Mattias Wadman
56fec2aa7e elf: Fix broken static and segfault tests 2023-08-02 22:13:33 +02:00
Mattias Wadman
a4a332bf4e formats: Clenaup naming a bit 2023-05-01 13:19:04 +02:00
Mattias Wadman
b08ef00dd1 decode,interp: Refactor format groups into a proper struct
Replaces []Format with a Group type.
A bit more type safe.
Breaking change for RegisterFormat, now takes a first argument that is a "single" format group.
Lots of naming cleanup.

This is also preparation for decode group argument which will enable doing intresting
probing, ex a format decoder could know it's decode as part of probe group  (html could
be probed possibly), or have "arg probe" group for decoder who inspect args to know
if they should probe (-d /path/to/schema etc) to enable nice CLI-ergonomics.
2023-04-29 20:02:34 +02:00
Mattias Wadman
0b6ef2a9d8 golangci-lint: Disable revive unused-parameter and update for new default config
unused-parameter disabled as i prefer to see the names
new default revive config seems to not warn about capital names
2023-03-18 22:56:58 +01:00
Mattias Wadman
8e0dde03d0 decode: Support multiple format args and some rename and refactor
This will allow passing both cli options and format options to sub decoder.
Ex: pass keylog option to a tls decoder when decoding a pcap.
Ex: pass decode options to a format inside a http body inside a pcap.

Add ArgAs method to lookup argument based on type. This also makes the format
decode function have same signature as sub decoders in the decode API.

This change decode.Format a bit:
DecodeFn is now just func(d *D) any
DecodeInArg renamed to DefaultInArg
2023-02-18 21:38:51 +01:00
Mattias Wadman
9b81d4d3ab decode: More type safe API and split scalar into multiple types
Preparation to make decoder use less memory and API more type safe.
Now each scalar type has it's own struct type so it can store different
things and enables to have a scalar interface.
Also own types will enable experimenting with decode DLS designs like
using chained methods that are type aware.
2022-12-14 16:23:58 +01:00
Mattias Wadman
e77f776999 decode,interp: Rename unknown gap fields from "unknown#" to "gap#"
Think it makes it clearer and also less likely to collide with a field
name a deocder wants to use.
2022-12-01 20:43:30 +01:00
Mattias Wadman
5382d46a9a elf: Basic program header notes decoding
Used in coredump to record various statees but also used in executables.

Next step would be to decode the note descriptions, like register info, file mappings etc.
Maybe a bit tricky as it is os/abi specific.
2022-09-09 14:35:35 +02:00
Mattias Wadman
12105d8c1d elf: Treat missing string tables as empty to be more robust
Makes it possible to decode shellcode snippets etc:

$ fq -n '"f0VMRgIBAQAAAAAAAAAAAAIAPgABAAAAeABAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAEAAOAABAAAAAAAAAAEAAAAFAAAAAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAIAAAAAAAsDxmvwYADwU=" | frombase64 | elf | d'
    |00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f|0123456789abcdef|.{}: (elf)
    |                                               |                |  error: elf: BitBufRange: failed at position 0 (read size 4294967296 seek pos 0): outside buffer
    |                                               |                |  header{}:
    |                                               |                |    ident{}:
0x00|7f 45 4c 46                                    |.ELF            |      magic: raw bits (valid)
0x00|            02                                 |    .           |      class: 64 (2)
0x00|               01                              |     .          |      data: "little_endian" (1)
0x00|                  01                           |      .         |      version: 1
0x00|                     00                        |       .        |      os_abi: "sysv" (0)
0x00|                        00                     |        .       |      abi_version: 0
0x00|                           00 00 00 00 00 00 00|         .......|      pad: raw bits (all zero)
0x10|02 00                                          |..              |    type: "exec" (0x2)
0x10|      3e 00                                    |  >.            |    machine: "x86_64" (0x3e) (AMD x86-64)
0x10|            01 00 00 00                        |    ....        |    version: 1
0x10|                        78 00 40 00 00 00 00 00|        x.@.....|    entry: 4194424
0x20|40 00 00 00 00 00 00 00                        |@.......        |    phoff: 64
0x20|                        00 00 00 00 00 00 00 00|        ........|    shoff: 0
0x30|00 00 00 00                                    |....            |    flags: 0
0x30|            40 00                              |    @.          |    ehsize: 64
0x30|                  38 00                        |      8.        |    phentsize: 56
0x30|                        01 00                  |        ..      |    phnum: 1
0x30|                              00 00            |          ..    |    shentsize: 0
0x30|                                    00 00      |            ..  |    shnum: 0
0x30|                                          00 00|              ..|    shstrndx: 0
    |                                               |                |  program_headers[0:1]:
    |                                               |                |    [0]{}: program_header
0x40|01 00 00 00                                    |....            |      type: "load" (1) (Loadable segment)
    |                                               |                |      flags{}:
0x40|            05                                 |    .           |        unused0: 0
0x40|            05                                 |    .           |        r: true
0x40|            05                                 |    .           |        w: false
0x40|            05                                 |    .           |        x: true
0x40|               00 00 00                        |     ...        |        unused1: 0
0x40|                        00 00 00 00 00 00 00 00|        ........|      offset: 0x0
0x50|00 00 40 00 00 00 00 00                        |..@.....        |      vaddr: 0x400000
0x50|                        00 00 40 00 00 00 00 00|        ..@.....|      paddr: 0x400000
0x60|00 00 00 00 01 00 00 00                        |........        |      filesz: 4294967296
0x60|                        00 00 00 00 01 00 00 00|        ........|      memsz: 4294967296
0x70|00 00 20 00 00 00 00 00                        |.. .....        |      align: 2097152
0x70|                        b0 3c 66 bf 06 00 0f 05|        .<f.....|  unknown0: raw bits
$ fq -n '"f0VMRgIBAQAAAAAAAAAAAAIAPgABAAAAeABAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAEAAOAABAAAAAAAAAAEAAAAFAAAAAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAIAAAAAAAsDxmvwYADwU=" | frombase64 | . as $b | elf | $b[.header.entry-.program_headers[0].vaddr:]' | ndisasm -b 64 -
00000000  B03C              mov al,0x3c
00000002  66BF0600          mov di,0x6
00000006  0F05              syscall
2022-09-06 11:31:48 +02:00
Mattias Wadman
9a5fcc89f1 xml: Allow trailing <?procinstr?>
Also more context in error messages and refactor trim function
2022-08-11 23:48:29 +02:00
Mattias Wadman
29005c70bf interp,dump: Show address bar for root, nested roots and on format change
Also increase indent for nested roots a bit.

Makes it a bit easier to read i think.
2022-08-09 18:09:41 +02:00
Mattias Wadman
502f451cc3 decode: Refactor to use scalar type assert helper 2022-08-08 18:09:49 +02:00
Mattias Wadman
840292ba6e decode: Simplify compound range sort behaviour
Doing it thru a propery in the decode fn feels a bit hidden and will
also not get set on failed decoding.

Now array is not range sorted, logic is you care about index number and ordering.
Struct is range sorted as you will prefer to fields by name.
2022-08-01 17:17:54 +02:00
Mattias Wadman
3613b6d4dd elf: Remove redundant program_header struct 2022-07-29 13:41:21 +02:00
Mattias Wadman
40481f6692 elf,fuzz: Error on too large string table
Fixes alloc/hang of giant string
2022-07-22 22:35:45 +02:00
Mattias Wadman
342612eb7e dev: Cleanup linters and fix some unused args 2022-07-19 18:33:50 +02:00
Mattias Wadman
1ddea1ada3 interp,format: Refactor registry usage and use function helpers
Move registry to interp and add support for functions and filesystems.
This will be used later for allow formats to add own functions and fq code.

Add gojqextra function helpers to have more comfortable API to add functions.
Takes care of argument type casting and JQValue:s and some more things.

Refactor interp package to use new function helper and registry. Probably
fixes a bunch of JQValue bugs and other type errors.

Refactor out some mpeg nal things to mpeg format.

Refactor interp jq code into display.q and init.jq.

Remove undocumented aes_ctr funciton, was a test. Hopefully will add more crypto things laster.
2022-07-16 19:24:13 +02:00
Mattias Wadman
f66a359c39 elf: Use correct offset to dynamic linking string table
String table used for "needed" tags (the string seen with ldd etc) was using address as
offset which seem to work as it's usually the same as correct offset. But this is not true
for some ELFs. Then we have to look up offset by finding the section by address and then use
its offset for the string table. Hope this is the correct way.

Not sure how to produce a small test for this.
2022-06-30 23:30:56 +02:00
Mattias Wadman
e9d9f8aef9 fq: Use go 1.18
Rename s/interface{}/any/g
Preparation for using generics in decode API and native jq funcations etc
Remove some unused linter ignores as linter has been fixed
2022-05-20 15:23:16 +02:00
Mattias Wadman
9133f0e527 scalar: Add *Fn type to map value and clearer naming
Should replace most of this with generics at some point
2022-05-07 12:46:34 +02:00
Mattias Wadman
c4dd518e04 decode: Make compound range sort optional
Some formats might want to control child order
mp4: Keep tracks in track id order
dns: Keep label component order
elf: Keep seciton order
macho: Keep command and section order
2022-05-03 16:16:09 +02:00
Mattias Wadman
0829c167cc asn1_ber: Add decoder
Basic support, can also do CER and DER but without any extra validation.
No schema support.

Redo format doc.md usage a bit, now format/<dir>/<format>.md instead.

Related to #20
2022-02-06 22:13:03 +01:00
Mattias Wadman
fc0aacb654 interp: Cleanup display aliases, now: d, da, dd, dv, ddv
Think it makes sense to have them all start with d.
Also f is often used as function argument name.
2022-01-28 18:25:38 +01:00
Mattias Wadman
cf8a50c150 decode: Use stable sort for values to not change order or values with same range start
Some decoders might relay on values added with same start to be kept in same order.
2022-01-20 16:21:50 +01:00
siddik.acil
0cf486dcf1 elf: fix all-platforms naming typo 2022-01-08 13:42:08 +01:00
Mattias Wadman
681dbc26b7 elf,ar: Add ar decoder, improved elf decoder
Add x86, 386, arm64, arm v6, arm v7 linux elf test files
2022-01-05 22:05:09 +01:00
Mattias Wadman
2fc0a71a47 decode: Refactor scalar usage
Move scalar into own package.
Split scalar code into decode related scalar code (that reads etc) and
scalar code that just transform the scalar value.
Use a scalar.Mapper interface instead of just a function.
Make mappers, assert and validat impement the interface.
2021-12-02 17:39:26 +01:00
Mattias Wadman
f40320b04c decode: Remove D.Scalar* and add d.(Try)FieldScala*Fn instead
Idea is scalar fn should not read
2021-11-21 13:08:18 +01:00
Mattias Wadman
986d5ecc50 decode: Move registry package to decode/registry and add a format group type 2021-11-17 16:59:56 +01:00
Mattias Wadman
5d98a6949a decode: Refactor Error/Fatal into printf functions 2021-11-17 16:26:13 +01:00
Mattias Wadman
ede2e77975 decode: Nicer scalar template and add doc 2021-11-17 16:13:10 +01:00
Mattias Wadman
b66ed32614 decode,interp: Make fuzzing work again and cleanup fatal/error code 2021-11-16 17:11:26 +01:00
Mattias Wadman
f9f866000e interp,decode: Add force option to ignore asserts 2021-11-16 13:03:56 +01:00
Mattias Wadman
1b32b42f93 decode: Major decode API refactor
Generate more code
More generic and comfortable API
Improve and Update format decoder to new API
Add some more format tests
2021-11-15 21:12:07 +01:00
Mattias Wadman
6a15625587 interp,decode: Refactor out Scalar from Value and merge Array/Struct into Compound
Also add tosym, toactual
2021-11-05 17:29:22 +01:00
Mattias Wadman
3ff0c9b5e0 lint: Enable errcheck adnd revive 2021-09-27 11:01:14 +02:00
Mattias Wadman
f1507f7f65 mod: Use proper path and dont use replace 2021-09-12 13:08:50 +02:00
Mattias Wadman
970465996c Init 2021-09-12 13:08:42 +02:00