1
1
mirror of https://github.com/wader/fq.git synced 2024-11-26 10:33:53 +03:00
Commit Graph

40 Commits

Author SHA1 Message Date
Mattias Wadman
c8f9cdc9d7 interp: Add string_truncate option
Was hardcoded to 50, now confiugurable also used affected by dd, dv etc
2024-04-08 17:45:10 +02:00
Mattias Wadman
02b3527608 exif,tiff: Handle broken last next ifd offset by treating it as end marker 2023-11-03 16:13:49 +01:00
Mattias Wadman
9e58067f4b webp: Refactor to use riff code and decode VP8X, EXIF, ICCP and XMP chunks 2023-11-01 18:58:09 +01:00
Mattias Wadman
051a70bd4b interp: Change bit ranges to use exclusive end
All other ranges and slicing uses exclusive end so i think it make sense
to make it consistent.

Update docs and add additional example for non-byte-aligned field.

Also fixes issue showing zero bit ranges as start-NA.
2023-10-20 15:37:26 +02:00
Mattias Wadman
a4a332bf4e formats: Clenaup naming a bit 2023-05-01 13:19:04 +02:00
Mattias Wadman
b08ef00dd1 decode,interp: Refactor format groups into a proper struct
Replaces []Format with a Group type.
A bit more type safe.
Breaking change for RegisterFormat, now takes a first argument that is a "single" format group.
Lots of naming cleanup.

This is also preparation for decode group argument which will enable doing intresting
probing, ex a format decoder could know it's decode as part of probe group  (html could
be probed possibly), or have "arg probe" group for decoder who inspect args to know
if they should probe (-d /path/to/schema etc) to enable nice CLI-ergonomics.
2023-04-29 20:02:34 +02:00
Mattias Wadman
0b6ef2a9d8 golangci-lint: Disable revive unused-parameter and update for new default config
unused-parameter disabled as i prefer to see the names
new default revive config seems to not warn about capital names
2023-03-18 22:56:58 +01:00
Mattias Wadman
8e0dde03d0 decode: Support multiple format args and some rename and refactor
This will allow passing both cli options and format options to sub decoder.
Ex: pass keylog option to a tls decoder when decoding a pcap.
Ex: pass decode options to a format inside a http body inside a pcap.

Add ArgAs method to lookup argument based on type. This also makes the format
decode function have same signature as sub decoders in the decode API.

This change decode.Format a bit:
DecodeFn is now just func(d *D) any
DecodeInArg renamed to DefaultInArg
2023-02-18 21:38:51 +01:00
Mattias Wadman
e3ae1440c9 interp: Rename to/from<format> functions to to_/from_<format>
Feels less cluttered, easier to read and more consistent.

Still keep tovalue, tobytes etc that are more basic functions this
only renamed format related functions.
Also there is an exceptin for to/fromjson as it comes from jq.

Also fixes lots of spelling errors while reading thru.
2022-12-21 17:48:39 +01:00
Mattias Wadman
9b81d4d3ab decode: More type safe API and split scalar into multiple types
Preparation to make decoder use less memory and API more type safe.
Now each scalar type has it's own struct type so it can store different
things and enables to have a scalar interface.
Also own types will enable experimenting with decode DLS designs like
using chained methods that are type aware.
2022-12-14 16:23:58 +01:00
Mattias Wadman
e77f776999 decode,interp: Rename unknown gap fields from "unknown#" to "gap#"
Think it makes it clearer and also less likely to collide with a field
name a deocder wants to use.
2022-12-01 20:43:30 +01:00
Mattias Wadman
9a5fcc89f1 xml: Allow trailing <?procinstr?>
Also more context in error messages and refactor trim function
2022-08-11 23:48:29 +02:00
Mattias Wadman
342612eb7e dev: Cleanup linters and fix some unused args 2022-07-19 18:33:50 +02:00
Mattias Wadman
1ddea1ada3 interp,format: Refactor registry usage and use function helpers
Move registry to interp and add support for functions and filesystems.
This will be used later for allow formats to add own functions and fq code.

Add gojqextra function helpers to have more comfortable API to add functions.
Takes care of argument type casting and JQValue:s and some more things.

Refactor interp package to use new function helper and registry. Probably
fixes a bunch of JQValue bugs and other type errors.

Refactor out some mpeg nal things to mpeg format.

Refactor interp jq code into display.q and init.jq.

Remove undocumented aes_ctr funciton, was a test. Hopefully will add more crypto things laster.
2022-07-16 19:24:13 +02:00
Mattias Wadman
3b717c3ba4 interp: Add to/from<encoding> for some common serialzations, encodings and hashes
Add toxml/fromxml for XML encoding, mighe be lossy on ordering
fromxml has {seq:bool} to add #seq attributes to improve ordering
toxml has {indent:number} to choose space indent depth

Add tojson, same as in jq but also has {indent:number} options

Add toyaml/fromyaml for YAML

Add totoml/fromtoml for TOML

Add tojq/fromjq for jq-flavored JSON (optional quotes for keys, comments and trailing commas support)

Add tocsv/fromcsv for CSV
formcvs takes {comma:string, comment:string} for custom separtor and comment character

Rename/split hex into tohex/fromhex
Rename/split base64 into tobase64/frombase64
tobase64/frombase64 takes {encoding:string} option for base64 flavour (std, url, rawstd, rawurl)

Add to/from<format> urlpath, urlquery, url, xmlentities, base64, hex

Add to<hash> md4, md5, sha1, sha256, sha512, sha3_224, sha3_256, sha3_384, sha3_512

Add to/from<encoding> iso8859-1, utf8, utf16, utf16le, utf16be
2022-05-28 16:31:20 +02:00
Mattias Wadman
377af133e0 fqtest: Cleanup path usage
Make cwd for a test script the directory where the script is.
Use relative paths
2022-05-21 20:03:25 +02:00
Mattias Wadman
e9d9f8aef9 fq: Use go 1.18
Rename s/interface{}/any/g
Preparation for using generics in decode API and native jq funcations etc
Remove some unused linter ignores as linter has been fixed
2022-05-20 15:23:16 +02:00
Mattias Wadman
9133f0e527 scalar: Add *Fn type to map value and clearer naming
Should replace most of this with generics at some point
2022-05-07 12:46:34 +02:00
Mattias Wadman
fc0aacb654 interp: Cleanup display aliases, now: d, da, dd, dv, ddv
Think it makes sense to have them all start with d.
Also f is often used as function argument name.
2022-01-28 18:25:38 +01:00
Mattias Wadman
91217e8210 tiff: Fix endian typo and cleanup todos 2021-12-12 11:26:00 +01:00
Mattias Wadman
07a2ebe4b7 tiff,fuzz: Fatal error on infinite ifd loops 2021-12-12 11:20:01 +01:00
Mattias Wadman
6ed2e2e72e interp: dump: Indicate arrays using jq-syntax
Related to #16
2021-12-08 16:24:28 +01:00
Mattias Wadman
2fc0a71a47 decode: Refactor scalar usage
Move scalar into own package.
Split scalar code into decode related scalar code (that reads etc) and
scalar code that just transform the scalar value.
Use a scalar.Mapper interface instead of just a function.
Make mappers, assert and validat impement the interface.
2021-12-02 17:39:26 +01:00
Mattias Wadman
986d5ecc50 decode: Move registry package to decode/registry and add a format group type 2021-11-17 16:59:56 +01:00
Mattias Wadman
5d98a6949a decode: Refactor Error/Fatal into printf functions 2021-11-17 16:26:13 +01:00
Mattias Wadman
ede2e77975 decode: Nicer scalar template and add doc 2021-11-17 16:13:10 +01:00
Mattias Wadman
b66ed32614 decode,interp: Make fuzzing work again and cleanup fatal/error code 2021-11-16 17:11:26 +01:00
Mattias Wadman
1b32b42f93 decode: Major decode API refactor
Generate more code
More generic and comfortable API
Improve and Update format decoder to new API
Add some more format tests
2021-11-15 21:12:07 +01:00
Mattias Wadman
cc5f405554 interp,format: Update tests after decode refactor/tosym/toactual 2021-11-05 17:29:23 +01:00
Mattias Wadman
691688022f fqtest: Refactor our script part to own package
Hopefully can be reused for docs generation
2021-10-06 18:49:03 +02:00
Mattias Wadman
761c411ca9 exif: Add note about JPEGInterchangeFormat 2021-10-01 17:08:16 +02:00
Mattias Wadman
3ff0c9b5e0 lint: Enable errcheck adnd revive 2021-09-27 11:01:14 +02:00
Mattias Wadman
3db11d3fa8 decode: Add UTF8Fn functions and trim some null terminated strings 2021-09-17 22:13:35 +02:00
Mattias Wadman
ee972f490d dump: Add ascii header 2021-09-16 22:40:03 +02:00
Mattias Wadman
473b2243c9 decode: Simplify and move format arg into DecodeOptions 2021-09-16 15:29:11 +02:00
Mattias Wadman
798141a0b0 format: Cleanup comments 2021-09-14 17:01:25 +02:00
Mattias Wadman
f1507f7f65 mod: Use proper path and dont use replace 2021-09-12 13:08:50 +02:00
Mattias Wadman
c155c89ba3 decode: Rename format *Decode to *Format 2021-09-12 13:08:50 +02:00
Mattias Wadman
8dc0f06b62 cli: Add completion tests 2021-09-12 13:08:49 +02:00
Mattias Wadman
970465996c Init 2021-09-12 13:08:42 +02:00