mirror of
https://github.com/wader/fq.git
synced 2024-08-18 00:10:29 +03:00
Compare commits
30 Commits
5244172349
...
cff09c5d6c
Author | SHA1 | Date | |
---|---|---|---|
|
cff09c5d6c | ||
|
943743dbaf | ||
|
1b85ae622b | ||
|
f6d7235449 | ||
|
aef47df26a | ||
|
1ec9748046 | ||
|
2e5514fc50 | ||
|
a59ba2a2fa | ||
|
7714fcf423 | ||
|
544cf4cc09 | ||
|
25ad5f1c8f | ||
|
9ff7da12a7 | ||
|
94cfbc670c | ||
|
163b3b609c | ||
|
cabb67e8ab | ||
|
64df8bdbf5 | ||
|
481ac91880 | ||
|
14ada50806 | ||
|
12f332064c | ||
|
0ff3e53c5f | ||
|
586cf142e5 | ||
|
fb20db5eb7 | ||
|
48868bd4ee | ||
|
c5c8b75c56 | ||
|
a5de74cd23 | ||
|
42730d7586 | ||
|
3a683b64f8 | ||
|
496849daa5 | ||
|
ebf063d1c0 | ||
|
9c1c3c3fce |
6
.github/workflows/ci.yml
vendored
6
.github/workflows/ci.yml
vendored
@ -7,7 +7,7 @@ on:
|
||||
pull_request:
|
||||
|
||||
env:
|
||||
GOLANGCILINT_VERSION: "1.57.2"
|
||||
GOLANGCILINT_VERSION: "1.58.2"
|
||||
|
||||
jobs:
|
||||
lint:
|
||||
@ -15,7 +15,7 @@ jobs:
|
||||
steps:
|
||||
- uses: actions/setup-go@v3
|
||||
with:
|
||||
go-version: "1.22.2"
|
||||
go-version: "1.22.3"
|
||||
- uses: actions/checkout@v3
|
||||
- uses: golangci/golangci-lint-action@v3
|
||||
with:
|
||||
@ -47,7 +47,7 @@ jobs:
|
||||
- uses: actions/checkout@v3
|
||||
- uses: actions/setup-go@v3
|
||||
with:
|
||||
go-version: "1.22.2"
|
||||
go-version: "1.22.3"
|
||||
- name: Test
|
||||
env:
|
||||
GOARCH: ${{ matrix.goarch }}
|
||||
|
2
.github/workflows/release.yml
vendored
2
.github/workflows/release.yml
vendored
@ -15,7 +15,7 @@ jobs:
|
||||
- uses: actions/checkout@v3
|
||||
- uses: actions/setup-go@v3
|
||||
with:
|
||||
go-version: "1.22.2"
|
||||
go-version: "1.22.3"
|
||||
- uses: goreleaser/goreleaser-action@v5
|
||||
with:
|
||||
distribution: goreleaser
|
||||
|
@ -1,5 +1,5 @@
|
||||
# bump: docker-golang /FROM golang:([\d.]+)/ docker:golang|^1
|
||||
FROM golang:1.22.2-bookworm AS base
|
||||
FROM golang:1.22.3-bookworm AS base
|
||||
|
||||
# expect is used to test cli
|
||||
RUN \
|
||||
|
2
Makefile
2
Makefile
@ -61,7 +61,7 @@ gogenerate: always
|
||||
lint: always
|
||||
# bump: make-golangci-lint /golangci-lint@v([\d.]+)/ git:https://github.com/golangci/golangci-lint.git|^1
|
||||
# bump: make-golangci-lint link "Release notes" https://github.com/golangci/golangci-lint/releases/tag/v$LATEST
|
||||
go run github.com/golangci/golangci-lint/cmd/golangci-lint@v1.57.2 run
|
||||
go run github.com/golangci/golangci-lint/cmd/golangci-lint@v1.58.2 run
|
||||
|
||||
depgraph.svg: always
|
||||
go run github.com/kisielk/godepgraph@latest github.com/wader/fq | dot -Tsvg -o godepgraph.svg
|
||||
|
24
README.md
24
README.md
@ -2,25 +2,23 @@
|
||||
|
||||
Tool, language and decoders for working with binary data.
|
||||
|
||||
TLDR: it aims to be jq, hexdump, dd and gdb for files combined into one.
|
||||
|
||||
![fq demo](doc/demo.svg)
|
||||
|
||||
Basic usage is `fq . file` or `fq d file`.
|
||||
Basic usage is `fq . file`, `fq d file` or `fq 'some query' file ...`.
|
||||
|
||||
For details see [usage.md](doc/usage.md).
|
||||
|
||||
### Background
|
||||
|
||||
fq is inspired by the well known jq tool and language that allows you to work with binary formats the same way you would using jq. In addition it can present data like a hex viewer, transform, slice and concatenate binary data. It also supports nested formats and has an interactive REPL with auto-completion.
|
||||
fq is inspired by the [jq](https://jqlang.github.io/jq/) tool and language and allows you to work with binary formats in the same way. In addition to using jq expressions it can also present decoded tree structures, transform, slice and concatenate binary data. It also supports nested formats and features an interactive REPL with auto-completion of functions and names.
|
||||
|
||||
It was originally designed to query, inspect and debug media codecs and containers like mp4, flac, mp3, jpeg. But since then it has been extended to support a variety of formats like executables, packet captures (with TCP reassembly) and serialization formats like JSON, YAML, XML, ASN1 BER, Avro, CBOR, protobuf. In addition it also has functions to work with URLs, convert to/from hex, number bases, search for things etc.
|
||||
|
||||
In summary it aims to be jq, hexdump, dd and gdb for files combined into one.
|
||||
|
||||
**NOTE:** fq is still early in development so things might change, be broken or do not make sense. That also means that there is a great opportunity to help out!
|
||||
It was originally designed to query, inspect and debug media codecs and containers like MP4, FLAC and JPEG but has since been extended to support a variety of formats like executables, packet captures (with TCP reassembly) and serialization formats like JSON, YAML, XML, CBOR, protobuf. In addition it also has functions to work with URLs, convert to/from hex, number bases, search for patterns etc.
|
||||
|
||||
### Goals
|
||||
|
||||
- Make binaries accessible, queryable and sliceable.
|
||||
- Make binaries more accessible, queryable and sliceable.
|
||||
- Nested formats and bit-oriented decoding.
|
||||
- Quick and comfortable CLI tool.
|
||||
- Bits and bytes transformations.
|
||||
@ -243,7 +241,7 @@ apk add -X http://dl-cdn.alpinelinux.org/alpine/edge/testing fq
|
||||
|
||||
Make sure you have [go](https://go.dev) 1.20 or later installed.
|
||||
|
||||
To install directly from git repository (no clone needed):
|
||||
To install directly from git repository (no git clone needed):
|
||||
```sh
|
||||
# build and install latest release
|
||||
go install github.com/wader/fq@latest
|
||||
@ -267,10 +265,6 @@ go build -o fq .
|
||||
make test fq
|
||||
```
|
||||
|
||||
## TODO and ideas
|
||||
|
||||
See [TODO.md](doc/TODO.md)
|
||||
|
||||
## Development and adding a new decoder
|
||||
|
||||
See [dev.md](doc/dev.md)
|
||||
@ -306,6 +300,10 @@ for inventing the [jq](https://github.com/stedolan/jq) language.
|
||||
- [Sustainability of Digital Formats](https://www.loc.gov/preservation/digital/formats/) at Library of Congress.
|
||||
- [Data Format Description Language (DFDL)](https://en.wikipedia.org/wiki/Data_Format_Description_Language).
|
||||
|
||||
## TODO and ideas
|
||||
|
||||
See [TODO.md](doc/TODO.md)
|
||||
|
||||
## License
|
||||
|
||||
`fq` is distributed under the terms of the MIT License.
|
||||
|
@ -236,8 +236,8 @@ make test
|
||||
go test ./...
|
||||
# run all tests for one format
|
||||
go test -run TestFormats/mp4 ./format/
|
||||
# update all actual outputs in tests
|
||||
go test ./... -update
|
||||
# update all expected outputs for tests
|
||||
go test ./pkg/interp ./format -update
|
||||
# update actual output for specific tests
|
||||
go run ./format -run TestFormats/elf -update
|
||||
# color diff
|
||||
|
@ -46,6 +46,7 @@ import (
|
||||
_ "github.com/wader/fq/format/opentimestamps"
|
||||
_ "github.com/wader/fq/format/opus"
|
||||
_ "github.com/wader/fq/format/pcap"
|
||||
_ "github.com/wader/fq/format/pdf"
|
||||
_ "github.com/wader/fq/format/png"
|
||||
_ "github.com/wader/fq/format/postgres"
|
||||
_ "github.com/wader/fq/format/prores"
|
||||
|
@ -154,6 +154,7 @@ var (
|
||||
Opus_Packet = &decode.Group{Name: "opus_packet"}
|
||||
PCAP = &decode.Group{Name: "pcap"}
|
||||
PCAPNG = &decode.Group{Name: "pcapng"}
|
||||
PDF = &decode.Group{Name: "pdf"}
|
||||
Pg_BTree = &decode.Group{Name: "pg_btree"}
|
||||
Pg_Control = &decode.Group{Name: "pg_control"}
|
||||
Pg_Heap = &decode.Group{Name: "pg_heap"}
|
||||
|
418
format/pdf/pdf.go
Normal file
418
format/pdf/pdf.go
Normal file
@ -0,0 +1,418 @@
|
||||
package pdf
|
||||
|
||||
// https://www.adobe.com/content/dam/acom/en/devnet/pdf/pdfs/PDF32000_2008.pdf
|
||||
// https://feliam.wordpress.com/2010/08/14/pdf-a-broken-spec/
|
||||
// https://github.com/modesty/pdf2json
|
||||
|
||||
// TODO: parse-from-end if possible?
|
||||
// TODO: more unescape?
|
||||
// TODO: streams filters
|
||||
// TODO: refs
|
||||
// TODO: EOL between object number generation "obj"?
|
||||
// Ex:
|
||||
// 202 0
|
||||
// obj
|
||||
// endobj
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"io"
|
||||
"log"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/wader/fq/format"
|
||||
"github.com/wader/fq/pkg/decode"
|
||||
"github.com/wader/fq/pkg/interp"
|
||||
"github.com/wader/fq/pkg/scalar"
|
||||
"golang.org/x/text/encoding/charmap"
|
||||
"golang.org/x/text/encoding/unicode"
|
||||
)
|
||||
|
||||
var imageFormat decode.Group
|
||||
|
||||
func init() {
|
||||
interp.RegisterFormat(
|
||||
format.PDF,
|
||||
&decode.Format{
|
||||
Description: "Portable Document Format",
|
||||
Groups: []*decode.Group{format.Probe},
|
||||
DecodeFn: pdfDecode,
|
||||
Dependencies: []decode.Dependency{
|
||||
{Groups: []*decode.Group{format.Image}, Out: &imageFormat},
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
var eofLine = `%%EOF`
|
||||
var whitespaceChars = []byte(" \n\r")
|
||||
var delimiterChars = []byte(" \n\r()<>[]{}/%")
|
||||
|
||||
func isWhitespace(b byte) bool { return bytes.IndexByte(whitespaceChars, b) != -1 }
|
||||
|
||||
// \r, \n or \r\n
|
||||
// TODO: figure someting nicer
|
||||
func findLineEnd(d *decode.D) int64 {
|
||||
p := d.Pos()
|
||||
ei, v, err := d.TryPeekFind(8, 8, -1, func(v uint64) bool {
|
||||
return v == '\n' || v == '\r'
|
||||
})
|
||||
if errors.Is(err, io.EOF) {
|
||||
// to support %%EOF not having a line ending
|
||||
eofLineLen := int64(len(eofLine))
|
||||
if d.BitsLeft()*8 >= eofLineLen {
|
||||
return eofLineLen
|
||||
}
|
||||
return -1
|
||||
}
|
||||
if ei == -1 {
|
||||
return ei
|
||||
}
|
||||
ei += 8
|
||||
d.SeekRel(ei)
|
||||
if v == '\r' && d.PeekUintBits(8) == '\n' {
|
||||
ei += 8
|
||||
}
|
||||
d.SeekAbs(p)
|
||||
return ei / 8
|
||||
}
|
||||
|
||||
var strToNumber = scalar.StrFn(func(s scalar.Str) (scalar.Str, error) {
|
||||
if strings.Contains(s.Actual, ".") {
|
||||
// real
|
||||
n, err := strconv.ParseFloat(s.Actual, 64)
|
||||
if err != nil {
|
||||
return s, err
|
||||
}
|
||||
s.Sym = n
|
||||
} else {
|
||||
// integer
|
||||
n, err := strconv.ParseInt(s.Actual, 10, 64)
|
||||
if err != nil {
|
||||
return s, err
|
||||
}
|
||||
s.Sym = n
|
||||
}
|
||||
|
||||
return s, nil
|
||||
})
|
||||
|
||||
func decodeLineStr(d *decode.D) string {
|
||||
ei := findLineEnd(d)
|
||||
if ei == -1 {
|
||||
d.Errorf("could not find line ending")
|
||||
}
|
||||
return strings.TrimRight(d.UTF8(int(ei)), "\r\n")
|
||||
}
|
||||
|
||||
func decodeStrWhitespace(d *decode.D) string {
|
||||
ei, _ := d.PeekFind(8, 8, -1, func(v uint64) bool {
|
||||
return !isWhitespace(byte(v))
|
||||
})
|
||||
if ei == -1 {
|
||||
d.Errorf("could not find line ending")
|
||||
}
|
||||
return d.UTF8(int(ei / 8))
|
||||
}
|
||||
|
||||
func decodeStrUntil(b []byte) func(d *decode.D) string {
|
||||
return func(d *decode.D) string {
|
||||
ei, _ := d.PeekFind(8, 8, -1, func(v uint64) bool {
|
||||
return bytes.IndexByte(b, byte(v)) != -1
|
||||
})
|
||||
if ei == -1 {
|
||||
d.Errorf("could not find ending %d", b)
|
||||
}
|
||||
return d.UTF8(int(ei / 8))
|
||||
}
|
||||
}
|
||||
|
||||
var utf16BEExpect = unicode.UTF16(unicode.BigEndian, unicode.ExpectBOM)
|
||||
var pdfDocEncoding = charmap.ISO8859_1 // TODO: correc?
|
||||
|
||||
// TODO: unescape \N \NN \NNN?
|
||||
func decodeText(bs []byte) string {
|
||||
if s, err := utf16BEExpect.NewDecoder().String(string(bs)); err == nil {
|
||||
return s
|
||||
}
|
||||
if s, err := pdfDocEncoding.NewDecoder().String(string(bs)); err == nil {
|
||||
return s
|
||||
}
|
||||
return string(bs)
|
||||
}
|
||||
|
||||
// "\" + "\n" or "\r" or "\r\n"
|
||||
var eolEscapeRe = regexp.MustCompile("\\\\(?:\n|\r|\r\n)")
|
||||
|
||||
func decodeStrLiteralString(d *decode.D) string {
|
||||
ei, _ := d.PeekFind(16, 8, -1, func(v uint64) bool {
|
||||
c1 := v >> 8
|
||||
c0 := v & 0xff
|
||||
// find non-escaped ")"
|
||||
return c0 != '\\' && c1 == ')'
|
||||
})
|
||||
if ei == -1 {
|
||||
d.Errorf("could not find literal string ending")
|
||||
}
|
||||
|
||||
bs := d.BytesLen(int(ei / 8))
|
||||
bs = eolEscapeRe.ReplaceAll(bs, []byte{})
|
||||
|
||||
return decodeText(bs)
|
||||
}
|
||||
|
||||
var whitespaceRE = regexp.MustCompile(`\s`)
|
||||
|
||||
func decodeStrHexString(d *decode.D) string {
|
||||
ei, _ := d.PeekFind(8, 8, -1, func(v uint64) bool {
|
||||
return v == '>'
|
||||
})
|
||||
if ei == -1 {
|
||||
d.Errorf("could not find literal string ending")
|
||||
}
|
||||
s := d.UTF8(int(ei / 8))
|
||||
// whitespace should be ignored inside hex string
|
||||
hexStr := whitespaceRE.ReplaceAllString(s, "")
|
||||
buf, err := hex.DecodeString(hexStr)
|
||||
if err != nil {
|
||||
d.IOPanic(err, "hex.DecodeString")
|
||||
}
|
||||
return decodeText(buf)
|
||||
}
|
||||
|
||||
func decodeValue(d *decode.D) any {
|
||||
var r any
|
||||
|
||||
if isWhitespace(byte(d.PeekUintBits(8))) {
|
||||
d.FieldStrFn("heading_whitespace", decodeStrWhitespace)
|
||||
}
|
||||
|
||||
// currently only returns value for types whose values we are interested in, atm
|
||||
// only dictionary, name, string and number so we can get stream length and filter
|
||||
|
||||
s := string(d.PeekBytes(2))
|
||||
switch {
|
||||
case (s[0] >= '0' && s[0] <= '9') || s[0] == '+' || s[0] == '-':
|
||||
num := d.FieldStrFn("value", decodeStrUntil(delimiterChars), strToNumber)
|
||||
r = num
|
||||
case s == "<<":
|
||||
d.FieldUTF8("start", 2)
|
||||
d.FieldArray("pairs", func(d *decode.D) {
|
||||
pairs := map[string]any{}
|
||||
for string(d.PeekBytes(2)) != ">>" {
|
||||
d.FieldStruct("pair", func(d *decode.D) {
|
||||
var keyV any
|
||||
var valueV any
|
||||
d.FieldStruct("key", func(d *decode.D) { keyV = decodeValue(d) })
|
||||
d.FieldStruct("value", func(d *decode.D) { valueV = decodeValue(d) })
|
||||
|
||||
if s, ok := keyV.(string); ok {
|
||||
pairs[s] = valueV
|
||||
}
|
||||
})
|
||||
}
|
||||
r = pairs
|
||||
})
|
||||
d.FieldUTF8("end", 2)
|
||||
case s[0] == '[':
|
||||
d.FieldUTF8("start", 1)
|
||||
d.FieldArray("objects", func(d *decode.D) {
|
||||
var objectsV []any
|
||||
for d.PeekUintBits(8) != ']' {
|
||||
var valueV any
|
||||
d.FieldStruct("object", func(d *decode.D) { valueV = decodeValue(d) })
|
||||
objectsV = append(objectsV, valueV)
|
||||
}
|
||||
r = objectsV
|
||||
})
|
||||
d.FieldUTF8("end", 1)
|
||||
case s[0] == '/':
|
||||
d.FieldUTF8("start", 1)
|
||||
name := d.FieldStrFn("value", decodeStrUntil(delimiterChars))
|
||||
r = name
|
||||
case s[0] == '<':
|
||||
d.FieldUTF8("start", 1)
|
||||
s := d.FieldStrFn("value", decodeStrHexString)
|
||||
d.FieldUTF8("end", 1)
|
||||
r = s
|
||||
case s[0] == '(':
|
||||
d.FieldUTF8("start", 1)
|
||||
s := d.FieldStrFn("value", decodeStrLiteralString)
|
||||
d.FieldUTF8("end", 1)
|
||||
r = s
|
||||
case s[0] == 'R':
|
||||
// TODO: should handle references differently?
|
||||
d.FieldUTF8("reference", 1)
|
||||
case s[0] == 't':
|
||||
d.FieldUTF8("value", 4, scalar.StrSym(true))
|
||||
case s[0] == 'f':
|
||||
d.FieldUTF8("value", 5, scalar.StrSym(false))
|
||||
case s[0] == 'n':
|
||||
d.FieldUTF8("value", 4, scalar.StrSym(nil))
|
||||
default:
|
||||
d.Fatalf("unknown type %q", s)
|
||||
}
|
||||
|
||||
if isWhitespace(byte(d.PeekUintBits(8))) {
|
||||
d.FieldStrFn("tailing_whitespace", decodeStrWhitespace)
|
||||
}
|
||||
|
||||
return r
|
||||
}
|
||||
|
||||
func pdfObjBodyDecode(d *decode.D) {
|
||||
d.FieldValueStr("type", "body")
|
||||
d.FieldArray("objects", func(d *decode.D) {
|
||||
d.FieldStruct("object", func(d *decode.D) {
|
||||
d.FieldStrFn("start", decodeLineStr)
|
||||
var dictV any
|
||||
d.FieldStruct("dictionary", func(d *decode.D) { dictV = decodeValue(d) })
|
||||
|
||||
log.Printf("dictV: %#+v\n", dictV)
|
||||
|
||||
endObj := false
|
||||
for !endObj {
|
||||
e := d.Pos()
|
||||
line := decodeLineStr(d)
|
||||
d.SeekAbs(e)
|
||||
|
||||
switch line {
|
||||
case "endobj":
|
||||
d.FieldStrFn("end", decodeLineStr)
|
||||
endObj = true
|
||||
case "stream":
|
||||
d.FieldStruct("stream", func(d *decode.D) {
|
||||
|
||||
d.FieldStrFn("start", decodeLineStr)
|
||||
|
||||
// TODO: proper string find
|
||||
ei, _ := d.PeekFind(64, 8, -1, func(v uint64) bool {
|
||||
return v == (0 |
|
||||
'e'<<56 |
|
||||
'n'<<48 |
|
||||
'd'<<40 |
|
||||
's'<<32 |
|
||||
't'<<24 |
|
||||
'r'<<16 |
|
||||
'e'<<8 |
|
||||
'a')
|
||||
})
|
||||
|
||||
// _ = d.FieldRawLen("data", ei)
|
||||
|
||||
if dv, _, _ := d.TryFieldFormatLen("data", ei, &imageFormat, nil); dv == nil {
|
||||
_ = d.FieldRawLen("data", ei)
|
||||
}
|
||||
|
||||
// d.FieldFormatReaderLen("uncompressed", dataLen, zlib.NewReader, iccProfileFormat)
|
||||
|
||||
// var rFn func(r io.Reader) io.Reader
|
||||
// switch compressionMethod {
|
||||
// case delfateMethod:
|
||||
// *bitio.Buffer implements io.ByteReader so hat deflate don't do own
|
||||
// // buffering and might read more than needed messing up knowing compressed size
|
||||
// rFn = func(r io.Reader) io.Reader { return flate.NewReader(r) }
|
||||
// // }
|
||||
|
||||
// if rFn != nil {
|
||||
// readCompressedSize, uncompressedBB, dv, _, err := d.TryFieldReaderRangeFormat("uncompressed", d.Pos(), ei, rFn, imageFormat, nil)
|
||||
// log.Printf("err: %#+v\n", err)
|
||||
// if uncompressedBB != nil {
|
||||
// if dv == nil {
|
||||
// d.FieldRootBitBuf("uncompressed", uncompressedBB)
|
||||
// }
|
||||
// d.FieldRawLen("compressed", readCompressedSize)
|
||||
// }
|
||||
// }
|
||||
|
||||
d.FieldStrFn("end", decodeLineStr)
|
||||
})
|
||||
default:
|
||||
d.Fatalf("bla")
|
||||
}
|
||||
}
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
func pdfDecode(d *decode.D) any {
|
||||
d.FieldStruct("header", func(d *decode.D) {
|
||||
d.FieldUTF8("version", 8, d.StrAssert(
|
||||
"%PDF-1.0",
|
||||
"%PDF-1.1",
|
||||
"%PDF-1.2",
|
||||
"%PDF-1.3",
|
||||
"%PDF-1.4",
|
||||
"%PDF-1.5",
|
||||
"%PDF-1.6",
|
||||
"%PDF-1.7",
|
||||
))
|
||||
|
||||
d.SeekAbs(0)
|
||||
|
||||
// TODO: has binary comment, byte >= 128
|
||||
|
||||
d.FieldArray("comments", func(d *decode.D) {
|
||||
for d.PeekUintBits(8) == '%' {
|
||||
d.FieldStrFn("comment", decodeLineStr)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
d.FieldArray("parts", func(d *decode.D) {
|
||||
for !d.End() {
|
||||
p := d.Pos()
|
||||
|
||||
line := decodeLineStr(d)
|
||||
d.SeekAbs(p)
|
||||
|
||||
switch {
|
||||
case strings.HasPrefix(line, "%"):
|
||||
d.FieldStruct("comment", func(d *decode.D) {
|
||||
d.FieldValueStr("type", "comment")
|
||||
d.FieldStrFn("line", decodeLineStr)
|
||||
})
|
||||
case strings.TrimSpace(line) == "":
|
||||
d.FieldStruct("whitespace", func(d *decode.D) {
|
||||
d.FieldValueStr("type", "whitespace")
|
||||
d.FieldStrFn("line", decodeLineStr)
|
||||
})
|
||||
case strings.HasSuffix(line, "obj"):
|
||||
d.FieldStruct("body", pdfObjBodyDecode)
|
||||
case line == "xref":
|
||||
d.FieldStruct("xref", func(d *decode.D) {
|
||||
d.FieldValueStr("type", "xref")
|
||||
d.FieldStrFn("start", decodeLineStr)
|
||||
d.FieldArray("lines", func(d *decode.D) {
|
||||
for {
|
||||
b := byte(d.PeekUintBits(8))
|
||||
if !(b >= '0' && b <= '9') {
|
||||
break
|
||||
}
|
||||
d.FieldStrFn("start", decodeLineStr)
|
||||
}
|
||||
})
|
||||
})
|
||||
case line == "trailer":
|
||||
d.FieldStruct("trailer", func(d *decode.D) {
|
||||
d.FieldValueStr("type", "trailer")
|
||||
d.FieldStrFn("start", decodeLineStr)
|
||||
d.FieldStruct("dictionary", func(d *decode.D) { decodeValue(d) })
|
||||
})
|
||||
case line == "startxref":
|
||||
d.FieldStruct("startxref", func(d *decode.D) {
|
||||
d.FieldValueStr("type", "startxref")
|
||||
d.FieldStrFn("start", decodeLineStr)
|
||||
d.FieldStrFn("offset", decodeLineStr, strToNumber)
|
||||
})
|
||||
default:
|
||||
d.Fatalf("unknown line %q", line)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
return nil
|
||||
}
|
12
go.mod
12
go.mod
@ -19,7 +19,7 @@ require (
|
||||
// bump: gomod-ergochat-readline /github\.com\/ergochat\/readline v(.*)/ https://github.com/ergochat/readline.git|*
|
||||
// bump: gomod-ergochat-readline command go get -d github.com/ergochat/readline@v$LATEST && go mod tidy
|
||||
// bump: gomod-ergochat-readline link "Release notes" https://github.com/ergochat/readline/releases/tag/v$LATEST
|
||||
github.com/ergochat/readline v0.1.0
|
||||
github.com/ergochat/readline v0.1.1
|
||||
|
||||
// bump: gomod-golang-snappy /github\.com\/golang\/snappy v(.*)/ https://github.com/golang/snappy.git|^0
|
||||
// bump: gomod-golang-snappy command go get -d github.com/golang/snappy@v$LATEST && go mod tidy
|
||||
@ -48,7 +48,7 @@ require (
|
||||
// bump: gomod-golang-x-crypto /golang\.org\/x\/crypto v(.*)/ https://github.com/golang/crypto.git|^0
|
||||
// bump: gomod-golang-x-crypto command go get -d golang.org/x/crypto@v$LATEST && go mod tidy
|
||||
// bump: gomod-golang-x-crypto link "Tags" https://github.com/golang/crypto/tags
|
||||
golang.org/x/crypto v0.22.0
|
||||
golang.org/x/crypto v0.23.0
|
||||
|
||||
// has no tags
|
||||
// go get -d golang.org/x/exp@master && go mod tidy
|
||||
@ -57,17 +57,17 @@ require (
|
||||
// bump: gomod-golang-x-net /golang\.org\/x\/net v(.*)/ https://github.com/golang/net.git|^0
|
||||
// bump: gomod-golang-x-net command go get -d golang.org/x/net@v$LATEST && go mod tidy
|
||||
// bump: gomod-golang-x-net link "Tags" https://github.com/golang/net/tags
|
||||
golang.org/x/net v0.24.0
|
||||
golang.org/x/net v0.25.0
|
||||
|
||||
// bump: gomod-golang-x-term /golang\.org\/x\/term v(.*)/ https://github.com/golang/term.git|^0
|
||||
// bump: gomod-golang-x-term command go get -d golang.org/x/term@v$LATEST && go mod tidy
|
||||
// bump: gomod-golang-x-term link "Tags" https://github.com/golang/term/tags
|
||||
golang.org/x/term v0.19.0
|
||||
golang.org/x/term v0.20.0
|
||||
|
||||
// bump: gomod-golang/text /golang\.org\/x\/text v(.*)/ https://github.com/golang/text.git|^0
|
||||
// bump: gomod-golang/text command go get -d golang.org/x/text@v$LATEST && go mod tidy
|
||||
// bump: gomod-golang/text link "Source diff $CURRENT..$LATEST" https://github.com/golang/text/compare/v$CURRENT..v$LATEST
|
||||
golang.org/x/text v0.14.0
|
||||
golang.org/x/text v0.15.0
|
||||
|
||||
// bump: gomod-gopkg.in/yaml.v3 /gopkg\.in\/yaml\.v3 v(.*)/ https://github.com/go-yaml/yaml.git|^3
|
||||
// bump: gomod-gopkg.in/yaml.v3 command go get -d gopkg.in/yaml.v3@v$LATEST && go mod tidy
|
||||
@ -79,6 +79,6 @@ require (
|
||||
github.com/itchyny/timefmt-go v0.1.5 // indirect
|
||||
github.com/mitchellh/reflectwalk v1.0.2 // indirect
|
||||
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e // indirect
|
||||
golang.org/x/sys v0.19.0 // indirect
|
||||
golang.org/x/sys v0.20.0 // indirect
|
||||
gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f // indirect
|
||||
)
|
||||
|
24
go.sum
24
go.sum
@ -2,8 +2,8 @@ github.com/BurntSushi/toml v1.3.2 h1:o7IhLm0Msx3BaB+n3Ag7L8EVlByGnpq14C4YWiu/gL8
|
||||
github.com/BurntSushi/toml v1.3.2/go.mod h1:CxXYINrC8qIiEnFrOxCa7Jy5BFHlXnUU2pbicEuybxQ=
|
||||
github.com/creasty/defaults v1.7.0 h1:eNdqZvc5B509z18lD8yc212CAqJNvfT1Jq6L8WowdBA=
|
||||
github.com/creasty/defaults v1.7.0/go.mod h1:iGzKe6pbEHnpMPtfDXZEr0NVxWnPTjb1bbDy08fPzYM=
|
||||
github.com/ergochat/readline v0.1.0 h1:KEIiAnyH9qGZB4K8oq5mgDcExlEKwmZDcyyocgJiABc=
|
||||
github.com/ergochat/readline v0.1.0/go.mod h1:o3ux9QLHLm77bq7hDB21UTm6HlV2++IPDMfIfKDuOgY=
|
||||
github.com/ergochat/readline v0.1.1 h1:C8Uuo3ybB23GWOt0uxmHbGzKM9owmtXary6Clrj84s0=
|
||||
github.com/ergochat/readline v0.1.1/go.mod h1:o3ux9QLHLm77bq7hDB21UTm6HlV2++IPDMfIfKDuOgY=
|
||||
github.com/golang/snappy v0.0.4 h1:yAGX7huGHXlcLOEtBnF4w7FQwA26wojNCwOYAEhLjQM=
|
||||
github.com/golang/snappy v0.0.4/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
|
||||
github.com/gomarkdown/markdown v0.0.0-20240328165702-4d01890c35c0 h1:4gjrh/PN2MuWCCElk8/I4OCKRKWCCo2zEct3VKCbibU=
|
||||
@ -25,18 +25,18 @@ github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e h1:fD57ERR4JtEqsWb
|
||||
github.com/niemeyer/pretty v0.0.0-20200227124842-a10e7caefd8e/go.mod h1:zD1mROLANZcx1PVRCS0qkT7pwLkGfwJo4zjcN/Tysno=
|
||||
github.com/wader/gojq v0.12.1-0.20240401131232-6c6bc364201a h1:P881Oecjt9FEXrwkGJ6UObJksxejJaF/fKq1ZfXpiVE=
|
||||
github.com/wader/gojq v0.12.1-0.20240401131232-6c6bc364201a/go.mod h1:qVrzkUdnBtJvM4twyRQ6xdziPSnSp35dLm4s/DN2iP4=
|
||||
golang.org/x/crypto v0.22.0 h1:g1v0xeRhjcugydODzvb3mEM9SQ0HGp9s/nh3COQ/C30=
|
||||
golang.org/x/crypto v0.22.0/go.mod h1:vr6Su+7cTlO45qkww3VDJlzDn0ctJvRgYbC2NvXHt+M=
|
||||
golang.org/x/crypto v0.23.0 h1:dIJU/v2J8Mdglj/8rJ6UUOM3Zc9zLZxVZwwxMooUSAI=
|
||||
golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8=
|
||||
golang.org/x/exp v0.0.0-20240325151524-a685a6edb6d8 h1:aAcj0Da7eBAtrTp03QXWvm88pSyOt+UgdZw2BFZ+lEw=
|
||||
golang.org/x/exp v0.0.0-20240325151524-a685a6edb6d8/go.mod h1:CQ1k9gNrJ50XIzaKCRR2hssIjF07kZFEiieALBM/ARQ=
|
||||
golang.org/x/net v0.24.0 h1:1PcaxkF854Fu3+lvBIx5SYn9wRlBzzcnHZSiaFFAb0w=
|
||||
golang.org/x/net v0.24.0/go.mod h1:2Q7sJY5mzlzWjKtYUEXSlBWCdyaioyXzRB2RtU8KVE8=
|
||||
golang.org/x/sys v0.19.0 h1:q5f1RH2jigJ1MoAWp2KTp3gm5zAGFUTarQZ5U386+4o=
|
||||
golang.org/x/sys v0.19.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||
golang.org/x/term v0.19.0 h1:+ThwsDv+tYfnJFhF4L8jITxu1tdTWRTZpdsWgEgjL6Q=
|
||||
golang.org/x/term v0.19.0/go.mod h1:2CuTdWZ7KHSQwUzKva0cbMg6q2DMI3Mmxp+gKJbskEk=
|
||||
golang.org/x/text v0.14.0 h1:ScX5w1eTa3QqT8oi6+ziP7dTV1S2+ALU0bI+0zXKWiQ=
|
||||
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
|
||||
golang.org/x/net v0.25.0 h1:d/OCCoBEUq33pjydKrGQhw7IlUPI2Oylr+8qLx49kac=
|
||||
golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
|
||||
golang.org/x/sys v0.20.0 h1:Od9JTbYCk261bKm4M/mw7AklTlFYIa0bIp9BgSm1S8Y=
|
||||
golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||
golang.org/x/term v0.20.0 h1:VnkxpohqXaOBYJtBmEppKUG6mXpi+4O6purfc2+sMhw=
|
||||
golang.org/x/term v0.20.0/go.mod h1:8UkIAJTvZgivsXaD6/pH6U9ecQzZ45awqEOzuCvwpFY=
|
||||
golang.org/x/text v0.15.0 h1:h1V/4gjBv8v9cjcR6+AR5+/cIYK5N/WAgiv4xlsEtAk=
|
||||
golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f h1:BLraFXnmrev5lT+xlilqcH8XK9/i0At2xKjWk4p6zsU=
|
||||
gopkg.in/check.v1 v1.0.0-20200227125254-8fa46927fb4f/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
|
@ -54,9 +54,6 @@ Same as recurse without argument.
|
||||
}
|
||||
};
|
||||
|
||||
def help($_): error("help must be alone or last in pipeline. ex: help(length) or ... | help");
|
||||
def help: help(null);
|
||||
|
||||
def _help_format_enrich($arg0; $f; $include_basic):
|
||||
( if $include_basic then
|
||||
.examples +=
|
||||
@ -76,6 +73,9 @@ def _help_format_enrich($arg0; $f; $include_basic):
|
||||
end
|
||||
);
|
||||
|
||||
# trailing help gets rewritten to _help_slurp, these are here to catch other variants
|
||||
def help($_): error("help must be alone or last in pipeline. ex: help(length) or ... | help");
|
||||
def help: help(null);
|
||||
def _help($arg0; $topic):
|
||||
( $topic
|
||||
| if . == "usage" then
|
||||
@ -236,7 +236,7 @@ def _help($arg0; $topic):
|
||||
| ($args | length) as $argc
|
||||
| if $args == null then
|
||||
# help
|
||||
( "Type expression to evaluate"
|
||||
( "Type jq expression to evaluate"
|
||||
, "help(...) Help for topic. Ex: help(mp4), help(\"mp4\")"
|
||||
, "\\t Completion"
|
||||
, "Up/Down History"
|
||||
|
2
pkg/interp/testdata/help.fqtest
vendored
2
pkg/interp/testdata/help.fqtest
vendored
@ -1,6 +1,6 @@
|
||||
$ fq -ni
|
||||
null> help
|
||||
Type expression to evaluate
|
||||
Type jq expression to evaluate
|
||||
help(...) Help for topic. Ex: help(mp4), help("mp4")
|
||||
\t Completion
|
||||
Up/Down History
|
||||
|
Loading…
Reference in New Issue
Block a user