neilotoole/sq
1
1
Fork 0
mirror of https://github.com/neilotoole/sq.git synced 2024-12-18 13:41:49 +03:00
Code Issues Actions Packages Projects Releases Wiki Activity
f814b918eb
sq/CHANGELOG.md
neilotoole d8a1992da7 Release v0.47.1
2024-01-29 13:04:31 -07:00

1106 lines
42 KiB
Markdown
Raw Blame History

# CHANGELOG
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
Breaking changes are annotated with ☢️, and alpha/beta features with 🐥.
## [v0.47.1] - 2024-01-29
This is a tiny bugfix release for a runtime issue on some Linux distros. See
the previous [`v0.47.0`](https://github.com/neilotoole/sq/releases/tag/v0.47.0) release
for recent headline features.
### Fixed
- `sq` [panicked](https://github.com/neilotoole/sq/actions/runs/7701355729/job/20987599862#step:3:383) on some Linux distros that don't include timezone data (`tzdata`). It's now
explicitly [imported](https://wawand.co/blog/posts/go-timezonedata-go115/).
## [v0.47.0] - 2024-01-29
This is a significant release, focused on improving i/o, responsiveness,
and performance. The headline features are [caching](https://sq.io/docs/source#cache)
of [ingested](https://sq.io/docs/source#ingest) data for [document sources](https://sq.io/docs/source#document-source)
such as CSV or Excel, and [download](https://sq.io/docs/source#download) caching for remote document sources.
There are a lot of under-the-hood changes, so please [open an issue](https://github.com/neilotoole/sq/issues/new/choose) if
you encounter any weirdness.
### Added
- Long-running operations (such as data [ingestion](https://sq.io/docs/source#ingest),
or file [download](https://sq.io/docs/source#download)) now result
in a progress bar being displayed. Display of the progress bar is controlled
by the new config options [`progress`](https://sq.io/docs/config#progress)
and [`progress.delay`](https://sq.io/docs/config#progressdelay). You can also use
the `--no-progress` flag to disable the progress bar.
- 👉 The progress bar is rendered on `stderr` and is always zapped from the terminal when command output begins.
It won't corrupt the output.
- [#307]: Ingested [document sources](https://sq.io/docs/source#document-source) (such as
[CSV](https://sq.io/docs/drivers/csv) or [Excel](https://sq.io/docs/drivers/xlsx))
now make use of an [ingest](https://sq.io/docs/source#ingest) cache DB. Previously, ingestion
of document source data occurred on each `sq` command. It is now a one-time cost; subsequent
use of the document source utilizes
the cache DB. Until, that is, the source document changes: then the ingest cache DB is invalidated and
ingested again. This is a significantly improved experience for large document sources.
- There are several new commands to interact with the cache (although you shouldn't need to):
- [`sq cache enable`](https://sq.io/docs/cmd/cache-enable) and
[`sq cache disable`](https://sq.io/docs/cmd/cache-disable) control cache usage.
You can also instead use the new [`ingest.cache`](https://sq.io/docs/config#ingestcache)
config option.
- [`sq cache clear`](https://sq.io/docs/cmd/cache-clear) clears the cache.
- [`sq cache location`](https://sq.io/docs/cmd/cache-location) prints the cache location on disk.
- [`sq cache stat`](https://sq.io/docs/cmd/cache-stat) shows stats about the cache.
- [`sq cache tree`](https://sq.io/docs/cmd/cache-location) shows a tree view of the cache.
- [#24]: The [download](https://sq.io/docs/source#download) mechanism for remote document sources (e.g. a CSV file at
[`https://sq.io/testdata/actor.csv`](https://sq.io/testdata/actor.csv)) has been completely
overhauled. Previously, `sq` would re-download the remote file on every command. Now, the
remote file is downloaded and [cached](https://sq.io/docs/source#cache) locally.
Subsequent `sq` invocations check for staleness of the cached download, and re-download if necessary.
- As part of the download revamp, new config options have been introduced:
- [`http.request.timeout`](https://sq.io/docs/config#httprequesttimeout) is the timeout for the initial response from the server, and
[`http.response.timeout`](https://sq.io/docs/config#httpresponsetimeout) is the timeout for reading the entire response body. We separate
these two timeouts because it's possible that the server responds quickly, but then
for a large file, the download takes too long.
- [`https.insecure-skip-verify`](https://sq.io/docs/config#httpsinsecure-skip-verify) controls
whether HTTPS connections verify the server's certificate. This is useful for remote files served
with a self-signed certificate.
- [`download.cache`](https://sq.io/docs/config#downloadcache) controls whether remote files are
cached locally.
- [`download.refresh.ok-on-err`](https://sq.io/docs/config#downloadrefreshok-on-err)
controls whether `sq` should continue with a stale cached download if an error
occurred while trying to refresh the download. This is a sort
of "Airplane Mode" for remote document sources: `sq` continues with the cached download when
the network is unavailable.
- There are two more new config options introduced as part of the above work.
- [`cache.lock.timeout`](https://sq.io/docs/config#cachelocktimeout) controls the time that
`sq` will wait for a lock on the cache DB. The cache lock is introduced for when you have
multiple `sq` commands running concurrently, and you want to avoid them stepping on each other.
- Similarly, [`config.lock.timeout`](https://sq.io/docs/config#configlocktimeout) controls the
timeout for acquiring the (newly-introduced) lock on `sq`'s config file. This helps prevent
issues with multiple `sq` processes mutating the config concurrently.
- `sq`'s own [logs](https://sq.io/docs/config#logging) previously outputted in JSON
format. Now there's a new [`log.format`](https://sq.io/docs/config#logformat) config option
that permits setting the log format to `json` or `text`. The `text` format is more human-friendly, and
is now the default.
### Changed
- Ingestion performance for [`json`](https://sq.io/docs/drivers/json#json) and
[`jsonl`](https://sq.io/docs/drivers/json#jsonl) sources has been significantly improved.
### Fixed
- Opening a DB connection now correctly honors [`conn.open-timeout`](https://sq.io/docs/config#connopen-timeout).
## [v0.46.1] - 2023-12-06
### Fixed
- `sq` sometimes failed to read from stdin if piped input was slow
to arrive. This is now fixed.
## [v0.46.0] - 2023-11-22
### Added
- [#338]: While `sq` has had [`group_by`](https://sq.io/docs/query#group_by) for some time,
somehow the [`having`](https://sq.io/docs/query#having) mechanism was never implemented. That's fixed.
```shell
$ sq '.payment | .customer_id, sum(.amount) | group_by(.customer_id) | having(sum(.amount) > 200)'
customer_id sum(.amount)
526 221.55
148 216.54
```
- [#340]: The [`group_by`](https://sq.io/docs/query#group_by) function
now has a synonym `gb`, and [`order_by`](https://sq.io/docs/query#order_by) now has synonym `ob`.
These synonyms are experimental 🧪. The motivation is to reduce typing, especially the underscore (`_`)
in both function names, but it's not clear that the loss of clarity is worth it. Maybe synonyms
`group` and `order` might be better? Feedback welcome.
```shell
# Previously
$ sq '.payment | .customer_id, sum(.amount) | group_by(.customer_id) | order_by(.customer_id)'
# Now
$ sq '.payment | .customer_id, sum(.amount) | gb(.customer_id) | ob(.customer_id)'
```
- [#340]: [`sq inspect`](https://sq.io/docs/cmd/inspect): added flag shorthand `-C`
for `--catalogs` and `-S` for `--schemata`. These were the only `inspect`
flags without shorthand.
## [v0.45.0] - 2023-11-21
### Changed
- [#335]: Previously, `sq` didn't handle decimal values correctly. It basically
shoved a decimal value into a `float` or `string` and hoped for the best.
[As is known](https://medium.com/@mayuribudake999/difference-between-decimal-and-float-eede050f6c9a),
floats are imprecise, and so we saw [unwanted behavior](https://github.com/neilotoole/sq/actions/runs/6932116521/job/18855333269#step:6:2345), e.g.
```shell
db_type_test.go:194:
Error Trace: D:/a/sq/sq/drivers/sqlite3/db_type_test.go:194
Error: Not equal:
expected: "77.77"
actual : "77.77000000000001"
```
Now, `sq` uses a dedicated [`Decimal`](https://github.com/shopspring/decimal) type end-to-end.
No precision is lost, and at the output end, the value is rendered with the correct precision.
There is a [proposal](https://github.com/golang/go/issues/30870) to add decimal support to
the Go [`database/sql`](https://pkg.go.dev/database/sql) package. If that happens, `sq` will happily
switch to that mechanism.
- 👉 A side effect of decimal support is that some output formats may now render decimal values
differently (i.e. correctly). In particular, Excel output should now render decimals as a number
(as opposed to a string), and with the precision defined in the database. Previously,
a database `NUMERIC(10,5)` value might have been rendered as `100.00`,
but will now accurately render `100.00000`.
## [v0.44.0] - 2023-11-20
### Added
- New [SLQ](https://sq.io/docs/concepts#slq) function [`rownum()`](https://sq.io/docs/query#rownum) that returns the one-indexed
row number of the current record.
```shell
$ sq '.actor | rownum(), .actor_id, .first_name | order_by(.first_name)'
rownum() actor_id first_name
1 71 ADAM
2 132 ADAM
3 165 AL
```
- [`sq inspect`](https://sq.io/docs/inspect) has two new flags:
- [`--schemata`](https://sq.io/docs/inspect#schemata): list the source's schemas
```shell
$ sq inspect @sakila/pg12 --schemata -y
- schema: information_schema
catalog: sakila
owner: sakila
- schema: public
catalog: sakila
owner: sakila
active: true
```
- [`--catalogs`](https://sq.io/docs/inspect#catalogs): list the source's catalogs
```shell
$ sq inspect @sakila/pg12 --catalogs
CATALOG
postgres
sakila
```
### Fixed
- [`sq version`](https://sq.io/docs/cmd/version) now honors option
[`format.datetime`](https://sq.io/docs/config#formatdatetime) when outputting build timestamp.
- Fixed a fairly nasty bug that prevented correct rendering of SLQ functions nested inside
an expression.
### Changed
- The `--exec` and `--query` flags for [`sq sql`](https://sq.io/docs/cmd/sql) were removed in
the preceding release ([v0.43.1]).
That was probably a bit hasty, especially because it's possible those flags _could_ be reintroduced
when the _query vs exec_ situation is figured out. So, those two flags are now restored, in
that their use won't cause an error, but they've been hidden from command help, and remain no-op.
## [v0.43.1] - 2023-11-19
### Added
- Related to [#270], the output of [`sq inspect`](https://sq.io/docs/inspect) now includes the
source's catalog (in JSON and YAML output formats).
### Fixed
- MySQL driver didn't populate all expected values for [`sq inspect --overview`](https://sq.io/docs/inspect#source-overview).
### Changed
- ☢️ Removed unused `--exec` and `--query` flags from [`sq sql`](https://sq.io/docs/cmd/sql) command.
## [v0.43.0] - 2023-11-18
### Added
- [#270]: Flag [`--src.schema`](https://sq.io/docs/cmd/sq/#override-active-schema) permits switching the source's schema (and catalog)
for the duration of the command. The flag is supported for the
[`sq`](https://sq.io/docs/cmd/sq#override-active-schema), [`sql`](https://sq.io/docs/cmd/sql/#active-source--schema)
and [`inspect`](https://sq.io/docs/inspect#override-active-schema) commands.
- New SLQ functions [`catalog()`](https://sq.io/docs/query#catalog) and
[`schema()`](https://sq.io/docs/query#schema) return the catalog and schema of the DB connection.
- The SLQ [`unique`](https://sq.io/docs/query#unique) function now has a synonym `uniq`.
### Changed
- `sq src --text` now outputs only the handle of the active source. Previously it
also printed the driver type and short location. Instead use `sq src --text --verbose` to
see those details.
## [v0.42.1] - 2023-09-10
### Fixed
- [#308]: Fix to allow build on [32-bit systems](https://github.com/void-linux/void-packages/pull/45023).
Thanks [@icp](https://github.com/icp1994).
## [v0.42.0] - 2023-08-22
### Added
- 🐥 [#279]: The SQLite [driver](https://sq.io/docs/drivers/sqlite) now has initial support
for several SQLite [extensions](https://sq.io/docs/drivers/sqlite#extensions)
baked in, including [Virtual Table](https://www.sqlite.org/vtab.html) and [FTS5](https://www.sqlite.org/fts5.html).
Note that this is an early access release of extensions support. Please [open an issue](https://github.com/neilotoole/sq/issues/new/choose) if
you discover something bad.
## [v0.41.1] - 2023-08-20
### Fixed
- `sq version` was missing a newline in its output.
## [v0.41.0] - 2023-08-20
This release is heavily focused on improvements to Microsoft Excel support.
The underlying Excel library has been changed from [`tealeg/xlsx`](https://github.com/tealeg/xlsx)
to [`qax-os/excelize`](https://github.com/qax-os/excelize), largely because
`tealeg/xlsx` is no longer actively maintained. Thus, both the [XLSX output writer](https://sq.io/docs/output/#xlsx)
and the [XLSX driver](https://sq.io/docs/drivers/xlsx) have been rewritten. There should be some performance
improvements, but it's also possible that the rewrite introduced bugs. If you
discover anything strange, please [open an issue](https://github.com/neilotoole/sq/issues/new/choose).
### Added
- [#99]: The [CSV](https://sq.io/docs/drivers/csv) and [XLSX](https://sq.io/docs/drivers/xlsx)
drivers can now handle duplicate header column names in the ingest data.
For example, given a CSV file:
```csv
actor_id,first_name,actor_id
1,PENELOPE,1
2,NICK,2
```
The columns will be renamed to:
```csv
actor_id,first_name,actor_id_1
```
The renaming behavior is controlled by a new option [`ingest.column.rename`](https://sq.io/docs/config#ingestcolumnrename)
This new option is effectively the ingest counterpart of the existing output option
[`result.column.rename`](https://sq.io/docs/config#resultcolumnrename).
- [#191]: The [XLSX](https://sq.io/docs/drivers/xlsx) driver now [detects](https://sq.io/docs/drivers/xlsx#header-row) header rows, like
the CSV driver already does. Thus, you now typically don't need to specify
the `--ingest.header` flag for Excel files. However, the option remains available
in case `sq` can't figure it out for a particular file.
- The Excel writer has three new config options for controlling date/time output.
Note that these format strings are distinct from [`format.datetime`](https://sq.io/docs/config#formatdatetime)
and friends, because Excel has its own format string mechanism.
- [`format.excel.datetime`](https://sq.io/docs/config#formatexceldatetime): Controls datetime format, e.g. `2023-08-03 16:07:01`.
- [`format.excel.date`](https://sq.io/docs/config#formatexceldatetime): Controls date-only format, e.g. `2023-08-03`.
- [`format.excel.time`](https://sq.io/docs/config#formatexceldatetime): Controls time-only format, e.g. `4:07 pm`.
- The ingest [kind detectors](https://sq.io/docs/detect#kinds) (e.g. for `CSV` or `XLSX`)
now detect more [date & time formats](https://sq.io/docs/detect#datetime-formats) as `kind.Datetime`, `kind.Date`, and `kind.Time`.
- If an error occurs when the output format is `text`, a stack trace is printed
to `stderr` when the command is executed with `--verbose` (`-v`).
- There's a new option [`error.format`](https://sq.io/docs/config#errorformat) that controls error output format independent
of the main [`format`](https://sq.io/docs/config#format) option . The `error.format` value must be one of `text` or `json`.
### Changed
- ☢️ The default Excel date format has changed. Previously
the format was `11/9/89`, and now it is `1989-11-09`. The same applies
to datetimes, e.g. `11/9/1989 00:00:00` becomes `1989-11-09 00:00`.
This change is made to reduce ambiguity and confusion.
`sq` uses a [library](https://github.com/qax-os/excelize)
to interact with Excel files, and it seems that the library chooses a particular format
by default (`11/9/89`). There are several paths we could take here:
1. Interrogate the OS, and use the OS locale date format.
2. Stick with the library default `11/9/89`.
3. Pick a default other than `11/9/89`.
We pick the third option. The first option (locale-dependent)
is excluded because, as a general rule, we want `sq` to produce the same
output regardless of locale/system settings. We exclude the second option
because month/day confuses most of the world. Thus, we're left with picking a
default, and `1989-11-09` is the format used in
[RFC3339](https://datatracker.ietf.org/doc/html/rfc3339) and friends.
Whether this is the correct (standard?) approach is still unclear, and
feedback is welcome. However, the user can make use of the new config options
([`format.excel.datetime`](https://sq.io/docs/config#formatexceldatetime) etc.)
to customize the format as they see fit.
- The XLSX writer now outputs header rows in **bold text**.
- ☢️ The XLSX writer now outputs blob (`bytes`) cell data as a base64-encoded string,
instead of raw bytes.
### Fixed
- Fixed bug where source-specific config wasn't being propagated.
## [v0.40.0] - 2023-07-03
This release features a complete overhaul of the [`join`](https://sq.io/docs/query#joins)
mechanism.
### Added
- [#277]: A table selector can now have an alias. This in and of itself is not
particularly useful, but it's a building block for [multiple joins](https://github.com/neilotoole/sq/issues/12).
```shell
$ sq '@sakila | .actor:a | .a.first_name'
```
- New option `result.column.rename` that exposes a template used to rename
result set column names before display. The primary use case is to de-duplicate
columns names on a `SELECT * FROM tbl1 JOIN tbl2`, where `tbl1` and `tbl2`
have clashing column names ([docs](https://sq.io/docs/config#resultcolumnrename)).
- [#157]: Previously only `join` (`INNER JOIN`) was available: now the rest of
the join types such as `left_outer_join`, `cross_join`, etc. are
implemented ([docs](https://sq.io/docs/query#join-types)).
### Changed
- ☢️ [#12]: The table [join](https://sq.io/docs/query#joins) mechanism has been
completely overhauled. Now there's support for multiple joins. See [docs](https://sq.io/docs/query#joins).
```shell
# Previously, only a single join was possible
$ sq '.actor, .film_actor | join(.actor_id)'
# Now, an arbitrary number of joins
$ sq '.actor | join(.film_actor, .actor_id) | join(.film, .film_id)'
```
- ☢️ The alias for `--jsonl` (JSON Lines) has been changed to `-J`.
### Fixed
- Config options weren't being propagated correctly to all parts of the code.
## [v0.39.1] - 2023-06-22
### Fixed
- Bug with `sq version` output on Windows.
## [v0.39.0] - 2023-06-22
### Added
- [#263]: `sq version` now supports `--yaml` output.
- [#263]: `sq version` now outputs host OS details with `--verbose`, `--json`
and `--yaml` flags. The motivation behind this is bug submission: we want
to know which OS/arch the user is on. E.g. for `sq version -j`:
```json
{
"version": "v0.38.1",
"commit": "eedc11ec46d1f0e78628158cc6fd58850601d701",
"timestamp": "2023-06-21T11:41:34Z",
"latest_version": "v0.39.0",
"host": {
"platform": "darwin",
"arch": "arm64",
"kernel": "Darwin",
"kernel_version": "22.5.0",
"variant": "macOS",
"variant_version": "13.4"
}
}
```
- [#263]: The output of `sq inspect` and `sq inspect -v` has been refactored
significantly, and should now be easier to work with ([docs](https://sq.io/docs/inspect)).
## [v0.38.1] - 2023-06-19
### Fixed
- [#261]: The JSON writer (`--json`) could get deadlocked when a record contained
a large amount of data, triggering an internal `Flush()` (which is mutex-guarded)
from within the mutex-guarded `WriteRecords()` method.
## [v0.38.0] - 2023-06-18
This release has significant improvements (and breaking changes)
to SLQ (`sq`'s query language).
### Changed
- ☢️ [#254]: The formerly-implicit "WHERE" mechanism now requires an explicit `where()` function.
This, alas, is a fairly big breaking change. But it's necessary to remove an ambiguity roadblock.
See discussion in the [issue](https://github.com/neilotoole/sq/issues/254).
```shell
# Previously
$ sq '.actor | .actor_id <= 2'
# Now
$ sq '.actor | where(.actor_id <= 2)'
```
- [#256]: Column-only queries are now possible. This has the neat side effect
that `sq` can now be used as a calculator.
```shell
$ sq 1+2
1+2
3
```
You may want to use `--no-header` (`-H`) when using `sq` as a calculator.
```shell
$ sq -H 1+2
3
$ sq -H '(1+2)*3'
9
```
### Fixed
- Literals can now be selected ([docs](https://sq.io/docs/query#select-literal)).
```shell
$ sq '.actor | .first_name, "X":middle_name, .last_name | .[0:2]'
first_name middle_name last_name
PENELOPE X GUINESS
NICK X WAHLBERG
```
- Lots of expressions that previously failed badly, now work.
```shell
$ sq '.actor | .first_name, (1+2):addition | .[0:2]'
first_name addition
PENELOPE 3
NICK 3
```
- [#258]: Column aliases can now be arbitrary strings, instead of only a
valid identifier.
```shell
# Previously only valid identifier allowed
$ sq '.actor | .first_name:given_name | .[0:2]'
given_name
PENELOPE
NICK
# Now, any arbitrary string can be used
$ sq '.actor | .first_name:"Given Name" | .[0:2]'
Given Name
PENELOPE
NICK
```
## [v0.37.1] - 2023-06-15
### Fixed
- [#252]: Handle `*uint64` returned from DB.
## [v0.37.0] - 2023-06-13
### Added
- [#244]: Shell completion for `sq add LOCATION`. See [docs](https://sq.io/docs/source#location-completion).
## [v0.36.2] - 2023-05-27
### Changed
- ☢️ [Proprietary database functions](https://sq.io/docs/query#proprietary-functions) are now
invoked by prefixing the function name with an underscore. For example:
```shell
# mysql "date_format" func
$ sq '@sakila/mysql | .payment | _date_format(.payment_date, "%m")'
# Postgres "date_trunc" func
$ sq '@sakila/postgres | .payment | _date_trunc("month", .payment_date)'
```
## [v0.36.1] - 2023-05-26
### Fixed
- `sq diff`: Renamed `--count` flag to `--counts` as intended.
## [v0.36.0] - 2023-05-25
The major feature is the long-gestating [`sq diff`](https://sq.io/docs/diff).
## Added
- [#229]: `sq diff` compares two sources, or tables.
- `sq inspect --dbprops` is a new mode that returns only the DB properties.
Relatedly, the properties mechanism is now implemented for all four supported
DB types (previously, it was only implemented for Postgres and MySQL).
- [CSV](https://sq.io/docs/output#csv-tsv) format now colorizes output.
## Changed
- `sq inspect -v` previously returned DB properties in a field named `db_variables`.
This field has been renamed to `db_properties`. The renaming reflects the fact
that some of those properties aren't really variables in the sense that they
can be modified (e.g. DB server version or such).
- The structure of the former `db_variables` (now `db_properties`) field has
changed. Previously it was an array of `{"name": "XX", "value": "YY"}` values,
but now is a map, where the keys are strings, and the values can be either
a scalar (`bool`, `int`, `string`, etc.), or a nested value such as an array
or map. This change is made because some databases (e.g. SQLite) feature
complex data in some property values.
- CSV format now renders byte sequences as `[777 bytes]` instead of dumping
the raw bytes.
- ☢️ TSV format (`--tsv`) no longer has a shorthand form `-T`. Apparently that
shorthand wasn't used much, and `-T` is needed elsewhere.
- ☢️ Likewise, `--xml` no longer has shorthand `-X`. And `--markdown` has lost alias `--md`.
- In addition to the format flags `--text`, `--json`, etc., there is now
a `--format=FORMAT` flag, e.g. `--format=json`. This will allow `sq` to
continue to expand the number of output formats, without needing to have
a dedicated flag for each format.
## Fixed
- `sq config edit @source` was failing to save any edits.
## [v0.35.0] - 2023-05-10
### Added
- [#8]: Results can now be output in [YAML](https://sq.io/docs/output#yaml).
### Fixed
- `sq config get OPT --text` now prints only the value, not `KEY VALUE`.
If you want to see key and value, consider using `--yaml`, or `--text --verbose`.
## [v0.34.2] - 2023-05-08
### Fixed
- Both `--markdown` and the alias `--md` are now supported.
## [v0.34.1] - 2023-05-07
### Fixed
- Fixed a minor issue where `sq ls -jv` and `sq ls -yv` produced no output
if config contained no explicitly set options.
## [v0.34.0] - 2023-05-07
This release significantly overhauls `sq`'s config mechanism ([#199]).
For an overview, see the new [config docs](https://sq.io/docs/config).
Alas, this release has several minor breaking changes ☢️.
### Added
- `sq config ls` shows config.
- `sq config get` gets individual config option.
- `sq config set` sets config values.
- `sq config edit` edits config.
- Editor can be specified via `$EDITOR` or `$SQ_EDITOR`.
- `sq config location` prints the location of the config dir.
- `--config` flag is now honored globally.
- Many more knobs are exposed in config.
- Logging is much more configurable. There are new knobs:
```shell
$ sq config set log true
$ sq config set log.level INFO
$ sq config set log.file /var/log/sq.log
```
There are also equivalent flags (`--log`, `--log.file` and `--log.level`) and
envars (`SQ_LOG`, `SQ_LOG_FILE` and `SQ_LOG_LEVEL`).
- Several more commands support YAML output:
- [`sq group`](https://sq.io/docs/cmd/group)
- [`sq ls`](https://sq.io/docs/cmd/ls)
- [`sq mv`](https://sq.io/docs/cmd/mv)
- [`sq rm`](https://sq.io/docs/cmd/rm)
- [`sq src`](https://sq.io/docs/cmd/src)
### Changed
- The structure of `sq`'s config file (`sq.yml`) has changed. The config
file is automatically upgraded when using the new version.
- The default location of the `sq` log file has changed. The new location
is platform-dependent. Use `sq config get log.file -v` to view the location,
or `sq config set log.file /path/to/sq.log` to set it.
- ☢️ Envar `SQ_CONFIG` replaces `SQ_CONFIGDIR`.
- ☢️ Envar `SQ_LOG_FILE` replaces `SQ_LOGFILE`.
- ☢️ Format flag `--table` is renamed to `--text`. This is changed because while the
output is mostly in table format, sometimes it's just plain text. Thus
`table` was not quite accurate.
- ☢️ The flag to explicitly specify a driver when piping input to `sq` has been
renamed from `--driver` to `--ingest.driver`. This change aligns
the naming of the ingest options and reduces ambiguity.
```shell
# previously
$ cat mystery.data | sq --driver=csv '.data'
# now
$ cat mystery.data | sq --ingest.driver=csv '.data'
```
- ☢️ `sq add` no longer has the generic `--opts x=y` mechanism. This flag was
ambiguous and confusing. Instead, use explicit option flags.
```shell
# previously
$ sq add ./actor.csv --opts=header=false
# now
$ sq add ./actor.csv --ingest.header=false
```
- ☢️ The short form of the `sq add --handle` flag has been changed from `-h` to
`-n`. While this is not ideal, the `-h` shorthand is already in use everywhere
else as the short form of `--header`.
```shell
# previously
$ sq add ./actor.csv -h @actor
# now
$ sq add ./actor.csv -n @actor
```
- ☢️ The `--pretty` flag has been removed. Its only previous use was with the
`json` format, where if `--pretty=false` would output the JSON in compact form.
To better align with jq, there is now a `--compact` / `-c` flag that behaves
identically to jq.
- ☢️ Because of the above `--compact` / `-c` flag, the short form of the `--csv`
flag is changing from `-c` to `-C`. It's an unfortunate situation, but alignment
with jq's behavior is an overarching principle that justifies the change.
## [v0.33.0] - 2023-04-15
The headline feature is [source groups](https://sq.io/docs/source#groups).
This is the biggest change to the `sq` CLI in some time, and should
make working with lots of sources much easier.
### Added
- [#192]: `sq` now has a mechanism to group sources. A source handle can
now be scoped. For example, instead of `@sakila_prod`, `@sakila_staging`, etc,
you can use `@prod/sakila`, `@staging/sakila`. Use `sq group prod` to
set the active group (which `sq ls` respects). See [docs](https://sq.io/docs/source#groups).
- `sq group GROUP` sets the active group to `GROUP`.
- `sq group` returns the active group (default is `/`, the root group).
- `sq ls GROUP` lists the sources in `GROUP`.
- `sq ls --group` (or `sq ls -g`) lists all groups.
- `sq mv` moves/renames sources and groups.
### Changed
- `sq ls` now shows the active item in a distinct color. It no longer adds
an asterisk to the active item.
- `sq ls` now sorts alphabetically when using `--table` format.
- `sq ls` now shows the sources in the active group only. But note that
the default active group is `/` (the root group), so the default behavior
of `sq ls` is the same as before.
- `sq add hello.csv` will now generate the handle `@hello` instead of `@hello_csv`.
On a second invocation, it will return `@hello1` instead of `@hello_csv_1`. Why
this change? Well, with the availability of the source group mechanism, the `_` character
in the handle somehow looked ugly. And more importantly, `_` is a relative pain to type.
- `sq ping` has changed to support groups. Instead of `sq ping --all`, you can
do `sq ping GROUP`, e.g. `sq ping /`.
## [v0.32.0] - 2023-04-09
### Added
- [#187]: For `csv` sources, `sq` will now try to auto-detect if the CSV file
has a header row or not. Previously, this needed to be explicitly specified
via an awkward syntax:
```shell
$ sq add ./actor.csv --opts=header=true
````
This change makes working with CSV files significantly lower friction.
A command like the below now almost always works as expected:
```shell
$ cat ./actor.csv | sq .data
```
Support for Excel/XLSX header detection is in [#191].
### Fixed
- `sq` is now better at detecting the (data) kind of CSV fields. It now more
accurately distinguishes between `Decimal` and `Int`, and knows how to
handle `Datetime`.
- [#189]: `sq` now treats CSV empty fields as `NULL`.
## [v0.31.0] - 2023-03-08
### Added
- [#173]: Predefined variables via `--arg`
flag ([docs](https://sq.io/docs/query#predefined-variables)):
```shell
$ sq --arg first TOM '.actor | .first_name == $first'
```
### Changes
- Use `--md` instead of `--markdown` for outputting Markdown.
### Fixed
- [#185]: `sq inspect` now better handles "too many connections" situations.
- `go.mod`: Moved to `jackc/pgx` `v5`.
- Refactor: switched to [`slog`](https://pkg.go.dev/golang.org/x/exp/slog) logging library.
## [v0.30.0] - 2023-03-27
### Added
- [#164]: Implemented `unique` function ([docs](https://sq.io/docs/query#unique)):
```shell
$ sq '.actor | .first_name | unique'
```
This is equivalent to:
```sql
SELECT DISTINCT first_name FROM actor
```
- Implemented `count_unique` function ([docs](https://sq.io/docs/query#count_unique)).
```shell
$ sq '.actor | count_unique(.first_name)'
```
### Changed
- The `count` function has been changed ([docs](https://sq.io/docs/query#count))
- Added no-args version: `.actor | count` equivalent to `SELECT COUNT(*) AS "count" FROM "actor"`.
- ☢️ The "star" version (`.actor | count(*)`) is no longer supported; use the
naked version instead.
- Function columns are now named according to the `sq` token, not the SQL token.
```shell
# previous behavior
$ sq '.actor | max(.actor_id)'
max("actor_id")
200
# now
$ sq '.actor | max(.actor_id)'
max(.actor_id)
200
```
## [v0.29.0] - 2023-03-26
### Added
- [#162]: `group_by` now accepts [function arguments](https://sq.io/docs/query#group_by).
### Changed
- Renamed `groupby` to `group_by` to match jq.
- Renamed `orderby` to `order_by` to match jq.
## [v0.28.0] - 2023-03-26
### Added
- [#160]: Use `groupby()` to group results. See [query guide](https://sq.io/docs/query#group_by).
## [v0.27.0] - 2023-03-25
### Added
- [#158]: Use `orderby()` to order results. See [query guide](https://sq.io/docs/query#order_by).
## [v0.26.0] - 2023-03-22
### Added
- [#98]: Whitespace is now allowed in SLQ selector names. You can
do `@sakila | ."film actor" | ."actor id"`.
### Fixed
- [#155]: `sq inspect` now populates `schema` field in JSON for MySQL,
SQLite, and SQL Server (Postgres already worked).
## [v0.25.1] - 2023-03-19
### Fixed
- [#153]: Improved formatting of text table with long lines.
## [v0.25.0] - 2023-03-19
### Added
- [#15]: Column Aliases. You can now specify an alias for a column (or column expression
such as a function). For example: `sq '.actor | .first_name:given_name`, or `sq .actor | count(*):quantity`.
- [#151]: `sq add` now has a `--active` flag, which immediately sets the new source
as the active source.
## [v0.24.4] - 2023-03-15
### Fixed
- Fixed typos in `sq sql` command help.
## [v0.24.3] - 2023-03-14
### Added
- When a CSV source has explicit column names (via `--opts cols=A,B,C`), `sq` now verifies
that the CSV data record field count matches the number of explicit columns.
## [v0.24.2] - 2023-03-13
### Fixed
- [#142]: Improved error handling when Postgres `current_schema()` is unavailable.
## [v0.24.1] - 2023-03-11
### Fixed
- [#144]: Handle corrupted config active source.
## [v0.24.0] - 2022-12-31
### Added
- `sq ping` now respects `--json` flag.
### Fixed
- Improved handling of file paths on Windows.
## [v0.23.0] - 2022-12-31
### Added
- `sq ls` now respects `--json` flag.
- `sq rm` now respects `--json` flag.
- `sq add` now respects `--json` flag.\`
- CI pipeline now verifies install packages after publish.
### Changed
- `sq rm` can delete multiple sources.
- `sq rm` doesn't print output unless `--verbose`.
- Redacted snipped is now `xxxxx` instead of `****`, to match stdlib `url.URL.Redacted()`.
### Fixed
- Fixed crash on Fedora systems (needed `--tags=netgo`).
## [v0.21.3] - 2022-12-30
### Added
- `sq version` respects `--json` flag.
- `sq version` respects `--verbose` flag.
- `sq version` shows `latest_version` info when `--verbose` and there's a newer version available.
### Changed
- `sq version` shows less info when `--verbose` is not set.
## [v0.20.0] - 2022-12-29
### Added
- `sq` now generates manpages (and installs them).
## [v0.19.0] - 2022-12-29
### Added
- Installer for [Arch Linux](https://archlinux.org),
via [Arch User Repository](https://aur.archlinux.org).
## [v0.18.2] - 2022-12-25
### Added
- The build pipeline now produces `.apk` packages for [Alpine Linux](https://www.alpinelinux.org),
and `install.sh` has been updated accordingly. However, the `.apk` files
are not yet published to a repository, so it's necessary to run `apk` against
the downloaded `.apk` file (`install.sh` does this for you).
## [v0.18.0] - 2022-12-24
### Added
- [#95]: `sq add` now has a `--password` (`-p`) flag that prompts the user for the data source
password, instead of putting it in the location string. It will also read from stdin
if there's input there.
## [v0.17.0] - 2022-12-23
### Changed
- More or less every `go.mod` dependency has been updated to latest. This includes
drivers for `sqlite` and `sqlserver`. The driver updates led to some broken
things, which have been fixed.
## [v0.16.1] - 2022-12-23
### Fixed
- [#123]: Shell completion is better behaved when a source is offline.
## [v0.16.0] - 2022-12-16
### Added
- `--verbose` flag is now global
- `install.sh` install script.
### Changed
- Improved GH workflow
- `sq inspect` shows less output by default (use `-v` to restore previous behavior)
### Fixed
- `sq inspect` can now deal with Postgres sources that have null values for constraint fields
## [v0.15.11] - 2022-11-06
### Changed
- Yet more changes to GitHub workflow.
## [v0.15.4] - 2021-09-18
### Changed
- Bug fixes
## [v0.15.3] - 2021-03-13
### Changed
- [#91]: MySQL driver options no longer stripped
## [v0.15.2] - 2021-03-08
### Changed
- [#89]: Bug with SQL generated for joins.
[#8]: https://github.com/neilotoole/sq/issues/8
[#12]: https://github.com/neilotoole/sq/issues/12
[#15]: https://github.com/neilotoole/sq/issues/15
[#24]: https://github.com/neilotoole/sq/issues/24
[#89]: https://github.com/neilotoole/sq/pull/89
[#91]: https://github.com/neilotoole/sq/pull/91
[#95]: https://github.com/neilotoole/sq/issues/93
[#98]: https://github.com/neilotoole/sq/issues/98
[#99]: https://github.com/neilotoole/sq/issues/99
[#123]: https://github.com/neilotoole/sq/issues/123
[#142]: https://github.com/neilotoole/sq/issues/142
[#144]: https://github.com/neilotoole/sq/issues/144
[#151]: https://github.com/neilotoole/sq/issues/151
[#153]: https://github.com/neilotoole/sq/issues/153
[#155]: https://github.com/neilotoole/sq/issues/155
[#157]: https://github.com/neilotoole/sq/issues/157
[#158]: https://github.com/neilotoole/sq/issues/158
[#160]: https://github.com/neilotoole/sq/issues/160
[#162]: https://github.com/neilotoole/sq/issues/162
[#164]: https://github.com/neilotoole/sq/issues/164
[#173]: https://github.com/neilotoole/sq/issues/173
[#185]: https://github.com/neilotoole/sq/issues/185
[#187]: https://github.com/neilotoole/sq/issues/187
[#189]: https://github.com/neilotoole/sq/issues/189
[#191]: https://github.com/neilotoole/sq/issues/191
[#192]: https://github.com/neilotoole/sq/issues/192
[#199]: https://github.com/neilotoole/sq/issues/199
[#229]: https://github.com/neilotoole/sq/issues/229
[#244]: https://github.com/neilotoole/sq/issues/244
[#252]: https://github.com/neilotoole/sq/issues/252
[#254]: https://github.com/neilotoole/sq/issues/254
[#256]: https://github.com/neilotoole/sq/issues/256
[#258]: https://github.com/neilotoole/sq/issues/258
[#261]: https://github.com/neilotoole/sq/issues/261
[#263]: https://github.com/neilotoole/sq/issues/263
[#270]: https://github.com/neilotoole/sq/issues/270
[#277]: https://github.com/neilotoole/sq/issues/277
[#279]: https://github.com/neilotoole/sq/issues/279
[#307]: https://github.com/neilotoole/sq/issues/307
[#308]: https://github.com/neilotoole/sq/pull/308
[#335]: https://github.com/neilotoole/sq/issues/335
[#338]: https://github.com/neilotoole/sq/issues/338
[#340]: https://github.com/neilotoole/sq/pull/340
[v0.15.2]: https://github.com/neilotoole/sq/releases/tag/v0.15.2
[v0.15.3]: https://github.com/neilotoole/sq/compare/v0.15.2...v0.15.3
[v0.15.4]: https://github.com/neilotoole/sq/compare/v0.15.3...v0.15.4
[v0.15.11]: https://github.com/neilotoole/sq/compare/v0.15.4...v0.15.11
[v0.16.0]: https://github.com/neilotoole/sq/compare/v0.15.11...v0.16.0
[v0.16.1]: https://github.com/neilotoole/sq/compare/v0.16.0...v0.16.1
[v0.17.0]: https://github.com/neilotoole/sq/compare/v0.16.1...v0.17.0
[v0.18.0]: https://github.com/neilotoole/sq/compare/v0.17.0...v0.18.0
[v0.18.2]: https://github.com/neilotoole/sq/compare/v0.18.0...v0.18.2
[v0.19.0]: https://github.com/neilotoole/sq/compare/v0.18.2...v0.19.0
[v0.20.0]: https://github.com/neilotoole/sq/compare/v0.19.0...v0.20.0
[v0.21.3]: https://github.com/neilotoole/sq/compare/v0.20.0...v0.21.3
[v0.23.0]: https://github.com/neilotoole/sq/compare/v0.21.3...v0.23.0
[v0.24.0]: https://github.com/neilotoole/sq/compare/v0.23.0...v0.24.0
[v0.24.1]: https://github.com/neilotoole/sq/compare/v0.24.0...v0.24.1
[v0.24.2]: https://github.com/neilotoole/sq/compare/v0.24.1...v0.24.2
[v0.24.3]: https://github.com/neilotoole/sq/compare/v0.24.2...v0.24.3
[v0.24.4]: https://github.com/neilotoole/sq/compare/v0.24.3...v0.24.4
[v0.25.0]: https://github.com/neilotoole/sq/compare/v0.24.4...v0.25.0
[v0.25.1]: https://github.com/neilotoole/sq/compare/v0.25.0...v0.25.1
[v0.26.0]: https://github.com/neilotoole/sq/compare/v0.25.1...v0.26.0
[v0.27.0]: https://github.com/neilotoole/sq/compare/v0.26.0...v0.27.0
[v0.28.0]: https://github.com/neilotoole/sq/compare/v0.27.0...v0.28.0
[v0.29.0]: https://github.com/neilotoole/sq/compare/v0.28.0...v0.29.0
[v0.30.0]: https://github.com/neilotoole/sq/compare/v0.29.0...v0.30.0
[v0.31.0]: https://github.com/neilotoole/sq/compare/v0.30.0...v0.31.0
[v0.32.0]: https://github.com/neilotoole/sq/compare/v0.31.0...v0.32.0
[v0.33.0]: https://github.com/neilotoole/sq/compare/v0.32.0...v0.33.0
[v0.34.0]: https://github.com/neilotoole/sq/compare/v0.33.0...v0.34.0
[v0.34.1]: https://github.com/neilotoole/sq/compare/v0.34.0...v0.34.1
[v0.34.2]: https://github.com/neilotoole/sq/compare/v0.34.1...v0.34.2
[v0.35.0]: https://github.com/neilotoole/sq/compare/v0.34.2...v0.35.0
[v0.36.0]: https://github.com/neilotoole/sq/compare/v0.35.0...v0.36.0
[v0.36.1]: https://github.com/neilotoole/sq/compare/v0.36.0...v0.36.1
[v0.36.2]: https://github.com/neilotoole/sq/compare/v0.36.1...v0.36.2
[v0.37.0]: https://github.com/neilotoole/sq/compare/v0.36.2...v0.37.0
[v0.37.1]: https://github.com/neilotoole/sq/compare/v0.37.0...v0.37.1
[v0.38.0]: https://github.com/neilotoole/sq/compare/v0.37.1...v0.38.0
[v0.38.1]: https://github.com/neilotoole/sq/compare/v0.38.0...v0.38.1
[v0.39.0]: https://github.com/neilotoole/sq/compare/v0.38.1...v0.39.0
[v0.39.1]: https://github.com/neilotoole/sq/compare/v0.39.0...v0.39.1
[v0.40.0]: https://github.com/neilotoole/sq/compare/v0.39.1...v0.40.0
[v0.41.0]: https://github.com/neilotoole/sq/compare/v0.40.0...v0.41.0
[v0.41.1]: https://github.com/neilotoole/sq/compare/v0.41.0...v0.41.1
[v0.42.0]: https://github.com/neilotoole/sq/compare/v0.41.1...v0.42.0
[v0.42.1]: https://github.com/neilotoole/sq/compare/v0.42.0...v0.42.1
[v0.43.0]: https://github.com/neilotoole/sq/compare/v0.42.1...v0.43.0
[v0.43.1]: https://github.com/neilotoole/sq/compare/v0.43.0...v0.43.1
[v0.44.0]: https://github.com/neilotoole/sq/compare/v0.43.1...v0.44.0
[v0.45.0]: https://github.com/neilotoole/sq/compare/v0.44.0...v0.45.0
[v0.46.0]: https://github.com/neilotoole/sq/compare/v0.45.0...v0.46.0
[v0.46.1]: https://github.com/neilotoole/sq/compare/v0.46.0...v0.46.1
[v0.47.0]: https://github.com/neilotoole/sq/compare/v0.46.1...v0.47.0
[v0.47.1]: https://github.com/neilotoole/sq/compare/v0.47.0...v0.47.1
Reference in New Issue View Git Blame Copy Permalink