* Switched from xlsxw to excelw for Excel outpt
27 KiB
CHANGELOG
All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
Breaking changes are annotated with ☢️.
Upcoming
Added
-
#99: The CSV and XLSX drivers can now handle duplicate header column names in the ingest data. For example, given a CSV file:
actor_id,first_name,actor_id 1,PENELOPE,1 2,NICK,2
The columns will be renamed to:
actor_id,first_name,actor_id_1
The renaming behavior is controlled by a new option
ingest.column.rename
(docs). This new option is effectively the ingest counterpart of the existing output optionresult.column.rename
. -
#191: The XLSX driver now detects header rows, like the CSV driver already does. Thus, you now typically don't need to specify the
--ingest.header
flag for Excel files. However, the option remains available in casesq
can't figure it out for a particular file. -
If an error occurs when the output format is
text
, a stack trace is printed tostderr
when the command is executed with--verbose
(-v
). -
There's a new option
error.format
that controls error output format independent of the mainformat
option (docs). Theerror.format
value must be one oftext
orjson
.
Changed
-
☢️ The XLSX writer now formats dates differently. Previously the format was
11/9/89
, and now it is1989-11-09
. The same applies to datetimes, e.g.11/9/1989 00:00:00
becomes1989-11-09 00:00
.This change is made to reduce ambiguity and confusion. Apparently Microsoft Excel itself will pick up the date format from OS system settings. However,
sq
uses a library to interact with Excel files, and that library chooses a particular format by default (11/9/89
). There are several paths we could take here:- Interrogate the OS, and use the OS locale date format.
- Stick with the library default
11/9/89
. - Pick a default other than
11/9/89
.
We have chosen the third option. The first option (locale-dependent) is excluded because, as a general rule, we want
sq
to produce the same output regardless of locale/system settings. We exclude the second option because month/day confuses most of the world. Thus, we're left with picking a default, and1989-11-09
is the format used in RFC3339 and friends.Whether this is the correct (standard?) approach is still unclear, and feedback is welcome. Ultimately this may become a config option.
-
The XLSX writer now outputs header rows in bold text.
-
☢️ The XLSX writer now outputs blob (
bytes
) cell data as a base64-encoded string, instead of raw bytes.
Fixed
- Fixed bug where source-specific config wasn't being propagated.
v0.40.0 - 2023-07-03
This release features a complete overhaul of the join
mechanism.
Added
-
#277: A table selector can now have an alias. This in and of itself is not particularly useful, but it's a building block for multiple joins.
$ sq '@sakila | .actor:a | .a.first_name'
-
New option
result.column.rename
that exposes a template used to rename result set column names before display. The primary use case is to de-duplicate columns names on aSELECT * FROM tbl1 JOIN tbl2
, wheretbl1
andtbl2
have clashing column names (docs). -
#157: Previously only
join
(INNER JOIN
) was available: now the rest of the join types such asleft_outer_join
,cross_join
, etc. are implemented (docs).
Changed
-
☢️ #12: The table join mechanism has been completely overhauled. Now there's support for multiple joins. See docs.
# Previously, only a single join was possible $ sq '.actor, .film_actor | join(.actor_id)' # Now, an arbitrary number of joins $ sq '.actor | join(.film_actor, .actor_id) | join(.film, .film_id)'
-
☢️ The alias for
--jsonl
(JSON Lines) has been changed to-J
.
Fixed
- Fixed bug where config options weren't being propagated correctly.
v0.39.1 - 2023-06-22
Fixed
- Bug with
sq version
output on Windows.
v0.39.0 - 2023-06-22
Added
- #263:
sq version
now supports--yaml
output. - #263:
sq version
now outputs host OS details with--verbose
,--json
and--yaml
flags. The motivation behind this is bug submission: we want to know which OS/arch the user is on. E.g. forsq version -j
:
{
"version": "v0.38.1",
"commit": "eedc11ec46d1f0e78628158cc6fd58850601d701",
"timestamp": "2023-06-21T11:41:34Z",
"latest_version": "v0.39.0",
"host": {
"platform": "darwin",
"arch": "arm64",
"kernel": "Darwin",
"kernel_version": "22.5.0",
"variant": "macOS",
"variant_version": "13.4"
}
}
- #263: The output of
sq inspect
andsq inspect -v
has been refactored significantly, and should now be easier to work with (docs).
v0.38.1 - 2023-06-19
Fixed
- #261: The JSON writer (
--json
) could get deadlocked when a record contained a large amount of data, triggering an internalFlush()
(which is mutex-guarded) from within the mutex-guardedWriteRecords()
method.
v0.38.0 - 2023-06-18
This release has significant improvements (and breaking changes)
to SLQ (sq
's query language).
Changed
-
☢️ #254: The formerly-implicit "WHERE" mechanism now requires an explicit
where()
function. This, alas, is a fairly big breaking change. But it's necessary to remove an ambiguity roadblock. See discussion in the issue.# Previously $ sq '.actor | .actor_id <= 2' # Now $ sq '.actor | where(.actor_id <= 2)'
-
#256: Column-only queries are now possible. This has the neat side effect that
sq
can now be used as a calculator.$ sq 1+2 1+2 3
You may want to use
--no-header
(-H
) when usingsq
as a calculator.$ sq -H 1+2 3 $ sq -H '(1+2)*3' 9
Fixed
-
Literals can now be selected (docs).
$ sq '.actor | .first_name, "X":middle_name, .last_name | .[0:2]' first_name middle_name last_name PENELOPE X GUINESS NICK X WAHLBERG
-
Lots of expressions that previously failed badly, now work.
$ sq '.actor | .first_name, (1+2):addition | .[0:2]' first_name addition PENELOPE 3 NICK 3
-
#258: Column aliases can now be arbitrary strings, instead of only a valid identifier.
# Previously only valid identifier allowed $ sq '.actor | .first_name:given_name | .[0:2]' given_name PENELOPE NICK # Now, any arbitrary string can be used $ sq '.actor | .first_name:"Given Name" | .[0:2]' Given Name PENELOPE NICK
v0.37.1 - 2023-06-15
Fixed
- #252: Handle
*uint64
returned from DB.
v0.37.0 - 2023-06-13
Added
v0.36.2 - 2023-05-27
Changed
- ☢️ Proprietary database functions are now
invoked by prefixing the function name with an underscore. For example:
# mysql "date_format" func $ sq '@sakila/mysql | .payment | _date_format(.payment_date, "%m")' # Postgres "date_trunc" func $ sq '@sakila/postgres | .payment | _date_trunc("month", .payment_date)'
v0.36.1 - 2023-05-26
Fixed
sq diff
: Renamed--count
flag to--counts
as intended.
v0.36.0 - 2023-05-25
The major feature is the long-gestating sq diff
.
Added
- #229:
sq diff
compares two sources, or tables. sq inspect --dbprops
is a new mode that returns only the DB properties. Relatedly, the properties mechanism is now implemented for all four supported DB types (previously, it was only implemented for Postgres and MySQL).- CSV format now colorizes output.
Changed
sq inspect -v
previously returned DB properties in a field nameddb_variables
. This field has been renamed todb_properties
. The renaming reflects the fact that some of those properties aren't really variables in the sense that they can be modified (e.g. DB server version or such).- The structure of the former
db_variables
(nowdb_properties
) field has changed. Previously it was an array of{"name": "XX", "value": "YY"}
values, but now is a map, where the keys are strings, and the values can be either a scalar (bool
,int
,string
, etc.), or a nested value such as an array or map. This change is made because some databases (e.g. SQLite) feature complex data in some property values. - CSV format now renders byte sequences as
[777 bytes]
instead of dumping the raw bytes. - ☢️ TSV format (
--tsv
) no longer has a shorthand form-T
. Apparently that shorthand wasn't used much, and-T
is needed elsewhere. - ☢️ Likewise,
--xml
no longer has shorthand-X
. And--markdown
has lost alias--md
. - In addition to the format flags
--text
,--json
, etc., there is now a--format=FORMAT
flag, e.g.--format=json
. This will allowsq
to continue to expand the number of output formats, without needing to have a dedicated flag for each format.
Fixed
sq config edit @source
was failing to save any edits.
v0.35.0 - 2023-05-10
Added
Fixed
sq config get OPT --text
now prints only the value, notKEY VALUE
. If you want to see key and value, consider using--yaml
, or--text --verbose
.
v0.34.2 - 2023-05-08
Fixed
- Both
--markdown
and the alias--md
are now supported.
v0.34.1 - 2023-05-07
Fixed
- Fixed a minor issue where
sq ls -jv
andsq ls -yv
produced no output if config contained no explicitly set options.
v0.34.0 - 2023-05-07
This release significantly overhauls sq
's config mechanism (#199).
For an overview, see the new config docs.
Alas, this release has several minor breaking changes ☢️.
Added
sq config ls
shows config.sq config get
gets individual config option.sq config set
sets config values.sq config edit
edits config.- Editor can be specified via
$EDITOR
or$SQ_EDITOR
.
- Editor can be specified via
sq config location
prints the location of the config dir.--config
flag is now honored globally.- Many more knobs are exposed in config.
- Logging is much more configurable. There are new knobs:
There are also equivalent flags ($ sq config set log true $ sq config set log.level INFO $ sq config set log.file /var/log/sq.log
--log
,--log.file
and--log.level
) and envars (SQ_LOG
,SQ_LOG_FILE
andSQ_LOG_LEVEL
). - Several more commands support YAML output:
Changed
- The structure of
sq
's config file (sq.yml
) has changed. The config file is automatically upgraded when using the new version. - The default location of the
sq
log file has changed. The new location is platform-dependent. Usesq config get log.file -v
to view the location, orsq config set log.file /path/to/sq.log
to set it. - ☢️ Envar
SQ_CONFIG
replacesSQ_CONFIGDIR
. - ☢️ Envar
SQ_LOG_FILE
replacesSQ_LOGFILE
. - ☢️ Format flag
--table
is renamed to--text
. This is changed because while the output is mostly in table format, sometimes it's just plain text. Thustable
was not quite accurate. - ☢️ The flag to explicitly specify a driver when piping input to
sq
has been renamed from--driver
to--ingest.driver
. This change aligns the naming of the ingest options and reduces ambiguity.# previously $ cat mystery.data | sq --driver=csv '.data' # now $ cat mystery.data | sq --ingest.driver=csv '.data'
- ☢️
sq add
no longer has the generic--opts x=y
mechanism. This flag was ambiguous and confusing. Instead, use explicit option flags.# previously $ sq add ./actor.csv --opts=header=false # now $ sq add ./actor.csv --ingest.header=false
- ☢️ The short form of the
sq add --handle
flag has been changed from-h
to-n
. While this is not ideal, the-h
shorthand is already in use everywhere else as the short form of--header
.# previously $ sq add ./actor.csv -h @actor # now $ sq add ./actor.csv -n @actor
- ☢️ The
--pretty
flag has been removed. Its only previous use was with thejson
format, where if--pretty=false
would output the JSON in compact form. To better align with jq, there is now a--compact
/-c
flag that behaves identically to jq. - ☢️ Because of the above
--compact
/-c
flag, the short form of the--csv
flag is changing from-c
to-C
. It's an unfortunate situation, but alignment with jq's behavior is an overarching principle that justifies the change.
v0.33.0 - 2023-04-15
The headline feature is source groups.
This is the biggest change to the sq
CLI in some time, and should
make working with lots of sources much easier.
Added
- #192:
sq
now has a mechanism to group sources. A source handle can now be scoped. For example, instead of@sakila_prod
,@sakila_staging
, etc, you can use@prod/sakila
,@staging/sakila
. Usesq group prod
to set the active group (whichsq ls
respects). See docs. sq group GROUP
sets the active group toGROUP
.sq group
returns the active group (default is/
, the root group).sq ls GROUP
lists the sources inGROUP
.sq ls --group
(orsq ls -g
) lists all groups.sq mv
moves/renames sources and groups.
Changed
sq ls
now shows the active item in a distinct color. It no longer adds an asterisk to the active item.sq ls
now sorts alphabetically when using--table
format.sq ls
now shows the sources in the active group only. But note that the default active group is/
(the root group), so the default behavior ofsq ls
is the same as before.sq add hello.csv
will now generate the handle@hello
instead of@hello_csv
. On a second invocation, it will return@hello1
instead of@hello_csv_1
. Why this change? Well, with the availability of the source group mechanism, the_
character in the handle somehow looked ugly. And more importantly,_
is a relative pain to type.sq ping
has changed to support groups. Instead ofsq ping --all
, you can dosq ping GROUP
, e.g.sq ping /
.
v0.32.0 - 2023-04-09
Added
-
#187: For
csv
sources,sq
will now try to auto-detect if the CSV file has a header row or not. Previously, this needed to be explicitly specified via an awkward syntax:$ sq add ./actor.csv --opts=header=true
This change makes working with CSV files significantly lower friction. A command like the below now almost always works as expected:
$ cat ./actor.csv | sq .data
Support for Excel/XLSX header detection is in #191.
Fixed
-
sq
is now better at detecting the (data) kind of CSV fields. It now more accurately distinguishes betweenDecimal
andInt
, and knows how to handleDatetime
. -
#189:
sq
now treats CSV empty fields asNULL
.
v0.31.0 - 2023-03-08
Added
- #173: Predefined variables via
--arg
flag (docs):$ sq --arg first TOM '.actor | .first_name == $first'
Changes
- Use
--md
instead of--markdown
for outputting Markdown.
Fixed
- #185:
sq inspect
now better handles "too many connections" situations. go.mod
: Moved tojackc/pgx
v5
.- Refactor: switched to
slog
logging library.
v0.30.0 - 2023-03-27
Added
- #164: Implemented
unique
function (docs):
This is equivalent to:$ sq '.actor | .first_name | unique'
SELECT DISTINCT first_name FROM actor
- Implemented
count_unique
function (docs).$ sq '.actor | count_unique(.first_name)'
Changed
- The
count
function has been changed (docs)- Added no-args version:
.actor | count
equivalent toSELECT COUNT(*) AS "count" FROM "actor"
. - ☢️ The "star" version (
.actor | count(*)
) is no longer supported; use the naked version instead.
- Added no-args version:
- Function columns are now named according to the
sq
token, not the SQL token.# previous behavior $ sq '.actor | max(.actor_id)' max("actor_id") 200 # now $ sq '.actor | max(.actor_id)' max(.actor_id) 200
v0.29.0 - 2023-03-26
Added
- #162:
group_by
now accepts function arguments.
Changed
- Renamed
groupby
togroup_by
to match jq. - Renamed
orderby
toorder_by
to match jq.
v0.28.0 - 2023-03-26
Added
- #160: Use
groupby()
to group results. See query guide.
v0.27.0 - 2023-03-25
Added
- #158: Use
orderby()
to order results. See query guide.
v0.26.0 - 2023-03-22
Added
- #98: Whitespace is now allowed in SLQ selector names. You can
do
@sakila | ."film actor" | ."actor id"
.
Fixed
- #155:
sq inspect
now populatesschema
field in JSON for MySQL, SQLite, and SQL Server (Postgres already worked).
v0.25.1 - 2023-03-19
Fixed
- #153: Improved formatting of text table with long lines.
v0.25.0 - 2023-03-19
Added
- #15: Column Aliases. You can now change specify an alias for a column (or column expression
such as a function). For example:
sq '.actor | .first_name:given_name
, orsq .actor | count(*):quantity
. - #151:
sq add
now has a--active
flag, which immediately sets the new source as the active source.
v0.24.4 - 2023-03-15
Fixed
- Fixed typos in
sq sql
command help.
v0.24.3 - 2023-03-14
Added
- When a CSV source has explicit column names (via
--opts cols=A,B,C
),sq
now verifies that the CSV data record field count matches the number of explicit columns.
v0.24.2 - 2023-03-13
Fixed
- #142: Improved error handling when Postgres
current_schema()
is unavailable.
v0.24.1 - 2023-03-11
Fixed
- #144: Handle corrupted config active source.
v0.24.0 - 2022-12-31
Added
sq ping
now respects--json
flag.
Fixed
- Improved handling of file paths on Windows.
v0.23.0 - 2022-12-31
Added
sq ls
now respects--json
flag.sq rm
now respects--json
flag.sq add
now respects--json
flag.`- CI pipeline now verifies install packages after publish.
Changed
sq rm
can delete multiple sources.sq rm
doesn't print output unless--verbose
.- Redacted snipped is now
xxxxx
instead of****
, to match stdliburl.URL.Redacted()
.
Fixed
- Fixed crash on Fedora systems (needed
--tags=netgo
).
v0.21.3 - 2022-12-30
Added
sq version
respects--json
flag.sq version
respects--verbose
flag.sq version
showslatest_version
info when--verbose
and there's a newer version available.
Changed
sq version
shows less info when--verbose
is not set.
v0.20.0 - 2022-12-29
Added
sq
now generates manpages (and installs them).
v0.19.0 - 2022-12-29
Added
- Installer for Arch Linux, via Arch User Repository.
v0.18.2 - 2022-12-25
Added
- The build pipeline now produces
.apk
packages for Alpine Linux, andinstall.sh
has been updated accordingly. However, the.apk
files are not yet published to a repository, so it's necessary to runapk
against the downloaded.apk
file (install.sh
does this for you).
v0.18.0 - 2022-12-24
Added
- #95:
sq add
now has a--password
(-p
) flag that prompts the user for the data source password, instead of putting it in the location string. It will also read from stdin if there's input there.
v0.17.0 - 2022-12-23
Changed
- More or less every
go.mod
dependency has been updated to latest. This includes drivers forsqlite
andsqlserver
. The driver updates led to some broken things, which have been fixed.
v0.16.1 - 2022-12-23
Fixed
- #123: Shell completion is better behaved when a source is offline.
v0.16.0 - 2022-12-16
Added
--verbose
flag is now globalinstall.sh
install script.
Changed
- Improved GH workflow
sq inspect
shows less output by default (use-v
to restore previous behavior)
Fixed
sq inspect
can now deal with Postgres sources that have null values for constraint fields
v0.15.11 - 2022-11-06
Changed
- Yet more changes to GitHub workflow.
v0.15.4 - 2021-09-18
Changed
- Bug fixes
v0.15.3 - 2021-03-13
Changed
- #91: MySQL driver options no longer stripped
v0.15.2 - 2021-03-08
Changed
- #89: Bug with SQL generated for joins.