Commit Graph

129 Commits

Author SHA1 Message Date
Neil O'Toole
c3a543aa84
#229: refactor mysql source metadata impl (#236)
* refactor mysql source metadata impl

* Made loop condition stronger
2023-05-23 18:42:36 -06:00
Neil O'Toole
2f2dfd6e47
#229: More diff (#233)
- Implement `sq diff --data`.
2023-05-22 09:08:14 -06:00
Neil O'Toole
1ea24dac4a
#229: sq diff core (#230)
* "sq diff" initial implementation

* Refactor "cli" pkg.
2023-05-19 08:24:18 -06:00
Neil O'Toole
c5cf6c0cc0
#217: Configurable timestamp format (#218)
* Moved time functions from pkg stringz to pkg timez

* Refactor options.Opt

* wip: initial work on configurable time layout

* wip: most printers now respect format.datetime and friends

* Folded pkg timefmt into timez

* Refactor options.Opt; refine options

* Add 'sq config set OPTION --help' mechanism

* Finished completion of OptDateFormatAsNumber and OptTimeFormatAsNumber
2023-05-06 20:36:34 -06:00
Neil O'Toole
3180334c0c
#199: Config overhaul (#214)
* refactor: partially moved over driver.Tuning params to options

* All knobs moved to options

* sq config edit: now has comments for options

* Major work complete on config/options overhaul

* Major work complete on config/options overhaul

* Updated help text for 'sq version'
2023-05-03 06:36:10 -06:00
Neil O'Toole
4521008b72
#199: sq config set (#212)
* Initial impl of 'sq config set'

* Verbose output for 'sq config get'

* config/options: refining 'sq config get'

* Refine 'sq config get': now prints defaults when --verbose

* cli: 'sq config get option.key' implemented

* cli: initial impl of 'sq config get --src'

* cli: refine 'sq config get'

* cli: 'sq config set --src' is implemented
2023-04-30 21:59:34 -06:00
Neil O'Toole
e5db7990bc
#199: Kill RunContext.Log (#211)
* refactor: RunContext.Log is dead

* Fiddling with options

* Changed default max conns to zero (no limit)

* Linting
2023-04-30 08:18:56 -06:00
Neil O'Toole
5e31da4644
Refactor config options (#209)
* Refactor config and options.
2023-04-26 09:16:42 -06:00
Neil O'Toole
57cddd7590
Edit config (#208)
* cobra: Added 'sq config edit' cmd

* Linting

* Implemented 'sq config edit'

* whitespace

* Dialing in config cmd help
2023-04-22 16:31:28 -06:00
Neil O'Toole
81f631e135
Refactoring (#206)
* Renamed source.Type to source.DriverType for clarity

* More renaming wrt source.DriverType

* Renamed output.Formatting to output.Printing
2023-04-21 21:36:32 -06:00
neilotoole
c1b71fa68a Renamed testh.OptLongDB to testh.OptLongOpen; added it to other tests. 2023-04-19 08:08:26 -06:00
Neil O'Toole
98b47a2666
#199 - Config, refactoring (#204)
* refactor: moved cli flags to pkg cli/flag

* testh: add OptLongDB for long-running tests

* implement 'sq config dir'

* legacy dir migration: probably a bad idea

* cleanup

* Refactored SQ_CONFIG and --config

* added yaml writer

* Dialing in tests

* YAML output for 'sq driver ls'

* Significant refactoring of config

* Minor test for ioz

* Rename source.Set to source.Collection

* Cleaning up references to source.Set
2023-04-18 23:28:09 -06:00
Neil O'Toole
958d509088
Source groups (#198)
* Implemented source groups mechanism.
2023-04-15 16:28:51 -06:00
Neil O'Toole
ebb88b3a44
CSV empty string now treated as NULL (#190) 2023-04-09 19:29:13 -06:00
Neil O'Toole
ad23d5c116
Auto-detect CSV headers (#188)
* wip: initial header detection

* wip: refactored csv driver

* wip: Fixes to kind.Detector

* typo

* Refactor pkg csv files

* Update to changelog

* workflow: now running codacy and codeql on cron, not on push/pr
2023-04-09 08:44:27 -06:00
Neil O'Toole
54a0155ed6
Deal with lots of connections, especially for sq inspect. (#186)
* wip: deal with lots of connections

* Clean up TestDatabase_SourceMetadata_concurrent

* Fixed error message
2023-04-08 12:09:27 -06:00
Neil O'Toole
256d9c1ab6
go.mod: upgrade to jackc/pgx v5 (#184)
* jackc/pgx upgrade

* jackc/pgx upgrade
2023-04-08 06:39:06 -06:00
Neil O'Toole
9a1c6a7d09
Feature/173 args (#183)
- Implement --arg feature
- Refactor sqlbuilder package (now called "render").
- Bug fixes, especially around expressions.
2023-04-07 02:00:49 -06:00
Neil O'Toole
791d41ad1d
Removed dependencies on neilotoole/errgroup in favor of sync/errgroup (#176)
* Removed dependencies on neilotoole/errgroup in favor of sync/errgroup

* Removed deadcode
2023-04-02 15:17:15 -06:00
Neil O'Toole
3f6157c4c4
Change logging library to slog (#175)
- Switch to slog logger.
2023-04-02 13:49:45 -06:00
Neil O'Toole
bfd5542f75
Refactor/introduce query context (#174)
* Moved query_X_test.go from /drivers to /libsq

* Refactor: introduced libsq.QueryContext type
2023-04-01 03:48:24 -06:00
Neil O'Toole
8d47ef806d
go1.20 upgrade; lots of linting (#172) 2023-04-01 02:38:32 -06:00
Neil O'Toole
29e33ed2b1
Implement unique / DISTINCT (#165)
- Implemented "unique" function
- Implemented "count_unique" function
2023-03-28 00:48:24 -06:00
Neil O'Toole
fac3a27d7b
group_by() function (#163)
- `group_by()` now accepts function as argument.
- Refactored grammar.
- Broad improvements to function implementation.
2023-03-26 20:03:40 -06:00
Neil O'Toole
d9b56eea8a
Implement groupby() (#161)
* test: slq2sql more test cases; now executes the generated query

* groupby: first test case working against all DBs

* sakila: added sqlserver 2019 source

* groupby: tests for groupby synonyms

* groupby: final touches
2023-03-26 02:01:41 -06:00
Neil O'Toole
9746f4c1a2
orderby() (#159)
* wip: orderby impl

* Tests passing (note: ast.checkASTIntegrity is disabled)

* ExprNode now rendered via renderSelectorNode

* linting

* CHAGELOG for v0.27.0
2023-03-25 19:20:53 -06:00
Neil O'Toole
a1a89ee9dd
Support table and column names with spaces. (#156)
* sakila: initial test data

* sakila: more test data

* sakila: yet more test data setup

* whitespace cols: now working for sqlite

* grammar cleanup

* whitespace cols: now working inside count() func for sqlite

* whitespace cols: tests mostly passing; begining refactoring

* grammar: refactor handle

* grammar: more refactoring

* grammar: rename selElement to selector

* wip

* all tests passing

* all tests passing

* linting

* driver: implement CurrentSchema for all driver.SQLDriver impls

* driver: tests for AlterTableRename and AlterTableRenameColumn

* undo reformat of SQL

* undo reformat of SQL

* undo reformat of SQL

* undo reformat of SQL
2023-03-22 00:17:34 -06:00
Neil O'Toole
d3e6f89829
SLQ support for column aliases (#150)
* alias: more early work

* alias: test cases working for sqlite

* alias: SQL builder tests

* alias: func (col expr) aliases now working for SQLite

* linting

* CHANGELOG update

* Docs update

* Docs update

* Rename buildAst() -> buildAST()

* CHANGELOG typo
2023-03-18 22:58:00 -06:00
Neil O'Toole
161d6f9ee7
Fix to docs for cmd_sql (#148) 2023-03-15 01:43:48 -06:00
Neil O'Toole
7148505a84
CSV: check for mismatch field count vs explicitly specified column names (#147)
* csv: check for mismatch field count vs explicitly specified column names

* CHANGELOG update
2023-03-14 23:04:49 -06:00
Neil O'Toole
f71b124a1f
Testing for postgres alternate schema 2023-03-13 19:25:12 -06:00
Neil O'Toole
5d799a8820
Bug/windows file slash munge location (#140)
* Munge windows file location

* Windows filepath fix

* More Windows filepath issues

* Main workflow runs on matrix each time
2022-12-31 22:35:33 -07:00
Neil O'Toole
ed9aa38a67
Improvements to source commands (#139)
* Expose source.Set.Data() method

* jsonw.writeJSON cleaned up

* sq add now respects --json

* Location strings are subject to more scrutiny

* Ignore .db files in project dir

* sq add is more restrictive about location string

* source.RedactedLocation now uses 'xxxxx' per stdlib url.URL.Redacted()

* Update changelog for v0.23.0

* typos
2022-12-31 20:17:44 -07:00
neilotoole
48842fd0d3 linting 2022-12-24 22:00:28 -07:00
neilotoole
70e650ef87 linting 2022-12-24 21:09:20 -07:00
Neil O'Toole
a1ba6578da
"add" command supports hiding password input (#132)
* renamed cmdFlagChanged to flagChanged

* initial stdin stuff working

* wip: mostly working as expected

* Docs and lots of cleanup

* Mostly docs

* fixed behavior of source.LocationWithPassword, and tests
2022-12-24 21:04:18 -07:00
Neil O'Toole
e2a9605ec6
Chore/sqlite3 upgrade (#129)
* wip

* made progress on new SQLite type stuff

* we're getting places

* largely working now

* tidying up

* almost there

* time to deal with failing tests

* seemed to have fixed sqlserver issue

* seems to all be fixed

* minor tidy up
2022-12-23 21:05:26 -07:00
Neil O'Toole
fff01a32db
Chore/yet more mod upgrades (#128)
* gopkg.in/yaml.v3 upgrade

* emirpasic/gods@v1.18.1

* yet more module stuff
2022-12-23 14:05:24 -07:00
Neil O'Toole
8cbcaf6606
Chore/update go modules (#125)
* switched to microsoft/go-mssqldb

* upgraded go.uber.org packages

* updated ryboe/q

* more package updates
2022-12-23 11:09:06 -07:00
Neil O'Toole
fab365f43c
gofumpt on files (#122)
* gofumpt on files

* more gofumpt
2022-12-18 01:35:59 -07:00
Neil O'Toole
c778c8ced5
Linting of _test.go files (#121)
* test linting

* test linting

* test linting

* test linting
2022-12-18 00:18:35 -07:00
Neil O'Toole
f9c19785e3
Almost at the end of the linting road (#120)
* yet more linting

* yet more linting

* yet more linting

* yet more linting

* yet more linting

* yet more linting

* yet more linting

* yet more linting
2022-12-17 23:42:11 -07:00
Neil O'Toole
425702e7ba
The linting goes on forever (#119)
* linting

* yet more linting

* yet more linting

* yet more linting

* yet more linting

* yet more linting
2022-12-17 23:07:38 -07:00
Neil O'Toole
ac7535609d
Yet more linting (#118)
* lint fixes

* lint gosec
2022-12-17 22:16:10 -07:00
Neil O'Toole
08dfa10325
Errors linting (#117)
* errors linting

* errors linting

* formatting
2022-12-17 21:31:06 -07:00
Neil O'Toole
540adfac58
Lint long lines (#115)
* lint config now sorts results

* linted long lines

* linted long lines
2022-12-17 19:43:53 -07:00
Neil O'Toole
2831211ae9
Yet more linting (#114)
* wip: bunch o' linting

* bunch more linting
2022-12-17 17:51:33 -07:00
Neil O'Toole
5187e8000a
bunch of linting issues fixed (#113) 2022-12-17 16:11:33 -07:00
Neil O'Toole
e425da528f
pg constraints now scan into sql.NullString for some fields (#110) 2022-12-16 19:24:52 -07:00
Neil O'Toole
6a0878bc6b
fixed broken mysql tests (parseTime param); moved some test funcs to pkg tutil (#109) 2022-12-16 19:09:49 -07:00
Neil O'Toole
e674cdc724
change 'interface{}' to 'any' (#108) 2022-12-16 16:34:33 -07:00
Neil O'Toole
a419e0b693
Misc cleanup, goimports, updates (#107)
* debugging

* generic retry functionality; some additions to stringz.go

* fiddling with magefile_sakila

* added test lock function testh.Lock

* bunch of linting, goimports, what not
2022-12-16 15:54:09 -07:00
Neil O'Toole
d8ecff4234
drivers/mysql linting (#92) 2021-03-13 10:41:15 -07:00
Neil O'Toole
2d843c9550
MySQL DSN no longer strips driver options (#91)
* MySQL DSN no longer strips driver options
Fixes #90

* CHANGELOG update
2021-03-13 10:08:53 -07:00
Neil O'Toole
6870327508
Cobra upgrade: includes shell completion work (#81)
Addressed #80
2021-02-22 00:37:00 -07:00
Neil O'Toole
a1cfd5d525
Xlsx import handle empty sheets (#79)
* can now import XLSX with empty sheets

* renamed XLSX test article
2021-02-19 23:22:35 -07:00
Neil O'Toole
71a4f78e52
--insert now also creates the table (#78) 2021-01-03 19:56:22 -07:00
unknown
8d2398b309 close db resources 2021-01-01 18:57:07 -07:00
Neil O'Toole
5aebc04356
Fixed issues with files and databases not being closed correctly (#73)
* fiddling with scratch database close order

* files debugging

* files debugging2

* files debugging3

* files debugging 4

* files debugging 5

* didn't close the ReadCloser in csv import

* more closing cleanup
2020-12-30 11:57:58 -07:00
Neil O'Toole
8c883d276e
sq tbl truncate for postgres now outputs the number of rows truncated (#72) 2020-11-24 22:12:15 -07:00
Neil O'Toole
06900f8c84
fixed some panics with oddly-shaped xlsx imports (#71) 2020-11-02 10:40:29 -07:00
Neil O'Toole
a92b9abf34
Initial work on a JSON driver (#70)
* implementation work for json importers

* json driver checkpoint

* working on json.ParseObjectsInArray

* json.ParseObjectsInArray seems to be working

* checkpoint while tidying up ParseObjectsInArray

* more tidy checkpoint

* more tidy checkpoint 2

* tidying up ParseObjectsInArray

* tidy up

* code/docs cleanup

* more cleanup of json driver

* more cleanup of json driver

* flat json import seemingly working

* improvements to json driver

* json writer now prints empty [] for postgres empty tables
2020-10-20 09:05:43 -06:00
Neil O'Toole
0f3bc26f52
JSON driver: initial jsona driver (#69)
* jsona driver appears to be mostly working

* jsona verified as working

* minor tidy
2020-08-23 20:32:59 -06:00
Neil O'Toole
1818d50546
moved pkg libsq/sqlmodel to libsq/core/sqlmodel (#68) 2020-08-23 13:00:13 -06:00
Neil O'Toole
38be39a12f yet more pkg moving 2020-08-23 05:52:09 -06:00
Neil O'Toole
58ccfc9ded
Json driver; refactoring of core packages (#66)
* Type Detector refactor

* json driver impl; refactoring of source.Files reader et al

* working on kind detector

* significant switcheroo of packages

* partway throught refactoring Kind

* major package switcheroo for Kind
2020-08-23 04:42:15 -06:00
Neil O'Toole
b3667d60cd
implementing SQLDriver.AlterTableAddColumn (#65) 2020-08-20 21:08:59 -06:00
Neil O'Toole
9746f726f6
fixing issues with inspect/metadata (#64) 2020-08-20 14:19:16 -06:00
Neil O'Toole
929b81c6d3
MySQL "inspect" performance (#62)
* mysql inspect performance

* tidying up mysql metadata
2020-08-18 13:13:25 -06:00
Neil O'Toole
65259754f5
Improvements to metadata (sq inspect) (#61)
* improvements to sqlite source metadata query

* table metadatawriter colorization tuning; minor testh.Helper refactoring

* cleanup of metadata for multiple drivers

* more cleanup of source/table metadata

* yet more cleanup of source/table metadata

* improvements to mysql SourceMetadata

* improvements to mysql TableMetadata

* yet more fiddling with mysql metadata
2020-08-15 15:06:40 -06:00
Neil O'Toole
f00c1f2307
renamed driver.Driver impls from mysql.Driver to mysql.driveri etc (#60) 2020-08-12 21:40:03 -06:00
Neil O'Toole
48630a621f
Removed sqlx dependencies (#59) 2020-08-12 21:22:53 -06:00
Neil O'Toole
1ceb50e795
SQL batch insert (#58)
* initial refactoring for the numRows param

* work on driver.NewBatchInsert

* work on NewBatchInsert

* batch insert seems to work

* switched testh.Insert to use BatchInsert

* doc cleanup

* batch insert for dbwriter and csv

* removed unneeded NumRows from driver.StmtExecer

* minor tidyup
2020-08-12 12:24:01 -06:00
Neil O'Toole
3e10868a29
Switched pkg sakila "const" slices to functions (#57)
* replaced sakila source slices with funcs

* missed some pkg sakila slices to refactor
2020-08-10 09:16:44 -06:00
Neil O'Toole
8b1bd4c03a
replaced sakila source slices with funcs (#56) 2020-08-09 08:40:46 -06:00
Neil O'Toole
67cfb142e1
config cleanup; misc doc fixes; minor refactoring; fiddling with output writer presentation (#55) 2020-08-09 07:46:46 -06:00
Neil O'Toole
915e01fdda
added Go dependency cache to go.yml workflow, and go test now covers the full codebase (#53)
* added Go dependency cache to go.yml workflow

* more windows path issues

* docs

* restricting workflow triggers to master

* wrong cleanup order for downloaded files

* de-restricting workflow triggers, for now

* more out-of-order cleanup issues that only manifested on windows

* more out-of-order cleanup issues

* removed mediatype check from TestFetchFile; FetchFile is deprecated and not worth fixing

* tracked down out-of-sequence close/remove issue with sqlite

* probably long-running test failure

* disabling long-running tests that are breaking CI

* switching to go test -short because tests seem to be taking too long in CI
2020-08-08 15:23:30 -06:00
Neil O'Toole
062e2dea88
Filepath and colorization issues with the v0.12.1 release on Windows (#50)
- Windows filepath handling
- Windows colorization handling
- Minor code tidy up and refactor
2020-08-07 13:51:30 -06:00
Neil O'Toole
fd4ae53f31 codebase refactor 2020-08-06 11:58:47 -06:00