Commit Graph

50 Commits

Author SHA1 Message Date
Neil O'Toole
2898a92983
Refactor sqlmodel pkg (#364)
* Refactor sqlmodel pkg
2024-01-25 00:42:51 -07:00
Neil O'Toole
26f0c9a381
Refactor source.Files (#363)
* Moved `source.Files` to its own package, thus the type is now `files.Files`.
* Moved much of the location functionality from pkg `source` to its own package `location`.
2024-01-24 23:29:55 -07:00
Neil O'Toole
bb4a4edaab
Switch downloads to use streamcache. (#361)
* Switch downloads to use neilotoole/streamcache.
2024-01-24 21:01:24 -07:00
Neil O'Toole
9aa45b1db3
More progress bars (#357)
* More progress bars
2024-01-14 20:56:54 -07:00
Neil O'Toole
db55986980
#307: Ingest cache (#354)
- Support for ingest cache, download cache, and progress bars.
2024-01-14 18:45:34 -07:00
Neil O'Toole
f85da722ad
#335 decimal type support (#336)
* Now using a dedicated `decimal.Decimal` type instead of float/string.
2023-11-21 14:49:52 -07:00
Neil O'Toole
096e209a01
sq inspect now has --schemata and --catalogs modes (#334)
* Add --schemata and --catalogs flags to "sq inspect"
2023-11-20 14:42:38 -07:00
Neil O'Toole
2de993acaa
Sort imports using goimports-reviser (#331) 2023-11-19 18:06:36 -07:00
Neil O'Toole
82727b3890
Refactor/rename database to pool (#328)
* Renamed `driver.Database` to `driver.Pool` (and related things)

* workflow: Update tparse version

* workflow: Update golangci-lint version
2023-11-18 19:21:14 -07:00
Neil O'Toole
511e69f59b
Bunch o' dependency updates (#327)
* Bunch o' dependency updates

* All deps updated

* Linting
2023-11-18 18:02:50 -07:00
Neil O'Toole
f07edef14d
Add flag --src.schema (#326)
* Support for --src.schema in commands "slq", "sql", and "inspect"
2023-11-18 17:05:48 -07:00
Neil O'Toole
611ad30f60
Test ingest Excel file format variants (#303)
* Tests for additional formats

* Tests for ingest Excel file format variants
2023-08-20 07:22:24 -06:00
Neil O'Toole
db092d5453
CSV ingest improvements (#302)
* csv: additional datetime detect formats
2023-08-18 09:21:11 -06:00
Neil O'Toole
edc933d897
#200: Feature/200 excel driver (#301)
* Switch to excelize driver
2023-08-16 09:09:50 -06:00
Neil O'Toole
c7bba4dfe4
go1.21: changes to support slog as part of stdlib (#299)
* go1.21: changes to support slog as part of stdlib

* Removed accidentially checked-in line of code

* Fixed minor linting issues; reenable typecheck

* go1.21: switched to stdlib slices pkg
2023-08-12 12:54:14 -06:00
Neil O'Toole
6ca26f4e4f
Column rename: template now has Alpha field. (#285)
* wip: refactor col name mungeing

* Finished refactoring FieldMeta

* Renamed tpl .AlphaIndex to .Alpha

* wip: debugging source config override

* Source config override passing tests

* CHANGELOG update
2023-07-08 19:34:53 -06:00
Neil O'Toole
9c5836ef1c
#191: XLSX driver auto-detects header row (#284)
* xlsx driver now detects header row.
2023-07-08 09:21:27 -06:00
Neil O'Toole
4ffaae925f
#99: Rename duplicate ingest headers (#283)
* CSV now renames duplicate ingest headers

* Fix broken test

* xlsx ingester now handles duplicate col names

* Update CHANGELOG

* Additional tests for ingest.column.rename

* Removed dead comment in grammar
2023-07-04 11:31:47 -06:00
Neil O'Toole
a5c4f9d6bd
#263: Inspect --overview; improved output (#273)
* Refactor tablew md writer

* wip: cmd inspect -v

* cmd inspect output improved

* cmd inspect tests

* cmd inspect: help text
2023-06-21 23:48:58 -06:00
Neil O'Toole
44d27207f8
#256: column-only queries (#257)
* Column-only queries
2023-06-17 19:28:11 -06:00
neilotoole
4eb917f276 Make testh.OptLongOpen even longer 2023-05-25 23:02:11 -06:00
Neil O'Toole
1ea24dac4a
#229: sq diff core (#230)
* "sq diff" initial implementation

* Refactor "cli" pkg.
2023-05-19 08:24:18 -06:00
Neil O'Toole
3180334c0c
#199: Config overhaul (#214)
* refactor: partially moved over driver.Tuning params to options

* All knobs moved to options

* sq config edit: now has comments for options

* Major work complete on config/options overhaul

* Major work complete on config/options overhaul

* Updated help text for 'sq version'
2023-05-03 06:36:10 -06:00
Neil O'Toole
e5db7990bc
#199: Kill RunContext.Log (#211)
* refactor: RunContext.Log is dead

* Fiddling with options

* Changed default max conns to zero (no limit)

* Linting
2023-04-30 08:18:56 -06:00
Neil O'Toole
5e31da4644
Refactor config options (#209)
* Refactor config and options.
2023-04-26 09:16:42 -06:00
Neil O'Toole
81f631e135
Refactoring (#206)
* Renamed source.Type to source.DriverType for clarity

* More renaming wrt source.DriverType

* Renamed output.Formatting to output.Printing
2023-04-21 21:36:32 -06:00
neilotoole
c1b71fa68a Renamed testh.OptLongDB to testh.OptLongOpen; added it to other tests. 2023-04-19 08:08:26 -06:00
Neil O'Toole
98b47a2666
#199 - Config, refactoring (#204)
* refactor: moved cli flags to pkg cli/flag

* testh: add OptLongDB for long-running tests

* implement 'sq config dir'

* legacy dir migration: probably a bad idea

* cleanup

* Refactored SQ_CONFIG and --config

* added yaml writer

* Dialing in tests

* YAML output for 'sq driver ls'

* Significant refactoring of config

* Minor test for ioz

* Rename source.Set to source.Collection

* Cleaning up references to source.Set
2023-04-18 23:28:09 -06:00
Neil O'Toole
958d509088
Source groups (#198)
* Implemented source groups mechanism.
2023-04-15 16:28:51 -06:00
Neil O'Toole
54a0155ed6
Deal with lots of connections, especially for sq inspect. (#186)
* wip: deal with lots of connections

* Clean up TestDatabase_SourceMetadata_concurrent

* Fixed error message
2023-04-08 12:09:27 -06:00
Neil O'Toole
3f6157c4c4
Change logging library to slog (#175)
- Switch to slog logger.
2023-04-02 13:49:45 -06:00
Neil O'Toole
8d47ef806d
go1.20 upgrade; lots of linting (#172) 2023-04-01 02:38:32 -06:00
Neil O'Toole
fab365f43c
gofumpt on files (#122)
* gofumpt on files

* more gofumpt
2022-12-18 01:35:59 -07:00
Neil O'Toole
c778c8ced5
Linting of _test.go files (#121)
* test linting

* test linting

* test linting

* test linting
2022-12-18 00:18:35 -07:00
Neil O'Toole
425702e7ba
The linting goes on forever (#119)
* linting

* yet more linting

* yet more linting

* yet more linting

* yet more linting

* yet more linting
2022-12-17 23:07:38 -07:00
Neil O'Toole
540adfac58
Lint long lines (#115)
* lint config now sorts results

* linted long lines

* linted long lines
2022-12-17 19:43:53 -07:00
Neil O'Toole
2831211ae9
Yet more linting (#114)
* wip: bunch o' linting

* bunch more linting
2022-12-17 17:51:33 -07:00
Neil O'Toole
5187e8000a
bunch of linting issues fixed (#113) 2022-12-17 16:11:33 -07:00
Neil O'Toole
6a0878bc6b
fixed broken mysql tests (parseTime param); moved some test funcs to pkg tutil (#109) 2022-12-16 19:09:49 -07:00
Neil O'Toole
e674cdc724
change 'interface{}' to 'any' (#108) 2022-12-16 16:34:33 -07:00
Neil O'Toole
a1cfd5d525
Xlsx import handle empty sheets (#79)
* can now import XLSX with empty sheets

* renamed XLSX test article
2021-02-19 23:22:35 -07:00
Neil O'Toole
06900f8c84
fixed some panics with oddly-shaped xlsx imports (#71) 2020-11-02 10:40:29 -07:00
Neil O'Toole
a92b9abf34
Initial work on a JSON driver (#70)
* implementation work for json importers

* json driver checkpoint

* working on json.ParseObjectsInArray

* json.ParseObjectsInArray seems to be working

* checkpoint while tidying up ParseObjectsInArray

* more tidy checkpoint

* more tidy checkpoint 2

* tidying up ParseObjectsInArray

* tidy up

* code/docs cleanup

* more cleanup of json driver

* more cleanup of json driver

* flat json import seemingly working

* improvements to json driver

* json writer now prints empty [] for postgres empty tables
2020-10-20 09:05:43 -06:00
Neil O'Toole
0f3bc26f52
JSON driver: initial jsona driver (#69)
* jsona driver appears to be mostly working

* jsona verified as working

* minor tidy
2020-08-23 20:32:59 -06:00
Neil O'Toole
1818d50546
moved pkg libsq/sqlmodel to libsq/core/sqlmodel (#68) 2020-08-23 13:00:13 -06:00
Neil O'Toole
58ccfc9ded
Json driver; refactoring of core packages (#66)
* Type Detector refactor

* json driver impl; refactoring of source.Files reader et al

* working on kind detector

* significant switcheroo of packages

* partway throught refactoring Kind

* major package switcheroo for Kind
2020-08-23 04:42:15 -06:00
Neil O'Toole
65259754f5
Improvements to metadata (sq inspect) (#61)
* improvements to sqlite source metadata query

* table metadatawriter colorization tuning; minor testh.Helper refactoring

* cleanup of metadata for multiple drivers

* more cleanup of source/table metadata

* yet more cleanup of source/table metadata

* improvements to mysql SourceMetadata

* improvements to mysql TableMetadata

* yet more fiddling with mysql metadata
2020-08-15 15:06:40 -06:00
Neil O'Toole
1ceb50e795
SQL batch insert (#58)
* initial refactoring for the numRows param

* work on driver.NewBatchInsert

* work on NewBatchInsert

* batch insert seems to work

* switched testh.Insert to use BatchInsert

* doc cleanup

* batch insert for dbwriter and csv

* removed unneeded NumRows from driver.StmtExecer

* minor tidyup
2020-08-12 12:24:01 -06:00
Neil O'Toole
3e10868a29
Switched pkg sakila "const" slices to functions (#57)
* replaced sakila source slices with funcs

* missed some pkg sakila slices to refactor
2020-08-10 09:16:44 -06:00
Neil O'Toole
fd4ae53f31 codebase refactor 2020-08-06 11:58:47 -06:00