Commit Graph

28 Commits

Author SHA1 Message Date
Neil O'Toole
452a91547c
Better handling of download cache refresh (#366)
* A failed download cache refresh no longer destroys the previous cache.
2024-01-26 15:18:38 -07:00
Neil O'Toole
db55986980
#307: Ingest cache (#354)
- Support for ingest cache, download cache, and progress bars.
2024-01-14 18:45:34 -07:00
Neil O'Toole
2de993acaa
Sort imports using goimports-reviser (#331) 2023-11-19 18:06:36 -07:00
Neil O'Toole
f07edef14d
Add flag --src.schema (#326)
* Support for --src.schema in commands "slq", "sql", and "inspect"
2023-11-18 17:05:48 -07:00
Neil O'Toole
6b613d9adc
#279: SQLite virtual tables (#304)
* sqlite: initial extensions support, including virtual tables and fts5
* sqlite: virtual table columns now report type
2023-08-21 10:05:17 -06:00
Neil O'Toole
9c5836ef1c
#191: XLSX driver auto-detects header row (#284)
* xlsx driver now detects header row.
2023-07-08 09:21:27 -06:00
neilotoole
82dc378612 Disable XLSX test on windows 2023-06-22 12:30:38 -06:00
Neil O'Toole
a5c4f9d6bd
#263: Inspect --overview; improved output (#273)
* Refactor tablew md writer

* wip: cmd inspect -v

* cmd inspect output improved

* cmd inspect tests

* cmd inspect: help text
2023-06-21 23:48:58 -06:00
Neil O'Toole
964417dba7
Use flag --compact instead of --pretty (#216) 2023-05-05 11:41:22 -06:00
Neil O'Toole
e5db7990bc
#199: Kill RunContext.Log (#211)
* refactor: RunContext.Log is dead

* Fiddling with options

* Changed default max conns to zero (no limit)

* Linting
2023-04-30 08:18:56 -06:00
Neil O'Toole
958d509088
Source groups (#198)
* Implemented source groups mechanism.
2023-04-15 16:28:51 -06:00
Neil O'Toole
ebb88b3a44
CSV empty string now treated as NULL (#190) 2023-04-09 19:29:13 -06:00
Neil O'Toole
ad23d5c116
Auto-detect CSV headers (#188)
* wip: initial header detection

* wip: refactored csv driver

* wip: Fixes to kind.Detector

* typo

* Refactor pkg csv files

* Update to changelog

* workflow: now running codacy and codeql on cron, not on push/pr
2023-04-09 08:44:27 -06:00
Neil O'Toole
54a0155ed6
Deal with lots of connections, especially for sq inspect. (#186)
* wip: deal with lots of connections

* Clean up TestDatabase_SourceMetadata_concurrent

* Fixed error message
2023-04-08 12:09:27 -06:00
Neil O'Toole
d9b56eea8a
Implement groupby() (#161)
* test: slq2sql more test cases; now executes the generated query

* groupby: first test case working against all DBs

* sakila: added sqlserver 2019 source

* groupby: tests for groupby synonyms

* groupby: final touches
2023-03-26 02:01:41 -06:00
Neil O'Toole
a1a89ee9dd
Support table and column names with spaces. (#156)
* sakila: initial test data

* sakila: more test data

* sakila: yet more test data setup

* whitespace cols: now working for sqlite

* grammar cleanup

* whitespace cols: now working inside count() func for sqlite

* whitespace cols: tests mostly passing; begining refactoring

* grammar: refactor handle

* grammar: more refactoring

* grammar: rename selElement to selector

* wip

* all tests passing

* all tests passing

* linting

* driver: implement CurrentSchema for all driver.SQLDriver impls

* driver: tests for AlterTableRename and AlterTableRenameColumn

* undo reformat of SQL

* undo reformat of SQL

* undo reformat of SQL

* undo reformat of SQL
2023-03-22 00:17:34 -06:00
Neil O'Toole
e2a9605ec6
Chore/sqlite3 upgrade (#129)
* wip

* made progress on new SQLite type stuff

* we're getting places

* largely working now

* tidying up

* almost there

* time to deal with failing tests

* seemed to have fixed sqlserver issue

* seems to all be fixed

* minor tidy up
2022-12-23 21:05:26 -07:00
Neil O'Toole
fab365f43c
gofumpt on files (#122)
* gofumpt on files

* more gofumpt
2022-12-18 01:35:59 -07:00
Neil O'Toole
540adfac58
Lint long lines (#115)
* lint config now sorts results

* linted long lines

* linted long lines
2022-12-17 19:43:53 -07:00
Neil O'Toole
5aebc04356
Fixed issues with files and databases not being closed correctly (#73)
* fiddling with scratch database close order

* files debugging

* files debugging2

* files debugging3

* files debugging 4

* files debugging 5

* didn't close the ReadCloser in csv import

* more closing cleanup
2020-12-30 11:57:58 -07:00
Neil O'Toole
06900f8c84
fixed some panics with oddly-shaped xlsx imports (#71) 2020-11-02 10:40:29 -07:00
Neil O'Toole
0f3bc26f52
JSON driver: initial jsona driver (#69)
* jsona driver appears to be mostly working

* jsona verified as working

* minor tidy
2020-08-23 20:32:59 -06:00
Neil O'Toole
58ccfc9ded
Json driver; refactoring of core packages (#66)
* Type Detector refactor

* json driver impl; refactoring of source.Files reader et al

* working on kind detector

* significant switcheroo of packages

* partway throught refactoring Kind

* major package switcheroo for Kind
2020-08-23 04:42:15 -06:00
Neil O'Toole
65259754f5
Improvements to metadata (sq inspect) (#61)
* improvements to sqlite source metadata query

* table metadatawriter colorization tuning; minor testh.Helper refactoring

* cleanup of metadata for multiple drivers

* more cleanup of source/table metadata

* yet more cleanup of source/table metadata

* improvements to mysql SourceMetadata

* improvements to mysql TableMetadata

* yet more fiddling with mysql metadata
2020-08-15 15:06:40 -06:00
Neil O'Toole
3e10868a29
Switched pkg sakila "const" slices to functions (#57)
* replaced sakila source slices with funcs

* missed some pkg sakila slices to refactor
2020-08-10 09:16:44 -06:00
Neil O'Toole
8b1bd4c03a
replaced sakila source slices with funcs (#56) 2020-08-09 08:40:46 -06:00
Neil O'Toole
915e01fdda
added Go dependency cache to go.yml workflow, and go test now covers the full codebase (#53)
* added Go dependency cache to go.yml workflow

* more windows path issues

* docs

* restricting workflow triggers to master

* wrong cleanup order for downloaded files

* de-restricting workflow triggers, for now

* more out-of-order cleanup issues that only manifested on windows

* more out-of-order cleanup issues

* removed mediatype check from TestFetchFile; FetchFile is deprecated and not worth fixing

* tracked down out-of-sequence close/remove issue with sqlite

* probably long-running test failure

* disabling long-running tests that are breaking CI

* switching to go test -short because tests seem to be taking too long in CI
2020-08-08 15:23:30 -06:00
Neil O'Toole
fd4ae53f31 codebase refactor 2020-08-06 11:58:47 -06:00