Commit Graph

822 Commits

Author SHA1 Message Date
Simon Michael
2b18715885 fix:csv: fix tag: queries on CSV data (#2114) 2023-11-20 21:55:11 -10:00
Simon Michael
e92ab28cce imp:reading: better timing of strict checks and .latest writing (#2113)
Strict checks now run only once, at end of the high level read operation,
and not for each individual file; this fixes some spurious --strict failures,
like account declarations not affecting a sibling file as they should.

And .latest file writing now happens as the last step, after passing
strict checks. This is mainly for the import command, but it also
means that hledger print --new now does not update .latest files
if strict checks are failing.

The file reading API has been improved and documented in more detail.
2023-11-16 21:48:43 -10:00
Simon Michael
f8ffd9cdda imp: more precision handling fixes, debug output, test updates (precisiongeddon)
This and the preceding commits were "work in progress" that got out of control.
There's more to do, but this one brings these precision-related improvements
(at least):

When "infinite decimals" arise, they are now generally shown with
8 decimal digits rather than 255.

print and prices no longer add trailing decimal zeros unnecessarily.

Some code has been refactored or given more debug output.

All tests have been updated to match the recent changes.
2023-11-08 13:59:18 -08:00
Jonathan Dowland
8bfa382c68 feat: import: interpolate regex matches in field templates (#2009)
Replace occurrences of '\N' (where N is a positive number) in field
templates with the corresponding regular expression match group, if it
exists.

E.g. Warp the date to the first of the month for the second posting

    if %date (....-..)-..
        comment2 date:\1-01

E.g. Strip a prefix from an imported account name

    if %account1 liabilities:jon:(.*)
        account1 \1

Fixes #2009.

Signed-off-by: Jonathan Dowland <jon@dow.land>
2023-11-08 13:49:39 -08:00
Jonathan Dowland
c619e387ea ;fix: import: minor typo 2023-11-08 13:49:39 -08:00
Simon Michael
54b4c2ed22 fix: auto postings: detect redundant costs properly (fix #2110) 2023-11-07 04:38:39 -08:00
Peter Sagerson
efcea0600a feat: cli: Add tsv output (#869)
All commands that suport csv output now also support tsv output. The
data is identical, but the fields are separated by tab characters and
there is no quoting or escaping. Tab, carriage return, and newline
characters in data are converted to spaces (this should rarely if ever
happen in practice).
2023-11-06 16:46:04 -08:00
Simon Michael
63d41b30db ;doc: haddock, comment fixes 2023-10-21 17:58:43 +01:00
Simon Michael
5a72b9e9ea feat:print: add --round option for more control of precisions (#2085) 2023-10-18 13:56:24 +01:00
Simon Michael
c13c13ab1f lib!: use styleAmounts in more places; add rounding strategies 2023-10-18 13:56:24 +01:00
Simon Michael
94ab8b1ed7 dev: journalApplyCommodityStyles -> journalStyleAmounts 2023-10-18 13:56:24 +01:00
Simon Michael
f8027abb44 dev:print: refactor, add AmountStyle "rounding strategy"
Changes to enable more control of "rounding" behaviour
(ie, choosing display precisions for amounts).

This reverts 1.31's change of asprecision, making it a non-Maybe
again, and adds a new asrounding field providing more control over how
a target display precision is applied to existing amounts (two options
for now, more later). Functionality is in an interim state (reports do
no rounding).
2023-10-18 13:56:24 +01:00
Simon Michael
9ead8a64e1 fix:timedot: parse unitful quantities more accurately (fix #2096)
A quantity with a unit like "15m" was being parsed internally as
0.249999999... rather than 0.25 (and since hledger 1.21, printed that
way also). Now we round such quantities to two places during parsing,
to get exact quarter-hour amounts.
2023-10-17 12:05:10 +01:00
Simon Michael
e8da1f9f25 fix: balance-assigned amounts affect styles again (fix #2091) 2023-10-14 08:38:20 +01:00
bobobo1618
9fb5740045 Add support for negating a Matcher
https://github.com/simonmichael/hledger/issues/2054
2023-10-05 10:22:01 +01:00
Simon Michael
97be1646f1 dev: AmountStyle: make setting the precision optional
This simplifies the code for styling amounts with or without precision.
But it complicates the semantics (Nothing is useful only when setting style).
Not sure if it's the best way.
2023-09-02 06:46:14 +01:00
Simon Michael
85845e51b2 dev: AmountStyle: rename, reorder fields more mnemonically
Since this type is about to change anyway.
2023-09-02 06:46:14 +01:00
Simon Michael
5f1bf0404a imp:check: more intuitive "balanced..." check names
balancedwithautoconversion -> autobalanced
balancednoautoconversion   -> balanced
2023-08-22 08:38:52 +01:00
Simon Michael
1cfcb7fa76 ;cln: journalAddInferredEquityPostings -> journalInferEquityFromCosts 2023-07-04 13:21:55 -10:00
Simon Michael
e832c344c1 dev: comment 2023-05-20 17:29:06 -10:00
Simon Michael
029b59093b feat: csv: rules files can be read directly; data file can be specified
CSV rules files can now be read directly, eg you have the option of
writing `hledger -f foo.csv.rules CMD`. By default this will read data
from foo.csv in the same directory.  But you can also specify a
different data file with a new `source FILE` rule. This has some
convenience features:

- If the data file does not exist, it is treated as empty, not an
  error.

- If FILE is a relative path, it is relative to the rules file's
  directory. If it is just a file name with no path, it is relative
  to ~/Downloads/.

- If FILE is a glob pattern, the most recently modified matched file
  is used.

This helps remove some of the busywork of managing CSV downloads.
Most of your financial institutions's default CSV filenames are
different and can be recognised by a glob pattern.  So you can put a
rule like `source Checking1*.csv` in foo-checking.csv.rules,
periodically download CSV from Foo's website accepting your browser's
defaults, and then run `hledger import checking.csv.rules` to import
any new transactions. The next time, if you have done no cleanup, your
browser will probably save it as something like Checking1-2.csv, and
hledger will still see that because of the * wild card. You can choose
whether to delete CSVs after import, or keep them for a while as
temporary backups, or archive them somewhere.
2023-05-19 09:09:21 -10:00
Simon Michael
577e4b6347 fix!: csv: skip now counts non-blank lines more robustly (fix #2024)
Inner empty lines were not being skipped automatically, contrary to
docs. Now all empty lines are skipped automatically, and the `skip`
rule is needed only for non-empty lines, as intended.

This may be a breaking change: it's possible that the `skip` count
might need to be adjusted in some CSV rules files.
2023-05-11 17:06:12 -10:00
Simon Michael
69be1d4ef7 ;dev: csv: refactor, clarify 2023-05-11 15:44:56 -10:00
Simon Michael
70d4c0c638 ;dev: csv: refactor, clarify 2023-05-11 15:35:05 -10:00
Simon Michael
755c3d3dbb ;dev: csv: refactor 2023-05-11 15:34:31 -10:00
Simon Michael
bbecd611f1 imp: timedot: make one multi-posting transaction per date line (#1754)
Descriptions and comments are now more straightforward and similar to
journal format.
2023-05-03 19:18:35 -10:00
Simon Michael
50349f81f7 imp: timeclock: support comments and tags (fix #1220)
Breaking change: previously timeclock descriptions could contain
semicolons. Now a semicolon in the description will end it and
start a comment (which may contain tags).
2023-05-03 00:05:11 -10:00
Simon Michael
f7f86a709b imp: timedot: fix day description/comment parsing; parse posting comments/tags 2023-05-02 17:03:03 -10:00
Simon Michael
1b19f3d330 dev: timedot: clarify, cleanup 2023-05-02 17:02:30 -10:00
Simon Michael
a771c8fc19 imp: revert 2b5194238 imp: generate auto postings on forecast transactions by default (#2027)
I found at least one user for whom this would be a breaking change
(they generate forecast txns, and have auto posting rules, but don't
want the latter applied to the former). I guess it's better to keep
things as they were for now: if you need auto postings on your
forecast txns you must use two flags, --forecast --auto.
2023-04-30 08:08:26 -10:00
Simon Michael
7f713f6a44 imp: Generated/modified txns/postings are now tagged only with --verbose-tags 2023-04-29 18:25:11 -10:00
Simon Michael
2b5194238b imp: generate auto postings on forecast transactions by default 2023-04-29 16:00:14 -10:00
Chris Lemaire
cddbae6467 queries: Prefix boolean queries with expr:
Boolean queries are now prefixed with an 'expr:' prefix, making them
completely separable from old queries and making the addition of them a
little more migration proof.

The tests are updated accordingly, changes made to the tests previously
are removed and extra cautious documentation is also removed.
2023-03-27 10:29:26 -10:00
Simon Michael
c790aa6145 ;dev: lib: also build with GHC 9.6.1; add base-compat 2023-03-14 10:42:48 -10:00
Simon Michael
fa70f160ae imp: partial/inferred dates are flexible, full dates are not (#1982)
DateSpans are now now aware of exact/flexible dates.
2023-02-17 07:24:19 -10:00
Simon Michael
5537a251f3 imp: journal: periodic txns need not start on an interval boundary
Eg, ~ monthly from 1/15 now works, instead of giving an error message.
2023-02-17 07:24:19 -10:00
Simon Michael
7a9b0fd94c feat: check: the tags check checks tag names 2023-02-16 11:56:22 -10:00
Simon Michael
c51ebe7b8c imp: journal: remove the "adjacent conversion postings" check again
I find that single conversion postings are sometimes needed, eg with
paypal.
2023-02-01 13:30:21 -10:00
Simon Michael
de330bdfce imp: journal: check for adjacent conversion postings only in strict mode
This avoids breaking existing journals if, say, they happen to have
a single "equity:trading" posting somewhere.
2023-01-25 19:00:29 -10:00
Simon Michael
76f8eaf91a imp: journal: ignore redundant costs; --infer-costs more robustly
You can now write both @/@@ costs and corresponding equity conversion postings
in a transaction at any time, not just when using --infer-costs or --infer-equity.
hledger will recognise the redundancy and ignore it.

One thing has become more strict: hledger now requires conversion postings
to occur in adjacent pairs; an odd number of them is not allowed.
(Conversion postings are postings to accounts of type `V`/`Conversion`,
or named `equity:conversion`, `equity:trade`, `equity:trading`,
or subaccounts of these.)

And, --infer-costs now works in transactions with an implicit amount
(inferring costs from equity now happens after transaction balancing,
not before).
2023-01-25 19:00:29 -10:00
Chris Lemaire
43c9f018dc journal: Account directives reject account names with brackets
Previously, the accounts passed to account directives would be stripped
of their surrounding brackets, but the required behaviour is to have
account directives plain reject bracketed accounts. This change ensures
that accounts in account directives may not start with a bracket
character.
2023-01-24 13:40:01 -10:00
Simon Michael
9497472e64
Merge branch 'master' into 1984-source-name-and-position-for-forecast 2023-01-23 16:04:00 -10:00
Chris Lemaire
4ada2a3fbc journal: Fully unbracket AccountNames in account directives
Currently an account name like "a:(aa)" will not have (aa) unbracketed.
However, this seems reasonable since the full name is unbracketed and
thus will not be confused with virtual or virtual-balanced posting.
2023-01-23 12:43:13 -10:00
Chris Lemaire
9443fe0e0d journal: json: Add source positions to forecast transactions 2023-01-21 22:30:30 +01:00
Simon Michael
5b27554f0a dev: note why inferring costs from equity before balancing 2023-01-19 20:57:02 -10:00
Simon Michael
cf469c080d lib: journalInferCosts -> journalInferCostsFromEquity 2023-01-19 20:25:41 -10:00
Simon Michael
28eb8be4fa lib: BalancingOpts{infer_transaction_prices_ -> infer_balancing_costs_} 2023-01-19 20:21:31 -10:00
Simon Michael
dfea6eaee2 lib: Hledger.Data.Journal: journalAddPricesFromEquity -> journalInferCosts 2023-01-19 09:17:56 -10:00
Simon Michael
dfebf3174c imp: csv: check assigned account names are valid (parseable) (#1978) 2023-01-11 21:42:47 -10:00
Simon Michael
a9b63bb694 fix: csv: skip header lines before attempting to parse records (#1967) 2022-12-27 12:21:20 -10:00