Error messages for these four are now a bit fancier and more
consistent. But not yet optimised for machine readability.
Cf #1436.
Added to hledger-lib: chomp1, linesPrepend[2].
A "hledger check" argument may now be a quoted string containing
the check name followed by space-separated arguments, for
checks which make use of those. This means the check command
can replicate "check-dates --unique" and (in principle)
"check-fancyassertions ASSERTIONS..". Eg:
hledger check "dates --unique"
I think it'll be better for checks to take no arguments or options,
so this is probably just a transitional feature for compatibility.
Command-line account aliases now also affect transactions read
from these formats (not just journal format).
lib: journalApplyAliases, transactionApplyAliases, postingApplyAliases
helpers have been added.
Work on hledger-web tests showed some bad behaviour, in particular
journalReloadIfNewer would always reload a journal read from a string
or stdout. This is now fixed, and an ugly read.show conversion has
been cleaned up.
Hledger.Cli.Utils API changes:
removed:
- journalSpecifiedFileIsNewer
- fileModificationTime
added:
- utcTimeToClockTime
changed:
- journalFileIsNewer now requires a file argument
Journal keeps a new piece of parsing state, a decimal mark character,
which can optionally be set to force the number format expected by all
amount parsers.
querystring_.
This helps deal with tricky quoting issues, as we no longer have to make
sure everything is quoted properly before merging it into a string.
This introduces some new helper functions which are exactly the same
as what we had before, but do not call
normaliseMixedAmountSquashPricesForDisplay, so that we can use the new
functions for displaying Transaction and Posting. It also goes through
and gets rid of most uses of the old showMixed* functions which would
benefit from using the new interface.
This changes showMixedAmountElided so that the width to elide to is
given as an argument, rather than fixed at 22 characters. This
actually uses the new renderTable interface. Mostly this is just an
internal change, but since we have more information about the widths
of things, we can actually get rid of some superfluous spaces in the
budget report output, previously there to make sure it stayed aligned
with the largest reasonable contents.
This gives renderTable a little more customisation. Before any of the
commits of this PR, render would just receive a string to display in
each cell. After the second commit of this PR it would also receive a
width of the string (in place of stripping ANSI sequences and then
calculating the width). After this commit, it now also takes an
alignment, so you can make cells left or right aligned. The function
render calls renderTable with appropriate options to give the same
behaviour as before. Also, previously render would always put a border
around the table. We would take this output, and would sometimes strip
the border by dropping the first and last rows, and first and last
characters of every row. I've just added an option to control whether
to put the border in, so we can just not add it in the first place,
rather than stripping it later. Note that this is again just defining
helper functions; this extra power is not yet used anywhere.
Previously showMixedAmountElided would show two amounts and then the
elision string if necessary. Now it will display as many Amounts as it
can subject to the condition that the amounts plus the elision string
fit within 22 characters.
checkRawOpts has been a no-op for at least four years, and
checkReportOpts only makes sure that depth_ is positive, which is taken
care of by the maybeposintopt parser.
stripAnsi is called many times during rendering (by strWidth), so
should be fast. It was originally a regex replacement, and more
recently a custom parser. The parser was slower, particularly the one
in 1.19.1. See #1350, and this rough test:
time118ish = timeIt $ print $ length $ concat $ map (fromRight undefined . regexReplace (toRegex' "\ESC\\[([0-9]+;)*([0-9]+)?[ABCDHJKfmsu]") "") testdata
time119 = timeparser (many (takeWhile1P Nothing (/='\ESC') <|> "" <$ ansi))
time1191 = timeparser (many ("" <$ try ansi <|> pure <$> anySingle))
timeparser p = timeIt $ print $ length $ concat $ map (concat . fromJust . parseMaybe p) testdata
testdata = concat $ replicate 10000
[ "2008-01-01 income assets🏦checking $1 $1"
, "2008-06-01 gift assets🏦checking $1 $2"
, "2008-06-02 save assets🏦saving $1 $3"
, " assets🏦checking ..m$-1\ESC[m\ESC[m $2"
, "2008-06-03 eat & shop assets:cash ..m$-2\ESC[m\ESC[m 0"
, "2008-12-31 pay off assets🏦checking ..m$-1\ESC[m\ESC[m ..m$-1\ESC[m\ESC[m"
]
ghci> time118ish
4560000
CPU time: 0.17s
ghci> time119
4560000
CPU time: 0.91s
ghci> time1191
4560000
CPU time: 2.76s
Possibly a more careful parser could beat regexReplace. Note the
latter does memoisation, which could be faster and/or could also use
more resident memory in some situations.
Ideally we would calculate all widths before adding ANSI colour codes,
so we wouldn't have to wastefully strip them.
This PR #1330, addressing #1312 (parseQuery is partial) and #1245
(internal server error).
User-visible changes:
- hledger-web now handles malformed regular expressions
(eg, a query consisting of the single character `?`) gracefully,
showing a tidy error message instead "internal server error".
API/internal changes:
- The Regex type alias has been replaced by the Regexp ADT, which
contains both the compiled regular expression (so is guaranteed to
be usable at runtime) and the original string (so can be serialised,
printed, compared, etc.) A Regexp also knows whether is it case
sensitive or case insensitive. The Hledger.Utils.Regex api has changed.
- Typeable and Data instances are no longer derived for hledger's
data types; they were redundant/no longer needed
- NFData instances are no longer derived for hledger's data types.
This speeds up a full build by roughly 7%. But it means we can't
deep-evaluate hledger values, or time hledger code with Criterion.
https://github.com/simonmichael/hledger/pull/1330#issuecomment-684075129
has some ideas on this.
- Query no longer has a custom Show instance
- Some internal use of regexps was replaced by text replacement or
parsers.
- Hledger.Utils.String: quoteIfNeeded now actually escapes quotes in
strings; dropped escapeQuotes
- Hledger.Utils.Tree: dropped some old utilities
- dropped some obsolete code for the old --display option
Merge branch 'regexp' into master
matchesAccount_
matchesAmount_
matchesCommodity_
matchesPosting_
matchesPriceDirective_
matchesTags_
matchesTransaction_
These don't yet have tests of their own, but were converted
mechanically from the originals which should help.
;areg: debug output
;areg: show a title indicating which account was picked
This might be a bit of a pain for scripting, but otherwise it can be
quite confusing if your argument matches an account you didn't expect.
;areg: improve CSV headings
;areg: show at most two commodities per amount
accountTransactionsReport now filters transactions more thoroughly, so
eg transactions dated outside the report period will not be shown.
Previously the transaction would be shown if it had any posting dated
inside the report period. Possibly some other filter criteria now get
applied that didn't before. I think on balance this will give slightly
preferable results.
bal --budget was always showing the period as column heading,
as if for a change report. With --cumulative or --historical
it should show the end date, like other balance reports. Cf
https://hledger.org/hledger.html#multicolumn-balance-report.
This is an API change, but it seems better than having additional
colour-supporting variants and trying to avoid duplicated code.
I stopped short of changing showAmount, so cshowAmount still exists.
Multicolumn balance reports showing many commodities tend to become
unreadably wide, especially in tree mode. Now by default we show at
most two commodities, and a count of the rest if there are more than
two. This should help keep reports somewhat readable by default.
SMorgan:
This PR aims to accomplish two major goals:
- Get boring parent ellision working for multiBalanceReport
- Remove the special BalanceReport code, and just use multiBalanceReport
I believe it does both, with the following additional benefits:
A refactor of multiBalanceReportWith, to make the structure easier to follow, and with a clearer division of responsibilities
All decisions for how an account name is to be displayed are now made in multiBalanceReport, rather than scattered around the code base
Some miscellaneous improvements in account name rendering, including --drop now working with MultiBalanceReports, and addressing some of #373
Algorithmic changes:
- Using HashMap AccountName (Map DateSpan Account) instead of [[MixedAmount]] is new. I admit I didn't profile this change (though given the nubs and lookups, I thought it was appropriate), so I'm glad it produces a speedup.
- Producing the starting balances no longer calls the whole balanceReport, just the first few functions to get what it needs.
- displayedAccounts is completely rewritten. Perhaps one subtle thing to note is that in tree mode it no longer excludes nodes with zero inclusive balance unless they also have zero exclusive balance.
SMichael:
I'll mark the passing of the old multiBalanceReport, into which I poured many an hour :). It is in a way the heart (brain ?) of hledger - the key feature of ledgerlikes (balance report) and a key improvement introduced by hledger (tabular multiperiod balance reports). You have split that 300-line though well documented function into modular parts, which could be a little harder to understand in detail but are easier to understand in the large and more amenable to further refactoring. Then you fixed some old limitations (boring parent eliding in multi period balance reports, --drop with tree mode reports), allowing us to drop the old balanceReport and focus on just the new multiBalanceReport. And for representing the tabular data you replaced the semantically correct but inefficient list of lists with a map of maps, speeding up many-columned balance reports significantly (~40%). Last and not least you made it really easy to review. Thanks @Xitian9, great work.
This works with glob patterns too, applying the prefix to each path.
This can be useful when included files don't have the standard file
extension, eg:
include timedot:2020*.md
As mentioned by netvor on IRC, the unbalanced transaction error was
not too clear when postings all have the same sign.
Some other wording has been clarified, and the main error message is
now shown on multiple lines for readability (at the cost of
predictability/grepability..)
There's also a probably unnoticeable change: selecting which parts of
the error to show is now based on display precisions (reusing the
balanced check logic), rather than original precisions.
The ordering of journalfieldnames is changed in order to comply with the requirement stated in the comment:
"Names must precede any other name they contain, for the parser".
If left unchanged, "account1" would precede "account11", "account12", and so on, which would break the parsing.
With the new ordering, "account11" precedes "account1".
We now accept (but still ignore) a fixed or nonfixed ({=} or {}) lot
price following a posting amount, and it may appear before or after a
transaction price (@ or @@). And it may no longer appear after a
balance assertion.
Also: fixedlotpricep renamed to lotpricep, now also parses non-fixed
lot prices. A bit of amount parsers cleanup.
orgstruct-mode was dropped from org 9.2, and I shouldn't have been
forcing it on anyway.
The new config allows its "replacement", outshine-mode, to do similar
code folding when you press tab on any of the lines matching
outline-regexp. But only if you patch it as mentioned at
https://github.com/alphapapa/outshine/issues/77.
Enable it by, eg: (add-hook 'haskell-mode-hook 'outshine-mode)
The change for hledger 1.17.1 broke one of my csv rules, where I used
`amount` but then tried to override it with `amountN`s in a
conditional block; the two clashed. Now in that situation any
`amountN`s take precedence, causing `amount` to be ignored entirely.
Also clarified the "too many non-zero amounts" error message a bit.
A rewrite and simplification of the posting-generating code. The
"special handling for pre 1.17 rules" should now be less noticeable.
amount1/amount2 no longer force a second posting or explicit amounts
on both postings. (Only amount/amount-in/amount-out do that.)
Error messages and handling of corner cases may be more robust, also.
The include directive now tries just one reader, based on the file
extension and defaulting to journal, like the rest of hledger.
(It doesn't yet handle a reader prefix.)
Reader-finding utilities have moved from Hledger.Read to
Hledger.Read.JournalReader so the include directive can use them.
Reader changes:
- rExperimental flag removed
- old rParser renamed to rReadFn
- new rParser field provides the actual parser.
This seems to require making Reader a higher-kinded type, unfortunately.
For a long time hledger has auto-detected the file format when it's
not known, eg when reading from a file with unusual extension (like
.dat or .txt), or from standard input (-f-), or when using the include
directive (which currently ignores file extensions).
Auto-detecting has been done by trying all readers until one succeeds.
This could guess wrong in some cases, but it was so rare that it has
been working fine.
Recently, more conveniences have been added to timedot format,
increasing its overlap with journal format, which makes this kind of
auto-detection unreliable.
Auto-detection and auto-detection failures are (probably) still pretty
rare in practice. But when it does happen it's confusing, giving
misleading errors or false successes (eg printing timedot entries
instead of a journal error).
For predictability and to minimise confusion, hledger no longer tries
to guess; when there's no file extension or reader prefix, it assumes
journal format. To specify one of the other formats, you must use a
standard file extension (.timeclock, .timedot, .csv, .ssv, .tsv), or a
reader prefix (-f csv:foo.txt, -f timedot:-).
For now, the include directive still tries to autodetect
(journal/timeclock/timedot), and this can't be overridden; it will be
fixed later.
Experimental; testing and feedback welcome.
Now, org headlines before the first day entry are ignored,
regardless of content.
Note, blank lines inside a day entry are not allowed, currently.
It's now easier to be both valid journal and valid timedot at the same
time, so guessing the format of stdin is unreliable, and some tests
are failing. See following commit.
