Strip prices after valuing postings in PostingsReport.
Use renderRow interface for Register report.
For reg -f examples/10000x10000x10.journal, this results in:
- Heap allocations decreasing by 55%, from 68.6GB to 31.2GB
- Resident memory decreasing by 75%, from 254GB to 65GB
- Total (profiled) time decreasing by 55%, from 37s to 20s
former being a simple wrapper around the latter.
This removes the need for the showNormalised option, as showMixedAmountB
will always showNormalised and showAmountsB will never do so.
We also strip prices from MixedAmount before displaying if not displaying prices.
Exceptions are for dealing with the pamount field, which is really just
dealing with an unnormalised list of amounts.
This creates an API for dealing with MixedAmount, so we never have to
access the internals outside of Hledger.Data.Amount.
Also remove a comment, since it looks like #1207 has been resolved.
supplant the old interface, which relied on the Num typeclass.
MixedAmount did not have a very good Num instance. The only functions
which were defined were fromInteger, (+), and negate. Furthermore, it
was not law-abiding, as 0 + a /= a in general. Replacements for used
functions are:
0 -> nullmixedamt / mempty
(+) -> maPlus / (<>)
(-) -> maMinus
negate -> maNegate
sum -> maSum
sumStrict -> maSum
Also creates some new constructors for MixedAmount:
mixedAmount :: Amount -> MixedAmount
maAddAmount :: MixedAmount -> Amount -> MixedAmount
maAddAmounts :: MixedAmount -> [Amount] -> MixedAmount
Add Semigroup and Monoid instances for MixedAmount.
Ideally we would remove the Num instance entirely.
The only change needed have nullmixedamt/mempty substitute for
0 without problems was to not squash prices in
mixedAmount(Looks|Is)Zero. This is correct behaviour in any case.
price directives after the last transaction/posting date if using
--value=end.
Also enlarges the reportspan to encompass full intervals for budget
goals.
both the quantity and the cost are zero. This is usually what you want,
but if you do only want to check whether the quantity is zero, you
can run mixedAmountStripPrices (or similar) before this.
(multiply|divide)(Mixed)?Amount now also multiply or divide the
TotalPrice if it is present, and the old
(multiply|divide)(Mixed)?AmountAndPrice functions are removed.
internally, closing a big space leak.
This also now combines Amounts with TotalPrices in the same commodity
when normalising; amounts with TotalPrices were previously never
combined.
aquantity.
Journal entries still require a positive @@ price, but now the sign is
set after parsing, rather than when converting in amountToCost.
The reason for this change is that, if we're going to perform arithmetic
on Amount with TotalCost, then the presence of aquantity=0 means that
amountToCost would render the total cost as 0, because signum 0 == 0.
This makes journal entries like the following impossible to balance:
2000-01-01
a 0 @@ 10 A
b -10 A
add --debug=1 shows the top hits for similar past transactions.
added:
Hledger.Cli.Utils.journalSimilarTransaction
provides --debug=1 output
changed:
Hledger.Cli.Commands.Add.transactionsSimilarTo -> Hledger.Data.Journal.journalTransactionsSimilarTo
now takes an extra number-of-results argument
It would not add the tag when a comment already existed.
This affected hledger-print-location.hs and probably
the generated-transaction: tag in periodic transactions.
For clarity; infer-value was too vague. The old spelling remains
supported for compatibility, but is now deprecated.
When typing, --infer-market or even --infer (for now) is sufficient.
On the accounts screen and register screen we round amounts according
to commodity styles, but when you drill down to a transaction you
probably want to see the unrounded amounts.
Ensures parseable and more sensible-looking output in more cases, and behaves more like Ledger's print.
There is still an issue with adding trailing zeroes, which would be nice to prevent.
Since this is option is now just an alias for -B/--cost, and since it
may be removed soon, we make it undocumented, though it will still
behave as before. --value=cost,COMM is unsupported as well.
independently.
You can now combine costing and valuation, for example "--cost
--value=then" will first convert to costs, and then value according to
the "--value=then" strategy. Any valuation strategy can be used with or
without costing.
If multiple valuation and costing strategies are specified on the
command line, then if any of them include costing
(-B/--cost/--value=cost) then amounts will be converted to cost, and for
valuation strategy the rightmost will be used.
--value=cost is deprecated, but still supported and is equivalent to
--cost/-B. --value=cost,COMM is no longer supported, but this behaviour can be
achieved with "--cost --value=then,COMM".
costing and valuation.
This currently is given a dummy NoCost argument and is equivalent to
"maybe id (*ApplyValuation ...)", but provides a constant interface so
that internal behaviour can be changed freely.
simplifySign now covers a few more sign combinations that might arise.
And in particular, it strips a standalone sign with no number,
which simplifies sign flipping with amount-in/amount-out.
Also adds a postingDate argument to amountApplyValuation, and re-orders
the ValuationType and (Transaction/Posting) arguments to
(transaction/posting)ApplyValuation, to be consistent with
amountApplyValuation.
Searching for prices during valuation no longer now properly excludes
price loops, avoiding near infinite looping with certain
configurations of market prices. Also we now always use a direct price
when available, rather than searching unnecessarily.
Price searching progress info, useful for troubleshooting, is now
displayed with --debug=2.
There could still be some corner cases we don't handle correctly. We
now give up with an error message if the searched price chains get too
long (> 1000). More importantly, we should also give up if the search
iterates too many times, but this is not done yet.
Error messages for these four are now a bit fancier and more
consistent. But not yet optimised for machine readability.
Cf #1436.
Added to hledger-lib: chomp1, linesPrepend[2].
A "hledger check" argument may now be a quoted string containing
the check name followed by space-separated arguments, for
checks which make use of those. This means the check command
can replicate "check-dates --unique" and (in principle)
"check-fancyassertions ASSERTIONS..". Eg:
hledger check "dates --unique"
I think it'll be better for checks to take no arguments or options,
so this is probably just a transitional feature for compatibility.
Command-line account aliases now also affect transactions read
from these formats (not just journal format).
lib: journalApplyAliases, transactionApplyAliases, postingApplyAliases
helpers have been added.
Work on hledger-web tests showed some bad behaviour, in particular
journalReloadIfNewer would always reload a journal read from a string
or stdout. This is now fixed, and an ugly read.show conversion has
been cleaned up.
Hledger.Cli.Utils API changes:
removed:
- journalSpecifiedFileIsNewer
- fileModificationTime
added:
- utcTimeToClockTime
changed:
- journalFileIsNewer now requires a file argument
Journal keeps a new piece of parsing state, a decimal mark character,
which can optionally be set to force the number format expected by all
amount parsers.
querystring_.
This helps deal with tricky quoting issues, as we no longer have to make
sure everything is quoted properly before merging it into a string.
This introduces some new helper functions which are exactly the same
as what we had before, but do not call
normaliseMixedAmountSquashPricesForDisplay, so that we can use the new
functions for displaying Transaction and Posting. It also goes through
and gets rid of most uses of the old showMixed* functions which would
benefit from using the new interface.
This changes showMixedAmountElided so that the width to elide to is
given as an argument, rather than fixed at 22 characters. This
actually uses the new renderTable interface. Mostly this is just an
internal change, but since we have more information about the widths
of things, we can actually get rid of some superfluous spaces in the
budget report output, previously there to make sure it stayed aligned
with the largest reasonable contents.
This gives renderTable a little more customisation. Before any of the
commits of this PR, render would just receive a string to display in
each cell. After the second commit of this PR it would also receive a
width of the string (in place of stripping ANSI sequences and then
calculating the width). After this commit, it now also takes an
alignment, so you can make cells left or right aligned. The function
render calls renderTable with appropriate options to give the same
behaviour as before. Also, previously render would always put a border
around the table. We would take this output, and would sometimes strip
the border by dropping the first and last rows, and first and last
characters of every row. I've just added an option to control whether
to put the border in, so we can just not add it in the first place,
rather than stripping it later. Note that this is again just defining
helper functions; this extra power is not yet used anywhere.
Previously showMixedAmountElided would show two amounts and then the
elision string if necessary. Now it will display as many Amounts as it
can subject to the condition that the amounts plus the elision string
fit within 22 characters.
checkRawOpts has been a no-op for at least four years, and
checkReportOpts only makes sure that depth_ is positive, which is taken
care of by the maybeposintopt parser.
stripAnsi is called many times during rendering (by strWidth), so
should be fast. It was originally a regex replacement, and more
recently a custom parser. The parser was slower, particularly the one
in 1.19.1. See #1350, and this rough test:
time118ish = timeIt $ print $ length $ concat $ map (fromRight undefined . regexReplace (toRegex' "\ESC\\[([0-9]+;)*([0-9]+)?[ABCDHJKfmsu]") "") testdata
time119 = timeparser (many (takeWhile1P Nothing (/='\ESC') <|> "" <$ ansi))
time1191 = timeparser (many ("" <$ try ansi <|> pure <$> anySingle))
timeparser p = timeIt $ print $ length $ concat $ map (concat . fromJust . parseMaybe p) testdata
testdata = concat $ replicate 10000
[ "2008-01-01 income assets🏦checking $1 $1"
, "2008-06-01 gift assets🏦checking $1 $2"
, "2008-06-02 save assets🏦saving $1 $3"
, " assets🏦checking ..m$-1\ESC[m\ESC[m $2"
, "2008-06-03 eat & shop assets:cash ..m$-2\ESC[m\ESC[m 0"
, "2008-12-31 pay off assets🏦checking ..m$-1\ESC[m\ESC[m ..m$-1\ESC[m\ESC[m"
]
ghci> time118ish
4560000
CPU time: 0.17s
ghci> time119
4560000
CPU time: 0.91s
ghci> time1191
4560000
CPU time: 2.76s
Possibly a more careful parser could beat regexReplace. Note the
latter does memoisation, which could be faster and/or could also use
more resident memory in some situations.
Ideally we would calculate all widths before adding ANSI colour codes,
so we wouldn't have to wastefully strip them.
This PR #1330, addressing #1312 (parseQuery is partial) and #1245
(internal server error).
User-visible changes:
- hledger-web now handles malformed regular expressions
(eg, a query consisting of the single character `?`) gracefully,
showing a tidy error message instead "internal server error".
API/internal changes:
- The Regex type alias has been replaced by the Regexp ADT, which
contains both the compiled regular expression (so is guaranteed to
be usable at runtime) and the original string (so can be serialised,
printed, compared, etc.) A Regexp also knows whether is it case
sensitive or case insensitive. The Hledger.Utils.Regex api has changed.
- Typeable and Data instances are no longer derived for hledger's
data types; they were redundant/no longer needed
- NFData instances are no longer derived for hledger's data types.
This speeds up a full build by roughly 7%. But it means we can't
deep-evaluate hledger values, or time hledger code with Criterion.
https://github.com/simonmichael/hledger/pull/1330#issuecomment-684075129
has some ideas on this.
- Query no longer has a custom Show instance
- Some internal use of regexps was replaced by text replacement or
parsers.
- Hledger.Utils.String: quoteIfNeeded now actually escapes quotes in
strings; dropped escapeQuotes
- Hledger.Utils.Tree: dropped some old utilities
- dropped some obsolete code for the old --display option
Merge branch 'regexp' into master
matchesAccount_
matchesAmount_
matchesCommodity_
matchesPosting_
matchesPriceDirective_
matchesTags_
matchesTransaction_
These don't yet have tests of their own, but were converted
mechanically from the originals which should help.
;areg: debug output
;areg: show a title indicating which account was picked
This might be a bit of a pain for scripting, but otherwise it can be
quite confusing if your argument matches an account you didn't expect.
;areg: improve CSV headings
;areg: show at most two commodities per amount
accountTransactionsReport now filters transactions more thoroughly, so
eg transactions dated outside the report period will not be shown.
Previously the transaction would be shown if it had any posting dated
inside the report period. Possibly some other filter criteria now get
applied that didn't before. I think on balance this will give slightly
preferable results.
bal --budget was always showing the period as column heading,
as if for a change report. With --cumulative or --historical
it should show the end date, like other balance reports. Cf
https://hledger.org/hledger.html#multicolumn-balance-report.
This is an API change, but it seems better than having additional
colour-supporting variants and trying to avoid duplicated code.
I stopped short of changing showAmount, so cshowAmount still exists.
Multicolumn balance reports showing many commodities tend to become
unreadably wide, especially in tree mode. Now by default we show at
most two commodities, and a count of the rest if there are more than
two. This should help keep reports somewhat readable by default.
