Commit Graph

57 Commits

Author SHA1 Message Date
Lucas CHOLLET
75bd1308c5 Tests/LibCompress: Add a reproducer of oss-fuzz issue 58046
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=58046
2024-01-13 15:17:08 -07:00
Tim Schumacher
707a36dd79 LibCompress/Brotli: Update the lookback buffer with uncompressed data
We previously skipped updating the lookback buffer when copying
uncompressed data, which resulted in a wrong total byte count.
With a wrong total byte count, our decompressor implementation
ended up choosing a wrong offset into the dictionary.
2024-01-03 17:54:36 +01:00
Lucas CHOLLET
d748edd994 LibCompress: Add a PackBits decoder
This compression scheme was quite popular during the 80's, and we can
still find it in use inside file formats such as TIFF or PDF.
2023-12-27 17:40:11 +01:00
Ali Mohammad Pur
5e1499d104 Everywhere: Rename {Deprecated => Byte}String
This commit un-deprecates DeprecatedString, and repurposes it as a byte
string.
As the null state has already been removed, there are no other
particularly hairy blockers in repurposing this type as a byte string
(what it _really_ is).

This commit is auto-generated:
  $ xs=$(ack -l \bDeprecatedString\b\|deprecated_string AK Userland \
    Meta Ports Ladybird Tests Kernel)
  $ perl -pie 's/\bDeprecatedString\b/ByteString/g;
    s/deprecated_string/byte_string/g' $xs
  $ clang-format --style=file -i \
    $(git diff --name-only | grep \.cpp\|\.h)
  $ gn format $(git ls-files '*.gn' '*.gni')
2023-12-17 18:25:10 +03:30
Tim Schumacher
a1cf2708ee LibCompress: Implement the XZ BCJ filter for ARM64 2023-12-14 08:59:23 -07:00
Idan Horowitz
b749167506 LibCompress: Fix off-by-one error in generate_huffman_lengths
Previously we would calculate the index of the first parent node as
heap.size() (which is initialized to non_zero_freqs), so in the edge
case in which all symbols had a non-zero frequency, we would use the
Size-index entry in the array for both the first symbol's leaf node,
and the first parent node.

The result would either be a non-optimal huffman code (bad), or an
illegal huffman code that would then go on to crash due to an error
check in CanonicalCode::from_bytes. (worse)

We now store parent nodes starting at heap.size() - 1, which eliminates
the potential overlap, and resolves the issue.
2023-12-04 00:06:38 +01:00
Tim Schumacher
25642dfe87 LibCompress: Implement correct validation of last filters 2023-10-29 22:00:59 +01:00
Tim Schumacher
786e654dfd LibCompress: Implement the XZ delta filter 2023-10-29 22:00:59 +01:00
Tim Ledbetter
2f26a7bb12 LibCompress: Avoid buffer overrun when building canonical Huffman code
Previously, decompressing a DEFLATE stream an invalid canonical
Huffman code could cause a buffer overrun. We now return an error in
this case.
2023-10-10 13:24:05 +02:00
Tim Schumacher
dbc25f18ec LibCompress: Let BrotliDecompressionStream take a MaybeOwned 2023-08-23 12:03:37 +01:00
Tim Schumacher
8a853278d0 LibCompress: Port ZlibDecompressor to AK::Stream 2023-08-23 12:03:37 +01:00
Tim Schumacher
a01968ee6d LibCompress: Handle arbitrarily long FF-chains in the LZMA encoder 2023-05-19 23:40:33 +02:00
Tim Schumacher
e8067ac2ee Tests: Add an LZMA roundtrip compression test 2023-05-17 09:08:53 +02:00
Ben Wiederhake
f890b70eae Tests: Prefer TRY_OR_FAIL() and MUST() over EXPECT(!.is_error())
Note that in some cases (in particular SQL::Result and PDFErrorOr),
there is no Formatter defined for the error type, hence TRY_OR_FAIL
cannot work as-is. Furthermore, this commit leaves untouched the places
where MUST could be replaced by TRY_OR_FAIL.

Inspired by:
https://github.com/SerenityOS/serenity/pull/18710#discussion_r1186892445
2023-05-14 15:39:38 -06:00
Tim Schumacher
00ac73be57 Tests: Add a zlib test for "missing end bits"
The actual cause for the "missing bits" is currently unknown, and this
test case doesn't actually start obviously breaking yet unless we start
reporting errors about missing bits. However, since we are touching the
BitStream implementation already, let's add the test early to make extra
sure that we aren't breaking anything.
2023-05-04 20:01:16 +02:00
Tim Schumacher
f47f5ff473 Tests: Add a test for LZMA repetition lengths beyond the distance 2023-04-14 22:52:08 +02:00
Tim Schumacher
4098335600 LibCompress: Error on truncated uncompressed DEFLATE blocks 2023-04-12 14:02:13 -04:00
Tim Schumacher
d4e48db1e1 Tests: Convert LZMA and XZ tests to TRY_OR_FAIL
This is very useful for tracking down a breakage without having to add
temporary dbgln manually.
2023-04-09 18:27:06 +02:00
Timothy Flynn
15532df83d AK+Everywhere: Change AK::fill_with_random to accept a Bytes object
Rather than the very C-like API we currently have, accepting a void* and
a length, let's take a Bytes object instead. In almost all existing
cases, the compiler figures out the length.
2023-04-03 15:53:49 +02:00
Tim Schumacher
ad31265e60 LibCompress: Implement block size validation for XZ streams 2023-04-01 13:57:54 +02:00
Tim Schumacher
b451964bbc Tests: Document input of the xz_utils_good_1_block_header_1 test case 2023-04-01 13:57:54 +02:00
Tim Schumacher
fe761a4e9b LibCompress: Use LZMA context from preexisting dictionary 2023-03-30 14:39:31 +02:00
Tim Schumacher
c020ee8bfa LibCompress: Avoid overflowing the size of uncompressed LZMA2 chunks 2023-03-30 14:39:31 +02:00
Tim Schumacher
9ccb0fc1d8 LibCompress: Only require new LZMA2 properties after dictionary reset 2023-03-30 14:39:31 +02:00
Tim Schumacher
726963edc7 LibCompress: Implement support for multiple concatenated XZ streams 2023-03-30 14:38:47 +02:00
Tim Schumacher
00332c9b7d LibCompress: Move XZ header validation into the read function
The constructor is now only concerned with creating the required
streams, which means that it no longer fails for XZ streams with
invalid headers. Instead, everything is parsed and validated during the
first read, preparing us for files with multiple streams.
2023-03-30 14:38:47 +02:00
Tim Schumacher
8ff36e5910 LibCompress: Implement proper handling of LZMA end-of-stream markers 2023-03-30 08:45:35 +02:00
Tim Schumacher
78ed798852 Tests: Add XZ tests from XZ utils 2023-03-21 10:25:13 +01:00
Tim Schumacher
391485ccef Tests: Add LZMA test files from the LZMA specification 2023-03-20 12:15:38 +02:00
Tim Schumacher
d5871f5717 AK: Rename Stream::{read,write} to Stream::{read_some,write_some}
Similar to POSIX read, the basic read and write functions of AK::Stream
do not have a lower limit of how much data they read or write (apart
from "none at all").

Rename the functions to "read some [data]" and "write some [data]" (with
"data" being omitted, since everything here is reading and writing data)
to make them sufficiently distinct from the functions that ensure to
use the entire buffer (which should be the go-to function for most
usages).

No functional changes, just a lot of new FIXMEs.
2023-03-13 15:16:20 +00:00
Tim Schumacher
874c7bba28 LibCore: Remove Stream.h 2023-02-13 00:50:07 +00:00
Tim Schumacher
606a3982f3 LibCore: Move Stream-based file into the Core namespace 2023-02-13 00:50:07 +00:00
Tim Schumacher
220fbcaa7e AK: Remove the fallible constructor from FixedMemoryStream 2023-02-08 17:44:32 +00:00
Tim Schumacher
261d62438f AK: Remove the fallible constructor from LittleEndianInputBitStream 2023-02-08 17:44:32 +00:00
Tim Schumacher
093cf428a3 AK: Move memory streams from LibCore 2023-01-29 19:16:44 -07:00
Tim Schumacher
2470dd3bb5 AK: Move bit streams from LibCore 2023-01-29 19:16:44 -07:00
Tim Schumacher
ae64b68717 AK: Deprecate the old AK::Stream
This also removes a few cases where the respective header wasn't
actually required to be included.
2023-01-29 19:16:44 -07:00
Tim Schumacher
8cd2cf2b77 LibCompress: Port DeflateCompressor to Core::Stream 2023-01-10 10:28:26 +01:00
Tim Schumacher
a212bc3052 LibCompress: Port GzipCompressor to Core::Stream 2023-01-10 10:28:26 +01:00
Tim Schumacher
0bdbe27d6b LibCore: Rename InputBitStream.h to BitStream.h
We won't just be defining readable streams here from now on, but also
writable streams.
2023-01-10 10:28:26 +01:00
Tim Schumacher
23a9d62f39 LibCompress: Rename Compress::Zlib to Compress::ZlibDecompressor
Because that's what it is, even if it mainly relies on
`DeflateDecompressor` to do the heavy lifting.
2023-01-03 10:01:09 +00:00
Tim Schumacher
edd004fe90 Tests: Add a simple compression test for Zlib 2023-01-03 10:01:09 +00:00
Tim Schumacher
c6d71ca727 LibCore: Rename MemoryStream to FixedMemoryStream
This is to differentiate between the upcoming `AllocatingMemoryStream`,
which automatically allocates memory as needed instead of operating on a
static memory area.
2022-12-15 13:28:29 +00:00
Tim Schumacher
30abd47099 LibCompress: Port DeflateDecompressor to Core::Stream 2022-12-12 16:21:39 +00:00
Tim Schumacher
ed4c2f2f8e LibCore: Rename Stream::read_all to read_until_eof
This generally seems like a better name, especially if we somehow also
need a better name for "read the entire buffer, but not the entire file"
somewhere down the line.
2022-12-12 14:16:42 +01:00
Tim Schumacher
f93c7fbb5e LibCompress: Port GzipDecompressor to Core::Stream 2022-12-10 12:05:55 +00:00
Linus Groh
6e19ab2bbc AK+Everywhere: Rename String to DeprecatedString
We have a new, improved string type coming up in AK (OOM aware, no null
state), and while it's going to use UTF-8, the name UTF8String is a
mouthful - so let's free up the String name by renaming the existing
class.
Making the old one have an annoying name will hopefully also help with
quick adoption :^)
2022-12-06 08:54:33 +01:00
Tommy Murphy
0ee98c69c1 LibCompress: Brotli support metadata of skip_length=0
The relevant RFC section from
https://www.rfc-editor.org/rfc/rfc7932#section-9.2

MSKIPBYTES * 8 bits: MSKIPLEN - 1, where MSKIPLEN is
       the number of metadata bytes; this field is
       only present if MSKIPBYTES is positive;
       otherwise, MSKIPLEN is 0 (if MSKIPBYTES is
       greater than 1, and the last byte is all
       zeros, then the stream should be rejected as
       invalid)

So when skip_bytes is zero we need to break and
re-align bytes.

Added the relevant test case that demonstrates this from:
https://github.com/google/brotli/blob/master/tests/testdata/x.compressed
2022-10-11 22:49:33 -06:00
Andrew Kaster
828441852f Everywhere: Replace uses of __serenity__ with AK_OS_SERENITY
Now that we have OS macros for essentially every supported OS, let's try
to use them everywhere.
2022-10-10 12:23:12 +02:00
Ben Wiederhake
8de7a91571 LibCompress+Tests: Demonstrate and fix faulty metadata length
The test-case is heavily inspired by:
https://github.com/google/brotli/blob/master/tests/testdata/x.compressed.01
Or in words: A metadata meta-block containing `Y` (which should be
ignored), and then the actual data (a single `Z`). The bug used to skip
one metadata byte too few, and thus read garbage.
2022-09-17 04:10:11 +00:00